Databricks Runtime 7.2 (unsupported)

  • Article

Databricks released this image in August 2020.

The following release notes provide information about Databricks Runtime 7.2, powered by Apache Spark 3.0.

New features

Databricks Runtime 7.2 includes the following new features.

Auto Loader is generally available

Auto Loader is an efficient method for incrementally ingesting a large number of files into Delta Lake. It is now GA and adds the following features: directory listing, cloud resource management API, rate limiting option, and option validation.

Directory listing mode option

Auto Loader adds a new directory listing mode, in addition to the existing file notification mode, for determining when there are new files.

  • Directory listing mode: In some scenarios there are only a few files that need to be streamed in on a regular basis. In those cases, identifying new files by parallel listing of the input directory and ingesting them is reasonably efficient. This allows you to start Auto Loader streams without any permission configuration. Directory listing mode is the default mode for Auto Loader from Databricks Runtime 7.2 and above.
  • File notification mode: You can explicitly opt in to file notification mode, which is more performant and scalable for large input directories, but requires you to configure permissions for the notification service to subscribe to file events and set .option("cloudFiles.useNotifications", "true"). Existing Auto Loader streams started on Databricks Runtime 7.1 and below automatically opt in to file notification mode when they upgrade to Databricks Runtime 7.2.

For more information, see What is Auto Loader?.

Cloud resource management API

You can now use our Scala API to manage cloud resources created by Auto Loader. You can list notification services and tear down specific notification services using this API. For more information, see Manually configure or manage file notification resources.

Rate limiting option

You can now use the cloudFiles.maxBytesPerTrigger option to limit the amount of data processed in each microbatch. You can specify a byte string such as 10g to limit each microbatch to 10 GB of data. This is a soft maximum. If you have files that are 3 GB each, Azure Databricks processes 12 GB in a microbatch. This option has no effect when used with Trigger.Once(). When this option is not provided, the default value of cloudFiles.maxFilesPerTrigger of 1000 files is used for rate limiting. When used together with cloudFiles.maxFilesPerTrigger, Azure Databricks consumes up to the lowest limit of cloudFiles.maxFilesPerTrigger or cloudFiles.maxBytesPerTrigger, whichever is reached first.

For more information, see What is Auto Loader?.

Option validation

Auto Loader now validates the options you provide. If you provide an unknown option, Auto Loader returns an error instead of silently ignoring the option. The validation also detects inconsistencies in your provided options. For example, if you set cloudFiles.useNotifications to false and at the same time provide permission options for Azure Event Grid and Queue Storage services, validation will fail. To skip option validation, set cloudFiles.validateOptions to false.

Changes in default option values and compatibility

The default behavior of some Auto Loader options changed in Databricks Runtime 7.2. To ensure compatibility, the behavior of streams started on Databricks Runtime 7.1 and below will remain unchanged even after you upgrade to running such streams on a cluster running Databricks Runtime 7.2 or above. See What is Auto Loader?.

Efficiently copy a Delta table with clone (Public Preview)

You can now efficiently create a shallow or deep clone of a Delta table at a specific version with the clone command. A deep clone is a standalone clone of a Delta table formed by copying the files in parallel over from the source table to the target table. A shallow clone is a cheaply created clone that don't copy files to the target table but points to files in the original table. Any changes made to a deep or shallow clone affects only the clone and not the original table. See Clone a table on Azure Databricks.

Improvements

  • Snowflake connector: The connector has been upgraded to version 2.8.1, which includes Spark 3.0 support.

  • PySpark usability: when a PySpark or pandas UDF fails, Azure Databricks now surfaces the Python-side exception and most recently executed line of notebook code (if applicable) at the top of the error message.

  • Credential passthrough:

    • You can now use the LDA transform function on a passthrough-enabled cluster.
    • You can now set the storage checkpoint directory on a passthrough-enabled cluster.

  • Cluster libraries: Azure Databricks now processes all cluster libraries in the order that they were installed on the cluster. On Databricks Runtime 7.1 and below, only Maven and CRAN libraries were processed in the order they were installed on the cluster. Other libraries were installed in a nondeterministic order.

  • TensorBoard:

    • Manage TensorBoard processes with the tensorboard.notebook API.
    • You can have multiple TensorBoard backends running on a single cluster.
    • In Python notebooks, display the TensorBoard UI inline using the %tensorboard line magic command.
    • dbutils.tensorboard is deprecated.

    See TensorBoard.

  • Upgraded Python libraries:

    • jmespath upgraded from 0.9.4 to 0.10.0.
    • koalas upgraded from 1.0.1 to 1.1.0.

  • Upgraded several installed R libraries. See Installed R Libraries.

Apache Spark

Databricks Runtime 7.2 includes Apache Spark 3.0.0. This release includes all Spark fixes and improvements included in Databricks Runtime 7.1 (unsupported), as well as the following additional bug fixes and improvements made to Spark:

  • [SPARK-32234] [SQL] Spark sql commands are failing on selecting the orc tables
  • [SPARK-30616] [SQL] Introduce TTL config option for SQL Metadata Cache
  • [SPARK-21117] [SQL] Built-in SQL Function Support - WIDTH_BUCKET
  • [SPARK-32168] [SQL] Fix hidden partitioning correctness bug in SQL overwrite
  • [SPARK-31935] Hadoop file system config should be effective in data source options
  • [SPARK-32270] [SQL] Use TextFileFormat in CSV's schema inference with a different encoding
  • [SPARK-28481] [SQL] More expressions should extend NullIntolerant
  • [SPARK-32220] [SQL] SHUFFLE_REPLICATE_NL Hint should not change Non-Cartesian Product join result
  • [SPARK-32232] [ML][PYSPARK] Make sure ML has the same default solver values between Scala and Python
  • [SPARK-32238] [SQL] Use Utils.getSimpleName to avoid hitting Malformed class name in ScalaUDF
  • [SPARK-32220] [SQL] SHUFFLE_REPLICATE_NL Hint should not change Non-Cartesian Product join result
  • [SPARK-32256] [SQL] Force to initialize Hadoop VersionInfo in HiveExternalCatalog
  • [SPARK-32159] [SQL] Fix integration between Aggregator[Array[_], _, _] and UnresolvedMapObjects
  • [SPARK-32160] [CORE][PYSPARK] Disallow to create SparkContext in executors
  • [SPARK-32168] [SQL] Fix hidden partitioning correctness bug in SQL overwrite
  • [SPARK-32024] [WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
  • [SPARK-32214] [SQL] The type conversion function generated in makeFromJava for "other" type uses a wrong variable
  • [SPARK-30703] [SQL] Update SqlBase.g4 invalid comment
  • [SPARK-32163] [SQL] Nested pruning should work even with cosmetic variations
  • [SPARK-32124] [CORE] Use the invalid value Int.MinValue to fill the map index when the event logs from the old Spark version
  • [SPARK-32167] [SQL] Fix GetArrayStructFields to respect inner field's nullability together
  • [SPARK-32130] [SQL] Enable timestamps inference in JsonBenchmark
  • [SPARK-32121] [SHUFFLE] Support Windows OS in ExecutorDiskUtils
  • [SPARK-32130] [SQL] Disable the JSON option inferTimestamp by default
  • [SPARK-31935] [SQL] Hadoop file system config should be effective in data source options
  • [SPARK-32095] [SQL] Update documentation to reflect usage of updated statistics
  • [SPARK-32131] [SQL] Fix AnalysisException messages at UNION/EXCEPT/MINUS operations
  • [SPARK-32068] [WEBUI] Correct task lauchtime show issue due to timezone in stage tab
  • [SPARK-29999] [SS] Fix test to check the actual metadata log directory
  • [SPARK-32127] [SQL] Check rules for MERGE INTO should use MergeAction.conditition other than MergeAction.children
  • [SPARK-32038] [SQL] Make the alias name pretty after float/double normalization
  • [SPARK-32126] [SS] Scope Session.active in IncrementalExecution
  • [SPARK-32115] [SQL] Fix SUBSTRING to handle integer overflows
  • [SPARK-32098] [PYTHON] Use iloc for positional slicing instead of direct slicing in createDataFrame with Arrow
  • [SPARK-32080] [SPARK-31998] [SQL] Simplify ArrowColumnVector ListArray accessor
  • [SPARK-32028] [WEBUI] fix app id link for multi attempts app in history summary page
  • [SPARK-31918] [R] Ignore S4 generic methods under SparkR namespace in closure cleaning to support R 4.0.0+
  • [SPARK-32038] [SQL] NormalizeFloatingNumbers should also work on distinct aggregate
  • [SPARK-32034] [SQL] Port HIVE-14817: Shutdown the SessionManager timeoutChecker thread properly upon shutdown
  • [SPARK-32021] [SQL] Increase precision of seconds and fractions of make_interval
  • [SPARK-31980] [SQL] Function sequence() fails if start and end of range are equal dates
  • [SPARK-32020] [SQL] Better error message when SPARK_HOME or spark.test.home is not set
  • [SPARK-32011] [PYTHON][CORE] Remove warnings about pin-thread modes and guide to use collectWithJobGroup

Maintenance updates

See Databricks Runtime 7.2 maintenance updates.

System environment

  • Operating System: Ubuntu 18.04.4 LTS
  • Java: 1.8.0_252
  • Scala: 2.12.10
  • Python: 3.7.5
  • R: R version 3.6.3 (2020-02-29)
  • Delta Lake 0.7.0

Installed Python libraries

Library Version Library Version Library Version
asn1crypto 1.3.0 backcall 0.1.0 boto3 1.12.0
botocore 1.15.0 certifi 2020.6.20 cffi 1.14.0
chardet 3.0.4 cryptography 2.8 cycler 0.10.0
Cython 0.29.15 decorator 4.4.1 docutils 0.15.2
entrypoints 0.3 idna 2.8 ipykernel 5.1.4
ipython 7.12.0 ipython-genutils 0.2.0 jedi 0.14.1
jmespath 0.10.0 joblib 0.14.1 jupyter-client 5.3.4
jupyter-core 4.6.1 kiwisolver 1.1.0 koalas 1.1.0
matplotlib 3.1.3 numpy 1.18.1 pandas 1.0.1
parso 0.5.2 patsy 0.5.1 pexpect 4.8.0
pickleshare 0.7.5 pip 20.0.2 prompt-toolkit 3.0.3
psycopg2 2.8.4 ptyprocess 0.6.0 pyarrow 0.15.1
pycparser 2.19 Pygments 2.5.2 PyGObject 3.26.1
pyOpenSSL 19.1.0 pyparsing 2.4.6 PySocks 1.7.1
python-apt 1.6.5+ubuntu0.3 python-dateutil 2.8.1 pytz 2019.3
pyzmq 18.1.1 requests 2.22.0 s3transfer 0.3.3
scikit-learn 0.22.1 scipy 1.4.1 seaborn 0.10.0
setuptools 45.2.0 six 1.14.0 ssh-import-id 5.7
statsmodels 0.11.0 tornado 6.0.3 traitlets 4.3.3
unattended-upgrades 0.1 urllib3 1.25.8 virtualenv 16.7.10
wcwidth 0.1.8 wheel 0.34.2

Installed R libraries

R libraries are installed from Microsoft CRAN snapshot on 2020-07-14.

Library Version Library Version Library Version
askpass 1.1 assertthat 0.2.1 backports 1.1.8
base 3.6.3 base64enc 0.1-3 BH 1.72.0-3
bit 1.1-15.2 bit64 0.9-7 blob 1.2.1
boot 1.3-25 brew 1.0-6 broom 0.7.0
callr 3.4.3 caret 6.0-86 cellranger 1.1.0
chron 2.3-55 class 7.3-17 cli 2.0.2
clipr 0.7.0 cluster 2.1.0 codetools 0.2-16
colorspace 1.4-1 commonmark 1.7 compiler 3.6.3
config 0.3 covr 3.5.0 crayon 1.3.4
crosstalk 1.1.0.1 curl 4.3 data.table 1.12.8
datasets 3.6.3 DBI 1.1.0 dbplyr 1.4.4
desc 1.2.0 devtools 2.3.0 digest 0.6.25
dplyr 0.8.5 DT 0.14 ellipsis 0.3.1
evaluate 0.14 fansi 0.4.1 farver 2.0.3
fastmap 1.0.1 forcats 0.5.0 foreach 1.5.0
foreign 0.8-76 forge 0.2.0 fs 1.4.2
generics 0.0.2 ggplot2 3.3.2 gh 1.1.0
git2r 0.27.1 glmnet 3.0-2 globals 0.12.5
glue 1.4.1 gower 0.2.2 graphics 3.6.3
grDevices 3.6.3 grid 3.6.3 gridExtra 2.3
gsubfn 0.7 gtable 0.3.0 haven 2.3.1
highr 0.8 hms 0.5.3 htmltools 0.5.0
htmlwidgets 1.5.1 httpuv 1.5.4 httr 1.4.1
hwriter 1.3.2 hwriterPlus 1.0-3 ini 0.3.1
ipred 0.9-9 isoband 0.2.2 iterators 1.0.12
jsonlite 1.7.0 KernSmooth 2.23-17 knitr 1.29
labeling 0.3 later 1.1.0.1 lattice 0.20-41
lava 1.6.7 lazyeval 0.2.2 lifecycle 0.2.0
lubridate 1.7.9 magrittr 1.5 markdown 1.1
MASS 7.3-51.6 Matrix 1.2-18 memoise 1.1.0
methods 3.6.3 mgcv 1.8-31 mime 0.9
ModelMetrics 1.2.2.2 modelr 0.1.8 munsell 0.5.0
nlme 3.1-147 nnet 7.3-14 numDeriv 2016.8-1.1
openssl 1.4.2 parallel 3.6.3 pillar 1.4.6
pkgbuild 1.1.0 pkgconfig 2.0.3 pkgload 1.1.0
plogr 0.2.0 plyr 1.8.6 praise 1.0.0
prettyunits 1.1.1 pROC 1.16.2 processx 3.4.3
prodlim 2019.11.13 progress 1.2.2 promises 1.1.1
proto 1.0.0 ps 1.3.3 purrr 0.3.4
r2d3 0.2.3 R6 2.4.1 randomForest 4.6-14
rappdirs 0.3.1 rcmdcheck 1.3.3 RColorBrewer 1.1-2
Rcpp 1.0.5 readr 1.3.1 readxl 1.3.1
recipes 0.1.13 rematch 1.0.1 rematch2 2.1.2
remotes 2.1.1 reprex 0.3.0 reshape2 1.4.4
rex 1.2.0 rjson 0.2.20 rlang 0.4.7
rmarkdown 2.3 RODBC 1.3-16 roxygen2 7.1.1
rpart 4.1-15 rprojroot 1.3-2 Rserve 1.8-7
RSQLite 2.2.0 rstudioapi 0.11 rversions 2.0.2
rvest 0.3.5 scales 1.1.1 selectr 0.4-2
sessioninfo 1.1.1 shape 1.4.4 shiny 1.5.0
sourcetools 0.1.7 sparklyr 1.3.1 SparkR 3.0.0
spatial 7.3-11 splines 3.6.3 sqldf 0.4-11
SQUAREM 2020.3 stats 3.6.3 stats4 3.6.3
stringi 1.4.6 stringr 1.4.0 survival 3.1-12
sys 3.3 tcltk 3.6.3 TeachingDemos 2.10
testthat 2.3.2 tibble 3.0.3 tidyr 1.1.0
tidyselect 1.1.0 tidyverse 1.3.0 timeDate 3043.102
tinytex 0.24 tools 3.6.3 usethis 1.6.1
utf8 1.1.4 utils 3.6.3 uuid 0.1-4
vctrs 0.3.1 viridisLite 0.3.0 whisker 0.4
withr 2.2.0 xfun 0.15 xml2 1.3.2
xopen 1.0.0 xtable 1.8-4 yaml 2.2.1

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID Artifact ID Version
antlr antlr 2.7.7
com.amazonaws amazon-kinesis-client 1.12.0
com.amazonaws aws-java-sdk-autoscaling 1.11.655
com.amazonaws aws-java-sdk-cloudformation 1.11.655
com.amazonaws aws-java-sdk-cloudfront 1.11.655
com.amazonaws aws-java-sdk-cloudhsm 1.11.655
com.amazonaws aws-java-sdk-cloudsearch 1.11.655
com.amazonaws aws-java-sdk-cloudtrail 1.11.655
com.amazonaws aws-java-sdk-cloudwatch 1.11.655
com.amazonaws aws-java-sdk-cloudwatchmetrics 1.11.655
com.amazonaws aws-java-sdk-codedeploy 1.11.655
com.amazonaws aws-java-sdk-cognitoidentity 1.11.655
com.amazonaws aws-java-sdk-cognitosync 1.11.655
com.amazonaws aws-java-sdk-config 1.11.655
com.amazonaws aws-java-sdk-core 1.11.655
com.amazonaws aws-java-sdk-datapipeline 1.11.655
com.amazonaws aws-java-sdk-directconnect 1.11.655
com.amazonaws aws-java-sdk-directory 1.11.655
com.amazonaws aws-java-sdk-dynamodb 1.11.655
com.amazonaws aws-java-sdk-ec2 1.11.655
com.amazonaws aws-java-sdk-ecs 1.11.655
com.amazonaws aws-java-sdk-efs 1.11.655
com.amazonaws aws-java-sdk-elasticache 1.11.655
com.amazonaws aws-java-sdk-elasticbeanstalk 1.11.655
com.amazonaws aws-java-sdk-elasticloadbalancing 1.11.655
com.amazonaws aws-java-sdk-elastictranscoder 1.11.655
com.amazonaws aws-java-sdk-emr 1.11.655
com.amazonaws aws-java-sdk-glacier 1.11.655
com.amazonaws aws-java-sdk-iam 1.11.655
com.amazonaws aws-java-sdk-importexport 1.11.655
com.amazonaws aws-java-sdk-kinesis 1.11.655
com.amazonaws aws-java-sdk-kms 1.11.655
com.amazonaws aws-java-sdk-lambda 1.11.655
com.amazonaws aws-java-sdk-logs 1.11.655
com.amazonaws aws-java-sdk-machinelearning 1.11.655
com.amazonaws aws-java-sdk-opsworks 1.11.655
com.amazonaws aws-java-sdk-rds 1.11.655
com.amazonaws aws-java-sdk-redshift 1.11.655
com.amazonaws aws-java-sdk-route53 1.11.655
com.amazonaws aws-java-sdk-s3 1.11.655
com.amazonaws aws-java-sdk-ses 1.11.655
com.amazonaws aws-java-sdk-simpledb 1.11.655
com.amazonaws aws-java-sdk-simpleworkflow 1.11.655
com.amazonaws aws-java-sdk-sns 1.11.655
com.amazonaws aws-java-sdk-sqs 1.11.655
com.amazonaws aws-java-sdk-ssm 1.11.655
com.amazonaws aws-java-sdk-storagegateway 1.11.655
com.amazonaws aws-java-sdk-sts 1.11.655
com.amazonaws aws-java-sdk-support 1.11.655
com.amazonaws aws-java-sdk-swf-libraries 1.11.22
com.amazonaws aws-java-sdk-workspaces 1.11.655
com.amazonaws jmespath-java 1.11.655
com.chuusai shapeless_2.12 2.3.3
com.clearspring.analytics stream 2.9.6
com.databricks Rserve 1.8-3
com.databricks jets3t 0.7.1-0
com.databricks.scalapb compilerplugin_2.12 0.4.15-10
com.databricks.scalapb scalapb-runtime_2.12 0.4.15-10
com.esotericsoftware kryo-shaded 4.0.2
com.esotericsoftware minlog 1.3.0
com.fasterxml classmate 1.3.4
com.fasterxml.jackson.core jackson-annotations 2.10.0
com.fasterxml.jackson.core jackson-core 2.10.0
com.fasterxml.jackson.core jackson-databind 2.10.0
com.fasterxml.jackson.dataformat jackson-dataformat-cbor 2.10.0
com.fasterxml.jackson.datatype jackson-datatype-joda 2.10.0
com.fasterxml.jackson.module jackson-module-paranamer 2.10.0
com.fasterxml.jackson.module jackson-module-scala_2.12 2.10.0
com.github.ben-manes.caffeine caffeine 2.3.4
com.github.fommil jniloader 1.1
com.github.fommil.netlib core 1.1.2
com.github.fommil.netlib native_ref-java 1.1
com.github.fommil.netlib native_ref-java-natives 1.1
com.github.fommil.netlib native_system-java 1.1
com.github.fommil.netlib native_system-java-natives 1.1
com.github.fommil.netlib netlib-native_ref-linux-x86_64-natives 1.1
com.github.fommil.netlib netlib-native_system-linux-x86_64-natives 1.1
com.github.joshelser dropwizard-metrics-hadoop-metrics2-reporter 0.1.2
com.github.luben zstd-jni 1.4.4-3
com.github.wendykierp JTransforms 3.1
com.google.code.findbugs jsr305 3.0.0
com.google.code.gson gson 2.2.4
com.google.flatbuffers flatbuffers-java 1.9.0
com.google.guava guava 15.0
com.google.protobuf protobuf-java 2.6.1
com.h2database h2 1.4.195
com.helger profiler 1.1.1
com.jcraft jsch 0.1.50
com.jolbox bonecp 0.8.0.RELEASE
com.lihaoyi sourcecode_2.12 0.1.9
com.microsoft.azure azure-data-lake-store-sdk 2.2.8
com.microsoft.sqlserver mssql-jdbc 8.2.1.jre8
com.ning compress-lzf 1.0.3
com.sun.mail javax.mail 1.5.2
com.tdunning json 1.8
com.thoughtworks.paranamer paranamer 2.8
com.trueaccord.lenses lenses_2.12 0.4.12
com.twitter chill-java 0.9.5
com.twitter chill_2.12 0.9.5
com.twitter util-app_2.12 7.1.0
com.twitter util-core_2.12 7.1.0
com.twitter util-function_2.12 7.1.0
com.twitter util-jvm_2.12 7.1.0
com.twitter util-lint_2.12 7.1.0
com.twitter util-registry_2.12 7.1.0
com.twitter util-stats_2.12 7.1.0
com.typesafe config 1.2.1
com.typesafe.scala-logging scala-logging_2.12 3.7.2
com.univocity univocity-parsers 2.8.3
com.zaxxer HikariCP 3.1.0
commons-beanutils commons-beanutils 1.9.4
commons-cli commons-cli 1.2
commons-codec commons-codec 1.10
commons-collections commons-collections 3.2.2
commons-configuration commons-configuration 1.6
commons-dbcp commons-dbcp 1.4
commons-digester commons-digester 1.8
commons-fileupload commons-fileupload 1.3.3
commons-httpclient commons-httpclient 3.1
commons-io commons-io 2.4
commons-lang commons-lang 2.6
commons-logging commons-logging 1.1.3
commons-net commons-net 3.1
commons-pool commons-pool 1.5.4
info.ganglia.gmetric4j gmetric4j 1.0.10
io.airlift aircompressor 0.10
io.dropwizard.metrics metrics-core 4.1.1
io.dropwizard.metrics metrics-graphite 4.1.1
io.dropwizard.metrics metrics-healthchecks 4.1.1
io.dropwizard.metrics metrics-jetty9 4.1.1
io.dropwizard.metrics metrics-jmx 4.1.1
io.dropwizard.metrics metrics-json 4.1.1
io.dropwizard.metrics metrics-jvm 4.1.1
io.dropwizard.metrics metrics-servlets 4.1.1
io.netty netty-all 4.1.47.Final
jakarta.annotation jakarta.annotation-api 1.3.5
jakarta.validation jakarta.validation-api 2.0.2
jakarta.ws.rs jakarta.ws.rs-api 2.1.6
javax.activation activation 1.1.1
javax.el javax.el-api 2.2.4
javax.jdo jdo-api 3.0.1
javax.servlet javax.servlet-api 3.1.0
javax.servlet.jsp jsp-api 2.1
javax.transaction jta 1.1
javax.transaction transaction-api 1.1
javax.xml.bind jaxb-api 2.2.2
javax.xml.stream stax-api 1.0-2
javolution javolution 5.5.1
jline jline 2.14.6
joda-time joda-time 2.10.5
log4j apache-log4j-extras 1.2.17
log4j log4j 1.2.17
net.razorvine pyrolite 4.30
net.sf.jpam jpam 1.1
net.sf.opencsv opencsv 2.3
net.sf.supercsv super-csv 2.2.0
net.snowflake snowflake-ingest-sdk 0.9.6
net.snowflake snowflake-jdbc 3.12.8
net.snowflake spark-snowflake_2.12 2.8.1-spark_3.0
net.sourceforge.f2j arpack_combined_all 0.1
org.acplt.remotetea remotetea-oncrpc 1.1.2
org.antlr ST4 4.0.4
org.antlr antlr-runtime 3.5.2
org.antlr antlr4-runtime 4.7.1
org.antlr stringtemplate 3.2.1
org.apache.ant ant 1.9.2
org.apache.ant ant-jsch 1.9.2
org.apache.ant ant-launcher 1.9.2
org.apache.arrow arrow-format 0.15.1
org.apache.arrow arrow-memory 0.15.1
org.apache.arrow arrow-vector 0.15.1
org.apache.avro avro 1.8.2
org.apache.avro avro-ipc 1.8.2
org.apache.avro avro-mapred-hadoop2 1.8.2
org.apache.commons commons-compress 1.8.1
org.apache.commons commons-crypto 1.0.0
org.apache.commons commons-lang3 3.9
org.apache.commons commons-math3 3.4.1
org.apache.commons commons-text 1.6
org.apache.curator curator-client 2.7.1
org.apache.curator curator-framework 2.7.1
org.apache.curator curator-recipes 2.7.1
org.apache.derby derby 10.12.1.1
org.apache.directory.api api-asn1-api 1.0.0-M20
org.apache.directory.api api-util 1.0.0-M20
org.apache.directory.server apacheds-i18n 2.0.0-M15
org.apache.directory.server apacheds-kerberos-codec 2.0.0-M15
org.apache.hadoop hadoop-annotations 2.7.4
org.apache.hadoop hadoop-auth 2.7.4
org.apache.hadoop hadoop-client 2.7.4
org.apache.hadoop hadoop-common 2.7.4
org.apache.hadoop hadoop-hdfs 2.7.4
org.apache.hadoop hadoop-mapreduce-client-app 2.7.4
org.apache.hadoop hadoop-mapreduce-client-common 2.7.4
org.apache.hadoop hadoop-mapreduce-client-core 2.7.4
org.apache.hadoop hadoop-mapreduce-client-jobclient 2.7.4
org.apache.hadoop hadoop-mapreduce-client-shuffle 2.7.4
org.apache.hadoop hadoop-yarn-api 2.7.4
org.apache.hadoop hadoop-yarn-client 2.7.4
org.apache.hadoop hadoop-yarn-common 2.7.4
org.apache.hadoop hadoop-yarn-server-common 2.7.4
org.apache.hive hive-beeline 2.3.7
org.apache.hive hive-cli 2.3.7
org.apache.hive hive-common 2.3.7
org.apache.hive hive-exec-core 2.3.7
org.apache.hive hive-jdbc 2.3.7
org.apache.hive hive-llap-client 2.3.7
org.apache.hive hive-llap-common 2.3.7
org.apache.hive hive-metastore 2.3.7
org.apache.hive hive-serde 2.3.7
org.apache.hive hive-shims 2.3.7
org.apache.hive hive-storage-api 2.7.1
org.apache.hive hive-vector-code-gen 2.3.7
org.apache.hive.shims hive-shims-0.23 2.3.7
org.apache.hive.shims hive-shims-common 2.3.7
org.apache.hive.shims hive-shims-scheduler 2.3.7
org.apache.htrace htrace-core 3.1.0-incubating
org.apache.httpcomponents httpclient 4.5.6
org.apache.httpcomponents httpcore 4.4.12
org.apache.ivy ivy 2.4.0
org.apache.orc orc-core 1.5.10
org.apache.orc orc-mapreduce 1.5.10
org.apache.orc orc-shims 1.5.10
org.apache.parquet parquet-column 1.10.1.2-databricks4
org.apache.parquet parquet-common 1.10.1.2-databricks4
org.apache.parquet parquet-encoding 1.10.1.2-databricks4
org.apache.parquet parquet-format 2.4.0
org.apache.parquet parquet-hadoop 1.10.1.2-databricks4
org.apache.parquet parquet-jackson 1.10.1.2-databricks4
org.apache.thrift libfb303 0.9.3
org.apache.thrift libthrift 0.12.0
org.apache.velocity velocity 1.5
org.apache.xbean xbean-asm7-shaded 4.15
org.apache.yetus audience-annotations 0.5.0
org.apache.zookeeper zookeeper 3.4.14
org.codehaus.jackson jackson-core-asl 1.9.13
org.codehaus.jackson jackson-jaxrs 1.9.13
org.codehaus.jackson jackson-mapper-asl 1.9.13
org.codehaus.jackson jackson-xc 1.9.13
org.codehaus.janino commons-compiler 3.0.16
org.codehaus.janino janino 3.0.16
org.datanucleus datanucleus-api-jdo 4.2.4
org.datanucleus datanucleus-core 4.1.17
org.datanucleus datanucleus-rdbms 4.1.19
org.datanucleus javax.jdo 3.2.0-m3
org.eclipse.jetty jetty-client 9.4.18.v20190429
org.eclipse.jetty jetty-continuation 9.4.18.v20190429
org.eclipse.jetty jetty-http 9.4.18.v20190429
org.eclipse.jetty jetty-io 9.4.18.v20190429
org.eclipse.jetty jetty-jndi 9.4.18.v20190429
org.eclipse.jetty jetty-plus 9.4.18.v20190429
org.eclipse.jetty jetty-proxy 9.4.18.v20190429
org.eclipse.jetty jetty-security 9.4.18.v20190429
org.eclipse.jetty jetty-server 9.4.18.v20190429
org.eclipse.jetty jetty-servlet 9.4.18.v20190429
org.eclipse.jetty jetty-servlets 9.4.18.v20190429
org.eclipse.jetty jetty-util 9.4.18.v20190429
org.eclipse.jetty jetty-webapp 9.4.18.v20190429
org.eclipse.jetty jetty-xml 9.4.18.v20190429
org.fusesource.leveldbjni leveldbjni-all 1.8
org.glassfish.hk2 hk2-api 2.6.1
org.glassfish.hk2 hk2-locator 2.6.1
org.glassfish.hk2 hk2-utils 2.6.1
org.glassfish.hk2 osgi-resource-locator 1.0.3
org.glassfish.hk2.external aopalliance-repackaged 2.6.1
org.glassfish.hk2.external jakarta.inject 2.6.1
org.glassfish.jersey.containers jersey-container-servlet 2.30
org.glassfish.jersey.containers jersey-container-servlet-core 2.30
org.glassfish.jersey.core jersey-client 2.30
org.glassfish.jersey.core jersey-common 2.30
org.glassfish.jersey.core jersey-server 2.30
org.glassfish.jersey.inject jersey-hk2 2.30
org.glassfish.jersey.media jersey-media-jaxb 2.30
org.hibernate.validator hibernate-validator 6.1.0.Final
org.javassist javassist 3.25.0-GA
org.jboss.logging jboss-logging 3.3.2.Final
org.jdbi jdbi 2.63.1
org.joda joda-convert 1.7
org.jodd jodd-core 3.5.2
org.json4s json4s-ast_2.12 3.6.6
org.json4s json4s-core_2.12 3.6.6
org.json4s json4s-jackson_2.12 3.6.6
org.json4s json4s-scalap_2.12 3.6.6
org.lz4 lz4-java 1.7.1
org.mariadb.jdbc mariadb-java-client 2.1.2
org.objenesis objenesis 2.5.1
org.postgresql postgresql 42.1.4
org.roaringbitmap RoaringBitmap 0.7.45
org.roaringbitmap shims 0.7.45
org.rocksdb rocksdbjni 6.2.2
org.rosuda.REngine REngine 2.1.0
org.scala-lang scala-compiler_2.12 2.12.10
org.scala-lang scala-library_2.12 2.12.10
org.scala-lang scala-reflect_2.12 2.12.10
org.scala-lang.modules scala-collection-compat_2.12 2.1.1
org.scala-lang.modules scala-parser-combinators_2.12 1.1.2
org.scala-lang.modules scala-xml_2.12 1.2.0
org.scala-sbt test-interface 1.0
org.scalacheck scalacheck_2.12 1.14.2
org.scalactic scalactic_2.12 3.0.8
org.scalanlp breeze-macros_2.12 1.0
org.scalanlp breeze_2.12 1.0
org.scalatest scalatest_2.12 3.0.8
org.slf4j jcl-over-slf4j 1.7.30
org.slf4j jul-to-slf4j 1.7.30
org.slf4j slf4j-api 1.7.30
org.slf4j slf4j-log4j12 1.7.30
org.spark-project.spark unused 1.0.0
org.springframework spring-core 4.1.4.RELEASE
org.springframework spring-test 4.1.4.RELEASE
org.threeten threeten-extra 1.5.0
org.tukaani xz 1.5
org.typelevel algebra_2.12 2.0.0-M2
org.typelevel cats-kernel_2.12 2.0.0-M4
org.typelevel machinist_2.12 0.6.8
org.typelevel macro-compat_2.12 1.1.1
org.typelevel spire-macros_2.12 0.17.0-M1
org.typelevel spire-platform_2.12 0.17.0-M1
org.typelevel spire-util_2.12 0.17.0-M1
org.typelevel spire_2.12 0.17.0-M1
org.xerial sqlite-jdbc 3.8.11.2
org.xerial.snappy snappy-java 1.1.7.5
org.yaml snakeyaml 1.24
oro oro 2.0.8
pl.edu.icm JLargeArrays 1.5
software.amazon.ion ion-java 1.0.2
stax stax-api 1.0.1
xmlenc xmlenc 0.52