Databricks Runtime 7.5 (unsupported)
Databricks released this image in December 2020.
The following release notes provide information about Databricks Runtime 7.5, powered by Apache Spark 3.0.
New features
Delta Lake features and improvements
This release provides the following Delta Lake features and improvements:
- Asynchronous checkpointing eliminates streaming micro-batch duration spikes
- Latest table information is now propagated to catalogs
MERGE INTOoperation now supports schema evolution of nested columnsMERGE INTOoperation now automatically uses Optimized WritesMERGE INTOandUPDATEoperations now resolve nested struct columns by nameCLONEis GA and you can now override table propertiesRESTOREis GA- Scala implicits simplify Spark read and write APIs
- Improve streaming throughput with
fetchParallelismconnector option - Track stream progress with new Auto Loader metrics
Asynchronous checkpointing eliminates streaming micro-batch duration spikes
Delta Lake performs checkpoints every 10 commits by default. In streaming workloads, checkpointing can cause minor spikes in the micro-batch duration every 10 commits. Asynchronous checkpointing eliminates these spikes.
Asynchronous checkpointing is enabled by default for all non-Trigger Once streaming workloads. With asynchronous checkpointing, streaming writers also write enhanced checkpoints so you don't need to explicitly opt-in to enhanced checkpoints.
To disable asynchronous checkpointing, set the SQL configuration
spark.databricks.delta.checkpoint.async.enabled false.
Batch workloads continue to write checkpoints synchronously and we recommend against enabling asynchronous checkpointing for batch workloads.
Latest table information is now propagated to catalogs
Delta Lake now propagates the latest table information, such as schema and table properties, to catalogs such as the Hive metastore, on a best effort basis. Such information can help you leverage cataloging tools to understand the schema of your Delta tables.
Delta Lake does not internally depend on this information, but attempts to keep it up-to-date. In
addition, Delta Lake stores table properties delta.lastUpdateVersion and delta.lastCommitTimestamp
to denote the version and timestamp of the commit that last pushed a change to the metastore. The
metastore is updated only when the schema of the table or the properties in the table do not match.
Therefore, the aforementioned table properties do not necessarily reflect the latest version of the
table.
If your table information and the metastore information are out of sync, you can run a command such as DESCRIBE <table> to update the metastore.
Note
Because Hive doesn't allow the changing partition columns, but Delta tables can change their partitioning, Delta tables do not show up to as partitioned in Hive.
MERGE INTO operation now supports schema evolution of nested columns
Schema evolution of nested columns now have the same semantics as that of top-level columns. For example, new nested columns can be automatically added to a StructType column. See Automatic schema evolution in Merge for details.
MERGE INTO operation now automatically uses Optimized Writes
Optimized Writes can provide much better performance for MERGE INTO out-of-the-box, especially for MERGE INTO on partitioned tables. To disable Optimized Writes, set the Spark configuration spark.databricks.delta.optimizeWrite.enabled to false.
MERGE INTO and UPDATE operations now resolve nested struct columns by name
Update operations UPDATE and MERGE INTO commands now resolve nested struct columns by name. That is, when comparing or assigning columns of type StructType, the order of the nested columns does not matter (exactly in the same way as the order of top-level columns). To revert to resolving by position, set the Spark configuration spark.databricks.delta.resolveMergeUpdateStructsByName.enabled to false.
CLONE is GA and you can now override table properties
The CLONE command is now generally available. See CREATE TABLE CLONE. In addition, you can now specify table property overrides when using CLONE. For examples, see Clone a table on Azure Databricks.
RESTORE is GA
The RESTORE command is now generally available. See RESTORE.
Scala implicits simplify Spark read and write APIs
You can import io.delta.implicits._ to use the delta method with Spark read and write APIs.
import io.delta.implicits._
spark.read.delta("/my/table/path")
df.write.delta("/my/table/path")
spark.readStream.delta("/my/table/path")
df.writeStream.delta("/my/table/path")
Improve streaming throughput with fetchParallelism connector option
The AQS connector now supports a new option fetchParallelism to specify the parallelism to fetch messages from the queue service.
Track stream progress with new Auto Loader metrics
Auto Loader now reports how many files exist in the backlog and how large the backlog is after every batch. You can use these metrics to track stream progress. For details on how to view the metrics, see What is Auto Loader?.
Improvements
Wider support for notebook-scoped libraries using %pip and %conda magic commands
You can now install notebook-scoped libraries using %pip and %conda magic commands on High Concurrency clusters with either table ACLs or credential passthrough enabled.
Accurate Spark SQL DECIMAL display
The Spark SQL DECIMAL data type is now displayed in tables without loss of precision and scale.
ADLS Gen 2 storage connector based on Hadoop 3.3 for ABFS
An upgraded version of the ADLS Gen 2 storage connector is based on Hadoop 3.3 for ABFS. The upgraded connector contains many stability improvements (see Hadoop stability) and includes support for SAS token authentication.
Other fixes
The Databricks Kafka connector for Spark Streaming has been removed. Use the Apache Spark connector.
Library upgrades
- Upgraded Python libraries:
- koalas upgraded from 1.3.0 to 1.4.0.
- Upgraded several installed R libraries. See Installed R Libraries.
- OpenJDK 8 build upgraded to Zulu 8.50.0.51-CA-linux64 (build 1.8.0_275-b01).
Apache Spark
Databricks Runtime 7.5 includes Apache Spark 3.0.1. This release includes all Spark fixes and improvements included in Databricks Runtime 7.4 (unsupported), as well as the following additional bug fixes and improvements made to Spark:
- [SPARK-33611] [UI] Avoid encoding twice on the query parameter of rewritten proxy URL
- [SPARK-33587] [CORE] Kill the executor on nested fatal errors (7.x)
- [SPARK-33140] [SQL] Revert code that not use passed-in SparkSession to get SQLConf.
- [SPARK-33472] [SQL] Adjust RemoveRedundantSorts rule order
- [SPARK-33422] [DOC] Fix the correct display of left menu item
- [SPARK-27421] [SQL] Fix filter for int column and value class java.lang.String when pruning partition column
- [SPARK-33483] [INFRA][TESTS] Fix rat exclusion patterns and add a LICENSE
- [SPARK-33464] [INFRA] Add/remove (un)necessary cache and restructure GitHub Actions yaml
- [SPARK-33407] [PYTHON][7.X] Simplify the exception message from Python UDFs (disabled by default)
- [SPARK-33435] [SQL] DSv2: REFRESH TABLE should invalidate caches referencing the table
- [SPARK-33358] [SQL] Return code when command process failed
- [SPARK-33183] [DBCONNECT] Add RemoveRedundantSorts.scala to ALLOW_LIST.txt
- [SPARK-33290] [SQL] REFRESH TABLE should invalidate cache even though the table itself may not be cached
- [SPARK-33140] [SQL] Fix delta compile
- [SPARK-33316] [SQL] Add unit tests for from_avro and to_avro with Schema Registry and user provided nullable Avro schema
- [SPARK-33404] [SQL] Fix incorrect results in
date_truncexpression - [SPARK-33339] [PYTHON] Pyspark application will hang due to non Exception error
- [SPARK-33412] [SQL] OverwriteByExpression should resolve its delete condition based on the table relation not the input query
- [SPARK-33391] [SQL] element_at with CreateArray not respect one based index.
- [SPARK-33316] [SQL] Support user provided nullable Avro schema for non-nullable catalyst schema in Avro writing
- [SPARK-33372] [SQL] Fix InSet bucket pruning
- [SPARK-33371] [PYTHON] Update setup.py and tests for Python 3.9
- [SPARK-33140] [SQL] remove SQLConf and SparkSession in all sub-class of Rule[QueryPlan]
- [SPARK-33338] [SQL] GROUP BY using literal map should not fail
- [SPARK-33306] [SQL]Timezone is needed when cast date to string
- [SPARK-33284] [WEB-UI] In the Storage UI page, clicking any field to sort the table will cause the header content to be lost
- [SPARK-33362] [SQL] skipSchemaResolution should still require query to be resolved
- [SPARK-32257] [SQL] Reports explicit errors for invalid usage of SET/RESET command
- [SPARK-32406] [SQL] Make RESET syntax support single configuration reset
- [SPARK-32376] [SQL] Make unionByName null-filling behavior work with struct columns
- [SPARK-32308] [SQL] Move by-name resolution logic of unionByName from API code to analysis phase
- [SPARK-20044] [UI] Support Spark UI behind front-end reverse proxy using a path prefix Revert proxy url
- [SPARK-33268] [SQL][PYTHON] Fix bugs for casting data from/to PythonUserDefinedType
- [SPARK-33292] [SQL] Make Literal ArrayBasedMapData string representation disambiguous
- [SPARK-33183] [SQL] Fix Optimizer rule EliminateSorts and add a physical rule to remove redundant sorts
- [SPARK-33272] [SQL] prune the attributes mapping in QueryPlan.transformUpWithNewOutput
- [SPARK-32090] [SQL] Improve UserDefinedType.equal() to make it be symmetrical
- [SPARK-33267] [SQL] Fix NPE issue on 'In' filter when one of values contains null
- [SPARK-33208] [SQL] Update the document of SparkSession#sql
- [SPARK-33260] [SQL] Fix incorrect results from SortExec when sortOrder is Stream
- [SPARK-33230] [SQL] Hadoop committers to get unique job ID in "spark.sql.sources.writeJobUUID"
- [SPARK-33197] [SQL] Make changes to spark.sql.analyzer.maxIterations take effect at runtime
- [SPARK-33228] [SQL] Don't uncache data when replacing a view having the same logical plan
- [SPARK-32557] [CORE] Logging and swallowing the exception per entry in History server
- [SPARK-32436] [CORE] Initialize numNonEmptyBlocks in HighlyCompressedMapStatus.readExternal
- [SPARK-33131] [SQL] Fix grouping sets with having clause can not resolve qualified col name
- [SPARK-32761] [SQL] Allow aggregating multiple foldable distinct expressions
- [SPARK-33094] [SQL] Make ORC format propagate Hadoop config from DS options to underlying HDFS file system
Maintenance updates
See Databricks Runtime 7.5 maintenance updates.
System environment
- Operating System: Ubuntu 18.04.5 LTS
- Java: Zulu 8.50.0.51-CA-linux64 (build 1.8.0_275-b01)
- Scala: 2.12.10
- Python: 3.7.5
- R: R version 3.6.3 (2020-02-29)
- Delta Lake 0.7.0
Installed Python libraries
| Library | Version | Library | Version | Library | Version |
|---|---|---|---|---|---|
| asn1crypto | 1.3.0 | backcall | 0.1.0 | boto3 | 1.12.0 |
| botocore | 1.15.0 | certifi | 2020.6.20 | cffi | 1.14.0 |
| chardet | 3.0.4 | cryptography | 2.8 | cycler | 0.10.0 |
| Cython | 0.29.15 | decorator | 4.4.1 | docutils | 0.15.2 |
| entrypoints | 0.3 | idna | 2.8 | ipykernel | 5.1.4 |
| ipython | 7.12.0 | ipython-genutils | 0.2.0 | jedi | 0.17.2 |
| jmespath | 0.10.0 | joblib | 0.14.1 | jupyter-client | 5.3.4 |
| jupyter-core | 4.6.1 | kiwisolver | 1.1.0 | koalas | 1.4.0 |
| matplotlib | 3.1.3 | numpy | 1.18.1 | pandas | 1.0.1 |
| parso | 0.7.0 | patsy | 0.5.1 | pexpect | 4.8.0 |
| pickleshare | 0.7.5 | pip | 20.0.2 | prompt-toolkit | 3.0.3 |
| psycopg2 | 2.8.4 | ptyprocess | 0.6.0 | pyarrow | 1.0.1 |
| pycparser | 2.19 | Pygments | 2.5.2 | PyGObject | 3.26.1 |
| pyOpenSSL | 19.1.0 | pyparsing | 2.4.6 | PySocks | 1.7.1 |
| python-apt | 1.6.5+ubuntu0.3 | python-dateutil | 2.8.1 | pytz | 2019.3 |
| pyzmq | 18.1.1 | requests | 2.22.0 | s3transfer | 0.3.3 |
| scikit-learn | 0.22.1 | scipy | 1.4.1 | seaborn | 0.10.0 |
| setuptools | 45.2.0 | six | 1.14.0 | ssh-import-id | 5.7 |
| statsmodels | 0.11.0 | tornado | 6.0.3 | traitlets | 4.3.3 |
| unattended-upgrades | 0.1 | urllib3 | 1.25.8 | virtualenv | 16.7.10 |
| wcwidth | 0.1.8 | wheel | 0.34.2 |
Installed R libraries
R libraries are installed from the Microsoft CRAN snapshot on 2020-11-02.
| Library | Version | Library | Version | Library | Version |
|---|---|---|---|---|---|
| askpass | 1.1 | assertthat | 0.2.1 | backports | 1.1.10 |
| base | 3.6.3 | base64enc | 0.1-3 | BH | 1.72.0-3 |
| bit | 4.0.4 | bit64 | 4.0.5 | blob | 1.2.1 |
| boot | 1.3-25 | brew | 1.0-6 | brio | 1.1.0 |
| broom | 0.7.2 | callr | 3.5.1 | caret | 6.0-86 |
| cellranger | 1.1.0 | chron | 2.3-56 | class | 7.3-17 |
| cli | 2.1.0 | clipr | 0.7.1 | cluster | 2.1.0 |
| codetools | 0.2-18 | colorspace | 1.4-1 | commonmark | 1.7 |
| compiler | 3.6.3 | config | 0.3 | covr | 3.5.1 |
| cpp11 | 0.2.3 | crayon | 1.3.4 | crosstalk | 1.1.0.1 |
| curl | 4.3 | data.table | 1.13.2 | datasets | 3.6.3 |
| DBI | 1.1.0 | dbplyr | 1.4.4 | desc | 1.2.0 |
| devtools | 2.3.2 | diffobj | 0.3.2 | digest | 0.6.27 |
| dplyr | 0.8.5 | DT | 0.16 | ellipsis | 0.3.1 |
| evaluate | 0.14 | fansi | 0.4.1 | farver | 2.0.3 |
| fastmap | 1.0.1 | forcats | 0.5.0 | foreach | 1.5.1 |
| foreign | 0.8-76 | forge | 0.2.0 | fs | 1.5.0 |
| generics | 0.1.0 | ggplot2 | 3.3.2 | gh | 1.1.0 |
| git2r | 0.27.1 | glmnet | 3.0-2 | globals | 0.13.1 |
| glue | 1.4.2 | gower | 0.2.2 | graphics | 3.6.3 |
| grDevices | 3.6.3 | grid | 3.6.3 | gridExtra | 2.3 |
| gsubfn | 0.7 | gtable | 0.3.0 | haven | 2.3.1 |
| highr | 0.8 | hms | 0.5.3 | htmltools | 0.5.0 |
| htmlwidgets | 1.5.2 | httpuv | 1.5.4 | httr | 1.4.2 |
| hwriter | 1.3.2 | hwriterPlus | 1.0-3 | ini | 0.3.1 |
| ipred | 0.9-9 | isoband | 0.2.2 | iterators | 1.0.13 |
| jsonlite | 1.7.1 | KernSmooth | 2.23-18 | knitr | 1.30 |
| labeling | 0.4.2 | later | 1.1.0.1 | lattice | 0.20-41 |
| lava | 1.6.8 | lazyeval | 0.2.2 | lifecycle | 0.2.0 |
| lubridate | 1.7.9 | magrittr | 1.5 | markdown | 1.1 |
| MASS | 7.3-53 | Matrix | 1.2-18 | memoise | 1.1.0 |
| methods | 3.6.3 | mgcv | 1.8-33 | mime | 0.9 |
| ModelMetrics | 1.2.2.2 | modelr | 0.1.8 | munsell | 0.5.0 |
| nlme | 3.1-150 | nnet | 7.3-14 | numDeriv | 2016.8-1.1 |
| openssl | 1.4.3 | parallel | 3.6.3 | pillar | 1.4.6 |
| pkgbuild | 1.1.0 | pkgconfig | 2.0.3 | pkgload | 1.1.0 |
| plogr | 0.2.0 | plyr | 1.8.6 | praise | 1.0.0 |
| prettyunits | 1.1.1 | pROC | 1.16.2 | processx | 3.4.4 |
| prodlim | 2019.11.13 | progress | 1.2.2 | promises | 1.1.1 |
| proto | 1.0.0 | ps | 1.4.0 | purrr | 0.3.4 |
| r2d3 | 0.2.3 | R6 | 2.5.0 | randomForest | 4.6-14 |
| rappdirs | 0.3.1 | rcmdcheck | 1.3.3 | RColorBrewer | 1.1-2 |
| Rcpp | 1.0.5 | readr | 1.4.0 | readxl | 1.3.1 |
| recipes | 0.1.14 | rematch | 1.0.1 | rematch2 | 2.1.2 |
| remotes | 2.2.0 | reprex | 0.3.0 | reshape2 | 1.4.4 |
| rex | 1.2.0 | rjson | 0.2.20 | rlang | 0.4.8 |
| rmarkdown | 2.5 | RODBC | 1.3-16 | roxygen2 | 7.1.1 |
| rpart | 4.1-15 | rprojroot | 1.3-2 | Rserve | 1.8-7 |
| RSQLite | 2.2.1 | rstudioapi | 0.11 | rversions | 2.0.2 |
| rvest | 0.3.6 | scales | 1.1.1 | selectr | 0.4-2 |
| sessioninfo | 1.1.1 | shape | 1.4.4 | shiny | 1.5.0 |
| sourcetools | 0.1.7 | sparklyr | 1.4.0 | SparkR | 3.0.0 |
| spatial | 7.3-11 | splines | 3.6.3 | sqldf | 0.4-11 |
| SQUAREM | 2020.5 | stats | 3.6.3 | stats4 | 3.6.3 |
| stringi | 1.5.3 | stringr | 1.4.0 | survival | 3.2-7 |
| sys | 3.4 | tcltk | 3.6.3 | TeachingDemos | 2.10 |
| testthat | 3.0.0 | tibble | 3.0.4 | tidyr | 1.1.2 |
| tidyselect | 1.1.0 | tidyverse | 1.3.0 | timeDate | 3043.102 |
| tinytex | 0.27 | tools | 3.6.3 | usethis | 1.6.3 |
| utf8 | 1.1.4 | utils | 3.6.3 | uuid | 0.1-4 |
| vctrs | 0.3.4 | viridisLite | 0.3.0 | waldo | 0.2.2 |
| whisker | 0.4 | withr | 2.3.0 | xfun | 0.19 |
| xml2 | 1.3.2 | xopen | 1.0.0 | xtable | 1.8-4 |
| yaml | 2.2.1 |
Installed Java and Scala libraries (Scala 2.12 cluster version)
| Group ID | Artifact ID | Version |
|---|---|---|
| antlr | antlr | 2.7.7 |
| com.amazonaws | amazon-kinesis-client | 1.12.0 |
| com.amazonaws | aws-java-sdk-autoscaling | 1.11.655 |
| com.amazonaws | aws-java-sdk-cloudformation | 1.11.655 |
| com.amazonaws | aws-java-sdk-cloudfront | 1.11.655 |
| com.amazonaws | aws-java-sdk-cloudhsm | 1.11.655 |
| com.amazonaws | aws-java-sdk-cloudsearch | 1.11.655 |
| com.amazonaws | aws-java-sdk-cloudtrail | 1.11.655 |
| com.amazonaws | aws-java-sdk-cloudwatch | 1.11.655 |
| com.amazonaws | aws-java-sdk-cloudwatchmetrics | 1.11.655 |
| com.amazonaws | aws-java-sdk-codedeploy | 1.11.655 |
| com.amazonaws | aws-java-sdk-cognitoidentity | 1.11.655 |
| com.amazonaws | aws-java-sdk-cognitosync | 1.11.655 |
| com.amazonaws | aws-java-sdk-config | 1.11.655 |
| com.amazonaws | aws-java-sdk-core | 1.11.655 |
| com.amazonaws | aws-java-sdk-datapipeline | 1.11.655 |
| com.amazonaws | aws-java-sdk-directconnect | 1.11.655 |
| com.amazonaws | aws-java-sdk-directory | 1.11.655 |
| com.amazonaws | aws-java-sdk-dynamodb | 1.11.655 |
| com.amazonaws | aws-java-sdk-ec2 | 1.11.655 |
| com.amazonaws | aws-java-sdk-ecs | 1.11.655 |
| com.amazonaws | aws-java-sdk-efs | 1.11.655 |
| com.amazonaws | aws-java-sdk-elasticache | 1.11.655 |
| com.amazonaws | aws-java-sdk-elasticbeanstalk | 1.11.655 |
| com.amazonaws | aws-java-sdk-elasticloadbalancing | 1.11.655 |
| com.amazonaws | aws-java-sdk-elastictranscoder | 1.11.655 |
| com.amazonaws | aws-java-sdk-emr | 1.11.655 |
| com.amazonaws | aws-java-sdk-glacier | 1.11.655 |
| com.amazonaws | aws-java-sdk-iam | 1.11.655 |
| com.amazonaws | aws-java-sdk-importexport | 1.11.655 |
| com.amazonaws | aws-java-sdk-kinesis | 1.11.655 |
| com.amazonaws | aws-java-sdk-kms | 1.11.655 |
| com.amazonaws | aws-java-sdk-lambda | 1.11.655 |
| com.amazonaws | aws-java-sdk-logs | 1.11.655 |
| com.amazonaws | aws-java-sdk-machinelearning | 1.11.655 |
| com.amazonaws | aws-java-sdk-opsworks | 1.11.655 |
| com.amazonaws | aws-java-sdk-rds | 1.11.655 |
| com.amazonaws | aws-java-sdk-redshift | 1.11.655 |
| com.amazonaws | aws-java-sdk-route53 | 1.11.655 |
| com.amazonaws | aws-java-sdk-s3 | 1.11.655 |
| com.amazonaws | aws-java-sdk-ses | 1.11.655 |
| com.amazonaws | aws-java-sdk-simpledb | 1.11.655 |
| com.amazonaws | aws-java-sdk-simpleworkflow | 1.11.655 |
| com.amazonaws | aws-java-sdk-sns | 1.11.655 |
| com.amazonaws | aws-java-sdk-sqs | 1.11.655 |
| com.amazonaws | aws-java-sdk-ssm | 1.11.655 |
| com.amazonaws | aws-java-sdk-storagegateway | 1.11.655 |
| com.amazonaws | aws-java-sdk-sts | 1.11.655 |
| com.amazonaws | aws-java-sdk-support | 1.11.655 |
| com.amazonaws | aws-java-sdk-swf-libraries | 1.11.22 |
| com.amazonaws | aws-java-sdk-workspaces | 1.11.655 |
| com.amazonaws | jmespath-java | 1.11.655 |
| com.chuusai | shapeless_2.12 | 2.3.3 |
| com.clearspring.analytics | stream | 2.9.6 |
| com.databricks | Rserve | 1.8-3 |
| com.databricks | jets3t | 0.7.1-0 |
| com.databricks.scalapb | compilerplugin_2.12 | 0.4.15-10 |
| com.databricks.scalapb | scalapb-runtime_2.12 | 0.4.15-10 |
| com.esotericsoftware | kryo-shaded | 4.0.2 |
| com.esotericsoftware | minlog | 1.3.0 |
| com.fasterxml | classmate | 1.3.4 |
| com.fasterxml.jackson.core | jackson-annotations | 2.10.0 |
| com.fasterxml.jackson.core | jackson-core | 2.10.0 |
| com.fasterxml.jackson.core | jackson-databind | 2.10.0 |
| com.fasterxml.jackson.dataformat | jackson-dataformat-cbor | 2.10.0 |
| com.fasterxml.jackson.datatype | jackson-datatype-joda | 2.10.0 |
| com.fasterxml.jackson.module | jackson-module-paranamer | 2.10.0 |
| com.fasterxml.jackson.module | jackson-module-scala_2.12 | 2.10.0 |
| com.github.ben-manes.caffeine | caffeine | 2.3.4 |
| com.github.fommil | jniloader | 1.1 |
| com.github.fommil.netlib | core | 1.1.2 |
| com.github.fommil.netlib | native_ref-java | 1.1 |
| com.github.fommil.netlib | native_ref-java-natives | 1.1 |
| com.github.fommil.netlib | native_system-java | 1.1 |
| com.github.fommil.netlib | native_system-java-natives | 1.1 |
| com.github.fommil.netlib | netlib-native_ref-linux-x86_64-natives | 1.1 |
| com.github.fommil.netlib | netlib-native_system-linux-x86_64-natives | 1.1 |
| com.github.joshelser | dropwizard-metrics-hadoop-metrics2-reporter | 0.1.2 |
| com.github.luben | zstd-jni | 1.4.4-3 |
| com.github.wendykierp | JTransforms | 3.1 |
| com.google.code.findbugs | jsr305 | 3.0.0 |
| com.google.code.gson | gson | 2.2.4 |
| com.google.flatbuffers | flatbuffers-java | 1.9.0 |
| com.google.guava | guava | 15.0 |
| com.google.protobuf | protobuf-java | 2.6.1 |
| com.h2database | h2 | 1.4.195 |
| com.helger | profiler | 1.1.1 |
| com.jcraft | jsch | 0.1.50 |
| com.jolbox | bonecp | 0.8.0.RELEASE |
| com.lihaoyi | sourcecode_2.12 | 0.1.9 |
| com.microsoft.sqlserver | mssql-jdbc | 8.2.1.jre8 |
| com.microsoft.azure | azure-data-lake-store-sdk | 2.3.8 |
| com.ning | compress-lzf | 1.0.3 |
| com.sun.mail | javax.mail | 1.5.2 |
| com.tdunning | json | 1.8 |
| com.thoughtworks.paranamer | paranamer | 2.8 |
| com.trueaccord.lenses | lenses_2.12 | 0.4.12 |
| com.twitter | chill-java | 0.9.5 |
| com.twitter | chill_2.12 | 0.9.5 |
| com.twitter | util-app_2.12 | 7.1.0 |
| com.twitter | util-core_2.12 | 7.1.0 |
| com.twitter | util-function_2.12 | 7.1.0 |
| com.twitter | util-jvm_2.12 | 7.1.0 |
| com.twitter | util-lint_2.12 | 7.1.0 |
| com.twitter | util-registry_2.12 | 7.1.0 |
| com.twitter | util-stats_2.12 | 7.1.0 |
| com.typesafe | config | 1.2.1 |
| com.typesafe.scala-logging | scala-logging_2.12 | 3.7.2 |
| com.univocity | univocity-parsers | 2.9.0 |
| com.zaxxer | HikariCP | 3.1.0 |
| commons-beanutils | commons-beanutils | 1.9.4 |
| commons-cli | commons-cli | 1.2 |
| commons-codec | commons-codec | 1.10 |
| commons-collections | commons-collections | 3.2.2 |
| commons-configuration | commons-configuration | 1.6 |
| commons-dbcp | commons-dbcp | 1.4 |
| commons-digester | commons-digester | 1.8 |
| commons-fileupload | commons-fileupload | 1.3.3 |
| commons-httpclient | commons-httpclient | 3.1 |
| commons-io | commons-io | 2.4 |
| commons-lang | commons-lang | 2.6 |
| commons-logging | commons-logging | 1.1.3 |
| commons-net | commons-net | 3.1 |
| commons-pool | commons-pool | 1.5.4 |
| info.ganglia.gmetric4j | gmetric4j | 1.0.10 |
| io.airlift | aircompressor | 0.10 |
| io.dropwizard.metrics | metrics-core | 4.1.1 |
| io.dropwizard.metrics | metrics-graphite | 4.1.1 |
| io.dropwizard.metrics | metrics-healthchecks | 4.1.1 |
| io.dropwizard.metrics | metrics-jetty9 | 4.1.1 |
| io.dropwizard.metrics | metrics-jmx | 4.1.1 |
| io.dropwizard.metrics | metrics-json | 4.1.1 |
| io.dropwizard.metrics | metrics-jvm | 4.1.1 |
| io.dropwizard.metrics | metrics-servlets | 4.1.1 |
| io.netty | netty-all | 4.1.47.Final |
| io.prometheus | simpleclient | 0.7.0 |
| io.prometheus | simpleclient_common | 0.7.0 |
| io.prometheus | simpleclient_dropwizard | 0.7.0 |
| io.prometheus | simpleclient_pushgateway | 0.7.0 |
| io.prometheus | simpleclient_servlet | 0.7.0 |
| io.prometheus.jmx | collector | 0.12.0 |
| jakarta.annotation | jakarta.annotation-api | 1.3.5 |
| jakarta.validation | jakarta.validation-api | 2.0.2 |
| jakarta.ws.rs | jakarta.ws.rs-api | 2.1.6 |
| javax.activation | activation | 1.1.1 |
| javax.el | javax.el-api | 2.2.4 |
| javax.jdo | jdo-api | 3.0.1 |
| javax.servlet | javax.servlet-api | 3.1.0 |
| javax.servlet.jsp | jsp-api | 2.1 |
| javax.transaction | jta | 1.1 |
| javax.transaction | transaction-api | 1.1 |
| javax.xml.bind | jaxb-api | 2.2.2 |
| javax.xml.stream | stax-api | 1.0-2 |
| javolution | javolution | 5.5.1 |
| jline | jline | 2.14.6 |
| joda-time | joda-time | 2.10.5 |
| log4j | apache-log4j-extras | 1.2.17 |
| log4j | log4j | 1.2.17 |
| net.razorvine | pyrolite | 4.30 |
| net.sf.jpam | jpam | 1.1 |
| net.sf.opencsv | opencsv | 2.3 |
| net.sf.supercsv | super-csv | 2.2.0 |
| net.snowflake | snowflake-ingest-sdk | 0.9.6 |
| net.snowflake | snowflake-jdbc | 3.12.8 |
| net.snowflake | spark-snowflake_2.12 | 2.8.1-spark_3.0 |
| net.sourceforge.f2j | arpack_combined_all | 0.1 |
| org.acplt.remotetea | remotetea-oncrpc | 1.1.2 |
| org.antlr | ST4 | 4.0.4 |
| org.antlr | antlr-runtime | 3.5.2 |
| org.antlr | antlr4-runtime | 4.7.1 |
| org.antlr | stringtemplate | 3.2.1 |
| org.apache.ant | ant | 1.9.2 |
| org.apache.ant | ant-jsch | 1.9.2 |
| org.apache.ant | ant-launcher | 1.9.2 |
| org.apache.arrow | arrow-format | 0.15.1 |
| org.apache.arrow | arrow-memory | 0.15.1 |
| org.apache.arrow | arrow-vector | 0.15.1 |
| org.apache.avro | avro | 1.8.2 |
| org.apache.avro | avro-ipc | 1.8.2 |
| org.apache.avro | avro-mapred-hadoop2 | 1.8.2 |
| org.apache.commons | commons-compress | 1.8.1 |
| org.apache.commons | commons-crypto | 1.0.0 |
| org.apache.commons | commons-lang3 | 3.9 |
| org.apache.commons | commons-math3 | 3.4.1 |
| org.apache.commons | commons-text | 1.6 |
| org.apache.curator | curator-client | 2.7.1 |
| org.apache.curator | curator-framework | 2.7.1 |
| org.apache.curator | curator-recipes | 2.7.1 |
| org.apache.derby | derby | 10.12.1.1 |
| org.apache.directory.api | api-asn1-api | 1.0.0-M20 |
| org.apache.directory.api | api-util | 1.0.0-M20 |
| org.apache.directory.server | apacheds-i18n | 2.0.0-M15 |
| org.apache.directory.server | apacheds-kerberos-codec | 2.0.0-M15 |
| org.apache.hadoop | hadoop-annotations | 2.7.4 |
| org.apache.hadoop | hadoop-auth | 2.7.4 |
| org.apache.hadoop | hadoop-client | 2.7.4 |
| org.apache.hadoop | hadoop-common | 2.7.4 |
| org.apache.hadoop | hadoop-hdfs | 2.7.4 |
| org.apache.hadoop | hadoop-mapreduce-client-app | 2.7.4 |
| org.apache.hadoop | hadoop-mapreduce-client-common | 2.7.4 |
| org.apache.hadoop | hadoop-mapreduce-client-core | 2.7.4 |
| org.apache.hadoop | hadoop-mapreduce-client-jobclient | 2.7.4 |
| org.apache.hadoop | hadoop-mapreduce-client-shuffle | 2.7.4 |
| org.apache.hadoop | hadoop-yarn-api | 2.7.4 |
| org.apache.hadoop | hadoop-yarn-client | 2.7.4 |
| org.apache.hadoop | hadoop-yarn-common | 2.7.4 |
| org.apache.hadoop | hadoop-yarn-server-common | 2.7.4 |
| org.apache.hive | hive-beeline | 2.3.7 |
| org.apache.hive | hive-cli | 2.3.7 |
| org.apache.hive | hive-common | 2.3.7 |
| org.apache.hive | hive-exec-core | 2.3.7 |
| org.apache.hive | hive-jdbc | 2.3.7 |
| org.apache.hive | hive-llap-client | 2.3.7 |
| org.apache.hive | hive-llap-common | 2.3.7 |
| org.apache.hive | hive-metastore | 2.3.7 |
| org.apache.hive | hive-serde | 2.3.7 |
| org.apache.hive | hive-shims | 2.3.7 |
| org.apache.hive | hive-storage-api | 2.7.1 |
| org.apache.hive | hive-vector-code-gen | 2.3.7 |
| org.apache.hive.shims | hive-shims-0.23 | 2.3.7 |
| org.apache.hive.shims | hive-shims-common | 2.3.7 |
| org.apache.hive.shims | hive-shims-scheduler | 2.3.7 |
| org.apache.htrace | htrace-core | 3.1.0-incubating |
| org.apache.httpcomponents | httpclient | 4.5.6 |
| org.apache.httpcomponents | httpcore | 4.4.12 |
| org.apache.ivy | ivy | 2.4.0 |
| org.apache.mesos | mesos-shaded-protobuf | 1.4.0 |
| org.apache.orc | orc-core | 1.5.10 |
| org.apache.orc | orc-mapreduce | 1.5.10 |
| org.apache.orc | orc-shims | 1.5.10 |
| org.apache.parquet | parquet-column | 1.10.1-databricks6 |
| org.apache.parquet | parquet-common | 1.10.1-databricks6 |
| org.apache.parquet | parquet-encoding | 1.10.1-databricks6 |
| org.apache.parquet | parquet-format | 2.4.0 |
| org.apache.parquet | parquet-hadoop | 1.10.1-databricks6 |
| org.apache.parquet | parquet-jackson | 1.10.1-databricks6 |
| org.apache.thrift | libfb303 | 0.9.3 |
| org.apache.thrift | libthrift | 0.12.0 |
| org.apache.velocity | velocity | 1.5 |
| org.apache.xbean | xbean-asm7-shaded | 4.15 |
| org.apache.yetus | audience-annotations | 0.5.0 |
| org.apache.zookeeper | zookeeper | 3.4.14 |
| org.codehaus.jackson | jackson-core-asl | 1.9.13 |
| org.codehaus.jackson | jackson-jaxrs | 1.9.13 |
| org.codehaus.jackson | jackson-mapper-asl | 1.9.13 |
| org.codehaus.jackson | jackson-xc | 1.9.13 |
| org.codehaus.janino | commons-compiler | 3.0.16 |
| org.codehaus.janino | janino | 3.0.16 |
| org.datanucleus | datanucleus-api-jdo | 4.2.4 |
| org.datanucleus | datanucleus-core | 4.1.17 |
| org.datanucleus | datanucleus-rdbms | 4.1.19 |
| org.datanucleus | javax.jdo | 3.2.0-m3 |
| org.eclipse.jetty | jetty-client | 9.4.18.v20190429 |
| org.eclipse.jetty | jetty-continuation | 9.4.18.v20190429 |
| org.eclipse.jetty | jetty-http | 9.4.18.v20190429 |
| org.eclipse.jetty | jetty-io | 9.4.18.v20190429 |
| org.eclipse.jetty | jetty-jndi | 9.4.18.v20190429 |
| org.eclipse.jetty | jetty-plus | 9.4.18.v20190429 |
| org.eclipse.jetty | jetty-proxy | 9.4.18.v20190429 |
| org.eclipse.jetty | jetty-security | 9.4.18.v20190429 |
| org.eclipse.jetty | jetty-server | 9.4.18.v20190429 |
| org.eclipse.jetty | jetty-servlet | 9.4.18.v20190429 |
| org.eclipse.jetty | jetty-servlets | 9.4.18.v20190429 |
| org.eclipse.jetty | jetty-util | 9.4.18.v20190429 |
| org.eclipse.jetty | jetty-webapp | 9.4.18.v20190429 |
| org.eclipse.jetty | jetty-xml | 9.4.18.v20190429 |
| org.fusesource.leveldbjni | leveldbjni-all | 1.8 |
| org.glassfish.hk2 | hk2-api | 2.6.1 |
| org.glassfish.hk2 | hk2-locator | 2.6.1 |
| org.glassfish.hk2 | hk2-utils | 2.6.1 |
| org.glassfish.hk2 | osgi-resource-locator | 1.0.3 |
| org.glassfish.hk2.external | aopalliance-repackaged | 2.6.1 |
| org.glassfish.hk2.external | jakarta.inject | 2.6.1 |
| org.glassfish.jersey.containers | jersey-container-servlet | 2.30 |
| org.glassfish.jersey.containers | jersey-container-servlet-core | 2.30 |
| org.glassfish.jersey.core | jersey-client | 2.30 |
| org.glassfish.jersey.core | jersey-common | 2.30 |
| org.glassfish.jersey.core | jersey-server | 2.30 |
| org.glassfish.jersey.inject | jersey-hk2 | 2.30 |
| org.glassfish.jersey.media | jersey-media-jaxb | 2.30 |
| org.hibernate.validator | hibernate-validator | 6.1.0.Final |
| org.javassist | javassist | 3.25.0-GA |
| org.jboss.logging | jboss-logging | 3.3.2.Final |
| org.jdbi | jdbi | 2.63.1 |
| org.joda | joda-convert | 1.7 |
| org.jodd | jodd-core | 3.5.2 |
| org.json4s | json4s-ast_2.12 | 3.6.6 |
| org.json4s | json4s-core_2.12 | 3.6.6 |
| org.json4s | json4s-jackson_2.12 | 3.6.6 |
| org.json4s | json4s-scalap_2.12 | 3.6.6 |
| org.lz4 | lz4-java | 1.7.1 |
| org.mariadb.jdbc | mariadb-java-client | 2.1.2 |
| org.objenesis | objenesis | 2.5.1 |
| org.postgresql | postgresql | 42.1.4 |
| org.roaringbitmap | RoaringBitmap | 0.7.45 |
| org.roaringbitmap | shims | 0.7.45 |
| org.rocksdb | rocksdbjni | 6.2.2 |
| org.rosuda.REngine | REngine | 2.1.0 |
| org.scala-lang | scala-compiler_2.12 | 2.12.10 |
| org.scala-lang | scala-library_2.12 | 2.12.10 |
| org.scala-lang | scala-reflect_2.12 | 2.12.10 |
| org.scala-lang.modules | scala-collection-compat_2.12 | 2.1.1 |
| org.scala-lang.modules | scala-parser-combinators_2.12 | 1.1.2 |
| org.scala-lang.modules | scala-xml_2.12 | 1.2.0 |
| org.scala-sbt | test-interface | 1.0 |
| org.scalacheck | scalacheck_2.12 | 1.14.2 |
| org.scalactic | scalactic_2.12 | 3.0.8 |
| org.scalanlp | breeze-macros_2.12 | 1.0 |
| org.scalanlp | breeze_2.12 | 1.0 |
| org.scalatest | scalatest_2.12 | 3.0.8 |
| org.slf4j | jcl-over-slf4j | 1.7.30 |
| org.slf4j | jul-to-slf4j | 1.7.30 |
| org.slf4j | slf4j-api | 1.7.30 |
| org.slf4j | slf4j-log4j12 | 1.7.30 |
| org.spark-project.spark | unused | 1.0.0 |
| org.springframework | spring-core | 4.1.4.RELEASE |
| org.springframework | spring-test | 4.1.4.RELEASE |
| org.threeten | threeten-extra | 1.5.0 |
| org.tukaani | xz | 1.5 |
| org.typelevel | algebra_2.12 | 2.0.0-M2 |
| org.typelevel | cats-kernel_2.12 | 2.0.0-M4 |
| org.typelevel | machinist_2.12 | 0.6.8 |
| org.typelevel | macro-compat_2.12 | 1.1.1 |
| org.typelevel | spire-macros_2.12 | 0.17.0-M1 |
| org.typelevel | spire-platform_2.12 | 0.17.0-M1 |
| org.typelevel | spire-util_2.12 | 0.17.0-M1 |
| org.typelevel | spire_2.12 | 0.17.0-M1 |
| org.wildfly.openssl | wildfly-openssl | 1.0.7.Final |
| org.xerial | sqlite-jdbc | 3.8.11.2 |
| org.xerial.snappy | snappy-java | 1.1.7.5 |
| org.yaml | snakeyaml | 1.24 |
| oro | oro | 2.0.8 |
| pl.edu.icm | JLargeArrays | 1.5 |
| software.amazon.ion | ion-java | 1.0.2 |
| stax | stax-api | 1.0.1 |
| xmlenc | xmlenc | 0.52 |