Databricks Runtime 10.1 (EoS)
Note
Support for this Databricks Runtime version has ended. For the end-of-support date, see End-of-support history. For all supported Databricks Runtime versions, see Databricks Runtime release notes versions and compatibility.
The following release notes provide information about Databricks Runtime 10.1 and Databricks Runtime 10.1 Photon, powered by Apache Spark 3.2.0. Databricks released this version in November 2021. Photon is in Public Preview.
New features and improvements
- New Delta table settings for compacting small files
- New row group metrics for Parquet query scans
- Trigger.AvailableNow for Delta source streaming queries
- Trigger.AvailableNow for Auto Loader
- Delta support for Hadoop file system configurations that are passed through DataFrame options
- Asynchronous state checkpointing is now supported in stateful streaming jobs (Public Preview)
- Sparklyr with credential passthrough
- Databricks Utilities data summarize precision control (Public Preview)
- Azure Databricks notebooks can now be used with the IPython kernel
- New functions in Spark SQL
New Delta table settings for compacting small files
The Delta table property delta.autoOptimize.autoCompact now accepts the new values of auto and legacy in addition to the existing values of true and false. When set to auto (recommended), Auto Compaction uses better defaults, such as setting 32 MB as the target file size (although default behaviors are subject to change in the future). When set to legacy or true, Auto Compaction uses 128 MB as the target file size.
Instead of setting this property for an individual table, you can set the spark.databricks.delta.autoCompact.enabled configuration to auto, legacy, or true to enable Auto Compaction for all Delta tables. This configuration takes precedence over the related table property during the current session.
The target file size for Auto Compaction can be controlled with the spark.databricks.delta.autoCompact.maxFileSize configuration.
For more information about Auto Compaction, see Auto compaction for Delta Lake on Azure Databricks.
New row group metrics for Parquet query scans
Queries with Parquet scans now surface row group metrics in the Spark UI. These row group metrics are:
- total number of parquet row groups
- number of parquet row groups read
Trigger.AvailableNow for Delta source streaming queries
This change supports a new type of trigger for Delta source streaming queries in Scala: Trigger.AvailableNow. It is like Trigger.Once, which processes all available data then stops the query. However, Trigger.AvailableNow provides better scalability, because data can be processed in multiple batches instead of one. This change does not introduce a new interface; it implements an existing Spark API. This trigger also supports all of the rate limiting options that are already supported by Delta as a source.
Example:
spark.readStream
.format("delta")
.option("maxFilesPerTrigger", "1")
.load(inputDir)
.writeStream
.trigger(Trigger.AvailableNow)
.option("checkpointLocation", checkpointDir)
.start()
See coverage of Trigger.AvailableNow in What is Auto Loader?.
Trigger.AvailableNow for Auto Loader
Trigger.AvailableNow is a new streaming trigger mode similar to Trigger.Once. Trigger.Once processes all available data and then stops the query. Trigger.AvailableNow has better scalability, because data can be processed in multiple batches instead of one. Auto Loader now supports Trigger.AvailableNow in both directory listing and file notification modes for Scala. See coverage of Trigger.AvailableNow in What is Auto Loader?.
Delta support for Hadoop file system configurations that are passed through DataFrame options
Similar to Parquet, Delta now supports reading Hadoop file system configurations from DataFrameReader and DataFrameWriter options when the table is read or written, by using DataFrameReader.load(path) or DataFrameWriter.save(path). See Configure Delta storage credentials.
Asynchronous state checkpointing is now supported in stateful streaming jobs (Public Preview)
Asynchronous state checkpointing is a new feature which potentially reduces end-to-end microbatch latency in stateful streaming jobs with large state updates. See Asynchronous state checkpointing for stateful queries.
Sparklyr with credential passthrough
You can now use the sparklyr APIs to read and write data to and from cloud storage by using credential passthrough. This functionality is limited to standard clusters with single user access. See Access Azure Data Lake Storage using Microsoft Entra ID credential passthrough (legacy).
Databricks Utilities data summarize precision control (Public Preview)
A new precise parameter for dbutils.data.summarize controls the precision of summary results. When precise is set to false (the default), the summary returns approximate results for distinct count, percentile, and frequent item counts. When precise is set to true, summaries are now computed with exact distinct count and frequent item counts, with more precise percentile estimates. See summarize command (dbutils.data.summarize).
Azure Databricks notebooks can now be used with the IPython kernel
You can now configure an Azure Databricks cluster to use the IPython kernel to execute Python code. Using the IPython kernel on Azure Databricks adds support for IPython's display and output tooling. Also, the IPython kernel captures the stdout and stderr outputs of child processes created by a notebook, allowing that output to be included in the notebook's command results.
For more information, see IPython kernel.
New functions in Spark SQL
The following objects and commands have been added for working with data in Unity Catalog (Preview):
- External locations
- ALTER CATALOG
- ALTER CREDENTIAL
- ALTER LOCATION
- CREATE CATALOG
- CREATE LOCATION
- CREATE RECIPIENT
- DESCRIBE CATALOG
- DESCRIBE CREDENTIAL
- DESCRIBE LOCATION
- DROP CATALOG
- DROP CREDENTIAL
- DROP LOCATION
- LIST
- SHOW CATALOGS
- SHOW CREDENTIALS
- SHOW LOCATIONS
- USE CATALOG
The following commands have been added for Delta Sharing (Preview), which allows you to share read-only data with recipients outside your organization:
- Sharing
- ALTER SHARE
- CREATE SHARE
- DESCRIBE RECIPIENT
- DESCRIBE SHARE
- DROP RECIPIENT
- DROP SHARE
- GRANT SHARE
- REVOKE SHARE
- SHOW ALL IN SHARE
- SHOW RECIPIENTS
- SHOW SHARES
- SHOW GRANTS ON SHARE
- SHOW GRANTS TO RECIPIENT
Bug fixes
- Fixed a race condition that might cause a query failure with an
IOExceptionsuch asNo FileSystem for schemeor that might cause modifications tosparkContext.hadoopConfigurationto not take effect in queries.
Library upgrades
- Upgraded Python libraries:
- filelock from 3.0.12 to 3.3.1
- koalas from 1.8.1 to 1.8.2
- plotly from 5.1.0 to 5.3.0
- Upgraded R libraries:
- bslib from 0.3.0 to 0.3.1
- caret from 6.0-88 to 6.0-90
- cpp11 from 0.3.1 to 0.4.0
- data.table from 1.14.0 to 1.14.2
- desc from 1.3.0 to 1.4.0
- diffobj from 0.3.4 to 0.3.5
- digest from 0.6.27 to 0.6.28
- generics from 0.1.0 to 0.1.1
- hms from 1.1.0 to 1.1.1
- knitr from 1.34 to 1.36
- lattice from 0.20-44 to 0.20-45
- lifecycle from 1.0.0 to 1.0.1
- lubridate from 1.7.10 to 1.8.0
- mgcv from 1.8-37 to 1.8-38
- mime from 0.11 to 0.12
- pkgload from 1.2.2 to 1.2.3
- progressr from 0.8.0 to 0.9.0
- rcmdcheck from 1.3.3 to 1.4.0
- readr from 2.0.1 to 2.0.2
- recipes from 0.1.16 to 0.1.17
- remotes from 2.4.0 to 2.4.1
- rlang from 0.4.11 to 0.4.12
- rvest from 1.0.1 to 1.0.2
- shiny from 1.6.0 to 1.7.1
- stringi from 1.7.4 to 1.7.5
- testthat from 3.0.4 to 3.1.0
- tidyr from 1.1.3 to 1.1.4
- tinytex from 0.33 to 0.34
- usethis from 2.0.1 to 2.1.2
- xfun from 0.26 to 0.27
- Upgraded Java libraries:
- com.zaxxer.HikariCP from 3.1.0 to 4.0.3
Apache Spark
Databricks Runtime 10.1 includes Apache Spark 3.2.0. This release includes all Spark fixes and improvements included in Databricks Runtime 10.0 (EoS), as well as the following additional bug fixes and improvements made to Spark:
- [SPARK-37037] [SQL] Improve byte array sort by unify compareTo function of UTF8String and ByteArray
- [SPARK-37091] [R] SystemRequirements to include Java < 18
- [SPARK-37041] [SQL] Backport HIVE-15025: Secure-Socket-Layer (SSL) support for HMS
- [SPARK-37098] [SQL] Alter table properties should invalidate cache
- [SPARK-36992] [SQL] Improve byte array sort perf by unify getPrefix function of UTF8String and ByteArray
- [SPARK-37047] [SQL] Add lpad and rpad functions for binary strings
- [SPARK-37076] [SQL] Implement StructType.toString explicitly for Scala 2.13
- [SPARK-37075] [SQL] Move UDAF expression building from sql/catalyst to sql/core
- [SPARK-37084] [SQL] Set spark.sql.files.openCostInBytes to bytesConf
- [SPARK-37088] [PYSPARK][SQL] Writer thread must not access input after task completion listener returns
- [SPARK-37069] [SQL] Properly fallback when Hive.getWithoutRegisterFns is not available
- [SPARK-37050] [PYTHON] Update Conda installation instructions
- [SPARK-37067] [SQL] Use ZoneId.of() to handle timezone string in DatetimeUtils
- [SPARK-35925] [SQL] Support DayTimeIntervalType in width-bucket function
- [SPARK-35926] [SQL] Add support YearMonthIntervalType for width_bucket
- [SPARK-35973] [SQL] Add command
SHOW CATALOGS - [SPARK-36922] [SQL] The SIGN/SIGNUM functions should support ANSI intervals
- [SPARK-37078] [CORE] Support old 3-parameter Sink constructors
- [SPARK-37061] [SQL] Fix CustomMetrics when using Inner Classes
- [SPARK-35918] [AVRO] Unify schema mismatch handling for read/write and enhance error messages
- [SPARK-36871] [SQL]Migrate CreateViewStatement to v2 command
- [SPARK-36898] [SQL] Make the shuffle hash join factor configurable
- [SPARK-36921] [SQL] Support ANSI intervals by
DIV - [SPARK-36794] [SQL] Ignore duplicated join keys when building relation for SEMI/ANTI shuffled hash join
- [SPARK-36867] [SQL] Fix error message with GROUP BY alias
- [SPARK-36914] [SQL] Implement dropIndex and listIndexes in JDBC (MySQL dialect)
- [SPARK-35531] [SQL] Directly pass hive Table to HiveClient when call getPartitions to avoid unnecessary convert from HiveTable -> CatalogTable - -> HiveTable
- [SPARK-36718] [SQL] Only collapse projects if we don't duplicate expensive expressions
- [SPARK-37046] [SQL]: Alter view does not preserve column case
- [SPARK-36978] [SQL] InferConstraints rule should create IsNotNull constraints on the accessed nested field instead of the root nested type
- [SPARK-37052] [CORE] Spark should only pass -verbose argument to main class when is sql shell
- [SPARK-37017] [SQL] Reduce the scope of synchronized to prevent potential deadlock
- [SPARK-37032] [SQL] Fix broken SQL syntax link in SQL Reference page
- [SPARK-36905] [SQL] Fix reading hive views without explicit column names
- [SPARK-36678] Migrate SHOW TABLES to use V2 command by default
- [SPARK-36943] [SQL] Improve readability of missing column error message
- [SPARK-36980] [SQL] Insert support query with CTE
- [SPARK-37001] [SQL] Disable two level of map for final hash aggregation by default
- [SPARK-36970] [SQL] Manual disabled format
Bofdate_formatfunction to make Java 17 compatible with Java 8 - [SPARK-36546] [SQL] Add array support to union by name
- [SPARK-36979] [SQL] Add RewriteLateralSubquery rule into nonExcludableRules
- [SPARK-36949] [SQL] Disallow Hive provider tables with ANSI intervals
- [SPARK-36849] [SQL] Migrate UseStatement to v2 command framework
- [SPARK-36868] [SQL] Migrate CreateFunctionStatement to v2 command framework
- [SPARK-36841] [SQL] Add ansi syntax
set catalog xxxto change the current catalog - [SPARK-36821] [SQL] Make class ColumnarBatch extendable - addendum
- [SPARK-36963] [SQL] Add max_by/min_by to sql.functions
- [SPARK-36913] [SQL] Implement createIndex and IndexExists in DS V2 JDBC (MySQL dialect)
- [SPARK-36848] [SQL] Migrate ShowCurrentNamespaceStatement to v2 command framework
- [SPARK-36526] [SQL] DSV2 Index Support: Add supportsIndex interface
- [SPARK-36960] [SQL] Pushdown filters with ANSI interval values to ORC
- [SPARK-36929] [SQL] Remove Unused Method EliminateSubqueryAliasesSuite#assertEquivalent
- [SPARK-36931] [SQL] Support reading and writing ANSI intervals from/to ORC datasources
- [SPARK-34980] [SQL] Support coalesce partition through union in AQE
- [SPARK-36809] [SQL] Remove broadcast for InSubqueryExec used in DPP
- [SPARK-36813] [SQL][PYTHON] Propose an infrastructure of as-of join and imlement ps.merge_asof
- [SPARK-36918] [SQL] Ignore types when comparing structs for unionByName
- [SPARK-36891] [SQL] Refactor SpecificParquetRecordReaderBase and add more coverage on vectorized Parquet decoding
- [SPARK-36920] [SQL] Support ANSI intervals by
ABS() - [SPARK-36888] [SQL] add tests cases for sha2 function
- [SPARK-36889] [SQL] Respect
spark.sql.parquet.filterPushdownby v2 parquet scan builder - [SPARK-36830] [SQL] Support reading and writing ANSI intervals from/to JSON datasources
- [SPARK-36870] [SQL] Introduce INTERNAL_ERROR error class
- [SPARK-36831] [SQL] Support reading and writing ANSI intervals from/to CSV datasources.
- [SPARK-36550] [SQL] Propagation cause when UDF reflection fails
- [SPARK-36866] [SQL] Pushdown filters with ANSI interval values to parquet
- [SPARK-33832] [SQL] Force skew join code simplification and improvement
- [SPARK-36642] [SQL] Add df.withMetadata pyspark API
- [SPARK-35765] [SQL] Distinct aggs are not duplicate sensitive
- [SPARK-36825] [SQL] Read/write dataframes with ANSI intervals from/to parquet files
- [SPARK-36829] [SQL] Refactor NULL check for collectionOperators
- [SPARK-32712] [SQL] Support writing Hive bucketed table (Hive file formats with Hive hash)
- [SPARK-36797] [SQL] Union should resolve nested columns as top-level columns
- [SPARK-36838] [SQL] Improve InSet generated code performance
- [SPARK-36683] [SQL] Add new built-in SQL functions: SEC and CSC
- [SPARK-36721] [SQL] Simplify boolean equalities if one side is literal
- [SPARK-36760] [SQL] Add interface SupportsPushDownV2Filters
- [SPARK-36652] [SQL] AQE dynamic join selection should not apply to non-equi join
- [SPARK-36745] [SQL] ExtractEquiJoinKeys should return the original predicates on join keys
- [SPARK-36107] [SQL] Refactor first set of 20 query execution errors to use error classes
- [SPARK-32709] [SQL] Support writing Hive bucketed table (Parquet/ORC format with Hive hash)
- [SPARK-36735] [SQL] Adjust overhead of cached relation for DPP
- [SPARK-33832] [SQL] Support optimize skewed join even if introduce extra shuffle
- [SPARK-36822] [SQL] BroadcastNestedLoopJoinExec should use all condition instead of non-equi condition
- [SPARK-35221] [SQL] Add the check of supported join hints
- [SPARK-36814] [SQL] Make class ColumnarBatch extendable
- [SPARK-36663] [SQL] Support number-only column names in ORC data sources
- [SPARK-36751] [SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R
- [SPARK-36733] [SQL] Fix a perf issue in SchemaPruning when a struct has many fields
- [SPARK-36724] [SQL] Support timestamp_ntz as a type of time column for SessionWindow
- [SPARK-36556] [SQL] Add DSV2 filters
- [SPARK-36687] [SQL][CORE] Rename error classes with _ERROR suffix
- [SPARK-35803] [SQL] Support DataSource V2 CreateTempViewUsing
- [SPARK-36642] [SQL] Add df.withMetadata: a syntax suger to update the metadata of a dataframe
Maintenance updates
See Databricks Runtime 10.1 maintenance updates.
System environment
- Operating System: Ubuntu 20.04.3 LTS
- Java: Zulu 8.56.0.21-CA-linux64
- Scala: 2.12.14
- Python: 3.8.10
- R: 4.1.1
- Delta Lake: 1.0.0
Installed Python libraries
| Library | Version | Library | Version | Library | Version |
|---|---|---|---|---|---|
| Antergos Linux | 2015.10 (ISO-Rolling) | appdirs | 1.4.4 | argon2-cffi | 20.1.0 |
| async-generator | 1.10 | attrs | 20.3.0 | backcall | 0.2.0 |
| bleach | 3.3.0 | boto3 | 1.16.7 | botocore | 1.19.7 |
| certifi | 2020.12.5 | cffi | 1.14.5 | chardet | 4.0.0 |
| cycler | 0.10.0 | Cython | 0.29.23 | dbus-python | 1.2.16 |
| decorator | 5.0.6 | defusedxml | 0.7.1 | distlib | 0.3.3 |
| distro-info | 0.23ubuntu1 | entrypoints | 0.3 | facets-overview | 1.0.0 |
| filelock | 3.3.1 | idna | 2.10 | ipykernel | 5.3.4 |
| ipython | 7.22.0 | ipython-genutils | 0.2.0 | ipywidgets | 7.6.3 |
| jedi | 0.17.2 | Jinja2 | 2.11.3 | jmespath | 0.10.0 |
| joblib | 1.0.1 | jsonschema | 3.2.0 | jupyter-client | 6.1.12 |
| jupyter-core | 4.7.1 | jupyterlab-pygments | 0.1.2 | jupyterlab-widgets | 1.0.0 |
| kiwisolver | 1.3.1 | koalas | 1.8.2 | MarkupSafe | 2.0.1 |
| matplotlib | 3.4.2 | mistune | 0.8.4 | nbclient | 0.5.3 |
| nbconvert | 6.0.7 | nbformat | 5.1.3 | nest-asyncio | 1.5.1 |
| notebook | 6.3.0 | numpy | 1.19.2 | packaging | 20.9 |
| pandas | 1.2.4 | pandocfilters | 1.4.3 | parso | 0.7.0 |
| patsy | 0.5.1 | pexpect | 4.8.0 | pickleshare | 0.7.5 |
| Pillow | 8.2.0 | pip | 21.0.1 | plotly | 5.3.0 |
| prometheus-client | 0.10.1 | prompt-toolkit | 3.0.17 | protobuf | 3.17.2 |
| psycopg2 | 2.8.5 | ptyprocess | 0.7.0 | pyarrow | 4.0.0 |
| pycparser | 2.20 | Pygments | 2.8.1 | PyGObject | 3.36.0 |
| pyparsing | 2.4.7 | pyrsistent | 0.17.3 | python-apt | 2.0.0+ubuntu0.20.4.6 |
| python-dateutil | 2.8.1 | pytz | 2020.5 | pyzmq | 20.0.0 |
| requests | 2.25.1 | requests-unixsocket | 0.2.0 | s3transfer | 0.3.7 |
| scikit-learn | 0.24.1 | scipy | 1.6.2 | seaborn | 0.11.1 |
| Send2Trash | 1.5.0 | setuptools | 52.0.0 | six | 1.15.0 |
| ssh-import-id | 5.10 | statsmodels | 0.12.2 | tenacity | 8.0.1 |
| terminado | 0.9.4 | testpath | 0.4.4 | threadpoolctl | 2.1.0 |
| tornado | 6.1 | traitlets | 5.0.5 | unattended-upgrades | 0.1 |
| urllib3 | 1.25.11 | virtualenv | 20.4.1 | wcwidth | 0.2.5 |
| webencodings | 0.5.1 | wheel | 0.36.2 | widgetsnbextension | 3.5.1 |
Installed R libraries
R libraries are installed from the Microsoft CRAN snapshot on 2021-10-26.
| Library | Version | Library | Version | Library | Version |
|---|---|---|---|---|---|
| askpass | 1.1 | assertthat | 0.2.1 | backports | 1.2.1 |
| base | 4.1.1 | base64enc | 0.1-3 | bit | 4.0.4 |
| bit64 | 4.0.5 | blob | 1.2.2 | boot | 1.3-28 |
| brew | 1.0-6 | brio | 1.1.2 | broom | 0.7.9 |
| bslib | 0.3.1 | cachem | 1.0.6 | callr | 3.7.0 |
| caret | 6.0-90 | cellranger | 1.1.0 | chron | 2.3-56 |
| class | 7.3-19 | cli | 3.0.1 | clipr | 0.7.1 |
| cluster | 2.1.2 | codetools | 0.2-18 | colorspace | 2.0-2 |
| commonmark | 1.7 | compiler | 4.1.1 | config | 0.3.1 |
| cpp11 | 0.4.0 | crayon | 1.4.1 | credentials | 1.3.1 |
| curl | 4.3.2 | data.table | 1.14.2 | datasets | 4.1.1 |
| DBI | 1.1.1 | dbplyr | 2.1.1 | desc | 1.4.0 |
| devtools | 2.4.2 | diffobj | 0.3.5 | digest | 0.6.28 |
| dplyr | 1.0.7 | dtplyr | 1.1.0 | e1071 | 1.7-9 |
| ellipsis | 0.3.2 | evaluate | 0.14 | fansi | 0.5.0 |
| farver | 2.1.0 | fastmap | 1.1.0 | fontawesome | 0.2.2 |
| forcats | 0.5.1 | foreach | 1.5.1 | foreign | 0.8-81 |
| forge | 0.2.0 | fs | 1.5.0 | future | 1.22.1 |
| future.apply | 1.8.1 | gargle | 1.2.0 | generics | 0.1.1 |
| gert | 1.4.1 | ggplot2 | 3.3.5 | gh | 1.3.0 |
| gitcreds | 0.1.1 | glmnet | 4.1-2 | globals | 0.14.0 |
| glue | 1.4.2 | googledrive | 2.0.0 | googlesheets4 | 1.0.0 |
| gower | 0.2.2 | graphics | 4.1.1 | grDevices | 4.1.1 |
| grid | 4.1.1 | gridExtra | 2.3 | gsubfn | 0.7 |
| gtable | 0.3.0 | haven | 2.4.3 | highr | 0.9 |
| hms | 1.1.1 | htmltools | 0.5.2 | htmlwidgets | 1.5.4 |
| httpuv | 1.6.3 | httr | 1.4.2 | hwriter | 1.3.2 |
| hwriterPlus | 1.0-3 | ids | 1.0.1 | ini | 0.3.1 |
| ipred | 0.9-12 | isoband | 0.2.5 | iterators | 1.0.13 |
| jquerylib | 0.1.4 | jsonlite | 1.7.2 | KernSmooth | 2.23-20 |
| knitr | 1.36 | labeling | 0.4.2 | later | 1.3.0 |
| lattice | 0.20-45 | lava | 1.6.10 | lifecycle | 1.0.1 |
| listenv | 0.8.0 | lubridate | 1.8.0 | magrittr | 2.0.1 |
| markdown | 1.1 | MASS | 7.3-54 | Matrix | 1.3-4 |
| memoise | 2.0.0 | methods | 4.1.1 | mgcv | 1.8-38 |
| mime | 0.12 | ModelMetrics | 1.2.2.2 | modelr | 0.1.8 |
| munsell | 0.5.0 | nlme | 3.1-152 | nnet | 7.3-16 |
| numDeriv | 2016.8-1.1 | openssl | 1.4.5 | parallel | 4.1.1 |
| parallelly | 1.28.1 | pillar | 1.6.4 | pkgbuild | 1.2.0 |
| pkgconfig | 2.0.3 | pkgload | 1.2.3 | plogr | 0.2.0 |
| plyr | 1.8.6 | praise | 1.0.0 | prettyunits | 1.1.1 |
| pROC | 1.18.0 | processx | 3.5.2 | prodlim | 2019.11.13 |
| progress | 1.2.2 | progressr | 0.9.0 | promises | 1.2.0.1 |
| proto | 1.0.0 | proxy | 0.4-26 | ps | 1.6.0 |
| purrr | 0.3.4 | r2d3 | 0.2.5 | R6 | 2.5.1 |
| randomForest | 4.6-14 | rappdirs | 0.3.3 | rcmdcheck | 1.4.0 |
| RColorBrewer | 1.1-2 | Rcpp | 1.0.7 | readr | 2.0.2 |
| readxl | 1.3.1 | recipes | 0.1.17 | rematch | 1.0.1 |
| rematch2 | 2.1.2 | remotes | 2.4.1 | reprex | 2.0.1 |
| reshape2 | 1.4.4 | rlang | 0.4.12 | rmarkdown | 2.11 |
| RODBC | 1.3-19 | roxygen2 | 7.1.2 | rpart | 4.1-15 |
| rprojroot | 2.0.2 | Rserve | 1.8-8 | RSQLite | 2.2.8 |
| rstudioapi | 0.13 | rversions | 2.1.1 | rvest | 1.0.2 |
| sass | 0.4.0 | scales | 1.1.1 | selectr | 0.4-2 |
| sessioninfo | 1.1.1 | shape | 1.4.6 | shiny | 1.7.1 |
| sourcetools | 0.1.7 | sparklyr | 1.7.2 | SparkR | 3.2.0 |
| spatial | 7.3-11 | splines | 4.1.1 | sqldf | 0.4-11 |
| SQUAREM | 2021.1 | stats | 4.1.1 | stats4 | 4.1.1 |
| stringi | 1.7.5 | stringr | 1.4.0 | survival | 3.2-13 |
| sys | 3.4 | tcltk | 4.1.1 | TeachingDemos | 2.10 |
| testthat | 3.1.0 | tibble | 3.1.5 | tidyr | 1.1.4 |
| tidyselect | 1.1.1 | tidyverse | 1.3.1 | timeDate | 3043.102 |
| tinytex | 0.34 | tools | 4.1.1 | tzdb | 0.1.2 |
| usethis | 2.1.2 | utf8 | 1.2.2 | utils | 4.1.1 |
| uuid | 0.1-4 | vctrs | 0.3.8 | viridisLite | 0.4.0 |
| vroom | 1.5.5 | waldo | 0.3.1 | whisker | 0.4 |
| withr | 2.4.2 | xfun | 0.27 | xml2 | 1.3.2 |
| xopen | 1.0.0 | xtable | 1.8-4 | yaml | 2.2.1 |
| zip | 2.2.0 |
Installed Java and Scala libraries (Scala 2.12 cluster version)
| Group ID | Artifact ID | Version |
|---|---|---|
| antlr | antlr | 2.7.7 |
| com.amazonaws | amazon-kinesis-client | 1.12.0 |
| com.amazonaws | aws-java-sdk-autoscaling | 1.11.655 |
| com.amazonaws | aws-java-sdk-cloudformation | 1.11.655 |
| com.amazonaws | aws-java-sdk-cloudfront | 1.11.655 |
| com.amazonaws | aws-java-sdk-cloudhsm | 1.11.655 |
| com.amazonaws | aws-java-sdk-cloudsearch | 1.11.655 |
| com.amazonaws | aws-java-sdk-cloudtrail | 1.11.655 |
| com.amazonaws | aws-java-sdk-cloudwatch | 1.11.655 |
| com.amazonaws | aws-java-sdk-cloudwatchmetrics | 1.11.655 |
| com.amazonaws | aws-java-sdk-codedeploy | 1.11.655 |
| com.amazonaws | aws-java-sdk-cognitoidentity | 1.11.655 |
| com.amazonaws | aws-java-sdk-cognitosync | 1.11.655 |
| com.amazonaws | aws-java-sdk-config | 1.11.655 |
| com.amazonaws | aws-java-sdk-core | 1.11.655 |
| com.amazonaws | aws-java-sdk-datapipeline | 1.11.655 |
| com.amazonaws | aws-java-sdk-directconnect | 1.11.655 |
| com.amazonaws | aws-java-sdk-directory | 1.11.655 |
| com.amazonaws | aws-java-sdk-dynamodb | 1.11.655 |
| com.amazonaws | aws-java-sdk-ec2 | 1.11.655 |
| com.amazonaws | aws-java-sdk-ecs | 1.11.655 |
| com.amazonaws | aws-java-sdk-efs | 1.11.655 |
| com.amazonaws | aws-java-sdk-elasticache | 1.11.655 |
| com.amazonaws | aws-java-sdk-elasticbeanstalk | 1.11.655 |
| com.amazonaws | aws-java-sdk-elasticloadbalancing | 1.11.655 |
| com.amazonaws | aws-java-sdk-elastictranscoder | 1.11.655 |
| com.amazonaws | aws-java-sdk-emr | 1.11.655 |
| com.amazonaws | aws-java-sdk-glacier | 1.11.655 |
| com.amazonaws | aws-java-sdk-glue | 1.11.655 |
| com.amazonaws | aws-java-sdk-iam | 1.11.655 |
| com.amazonaws | aws-java-sdk-importexport | 1.11.655 |
| com.amazonaws | aws-java-sdk-kinesis | 1.11.655 |
| com.amazonaws | aws-java-sdk-kms | 1.11.655 |
| com.amazonaws | aws-java-sdk-lambda | 1.11.655 |
| com.amazonaws | aws-java-sdk-logs | 1.11.655 |
| com.amazonaws | aws-java-sdk-machinelearning | 1.11.655 |
| com.amazonaws | aws-java-sdk-opsworks | 1.11.655 |
| com.amazonaws | aws-java-sdk-rds | 1.11.655 |
| com.amazonaws | aws-java-sdk-redshift | 1.11.655 |
| com.amazonaws | aws-java-sdk-route53 | 1.11.655 |
| com.amazonaws | aws-java-sdk-s3 | 1.11.655 |
| com.amazonaws | aws-java-sdk-ses | 1.11.655 |
| com.amazonaws | aws-java-sdk-simpledb | 1.11.655 |
| com.amazonaws | aws-java-sdk-simpleworkflow | 1.11.655 |
| com.amazonaws | aws-java-sdk-sns | 1.11.655 |
| com.amazonaws | aws-java-sdk-sqs | 1.11.655 |
| com.amazonaws | aws-java-sdk-ssm | 1.11.655 |
| com.amazonaws | aws-java-sdk-storagegateway | 1.11.655 |
| com.amazonaws | aws-java-sdk-sts | 1.11.655 |
| com.amazonaws | aws-java-sdk-support | 1.11.655 |
| com.amazonaws | aws-java-sdk-swf-libraries | 1.11.22 |
| com.amazonaws | aws-java-sdk-workspaces | 1.11.655 |
| com.amazonaws | jmespath-java | 1.11.655 |
| com.chuusai | shapeless_2.12 | 2.3.3 |
| com.clearspring.analytics | stream | 2.9.6 |
| com.databricks | Rserve | 1.8-3 |
| com.databricks | jets3t | 0.7.1-0 |
| com.databricks.scalapb | compilerplugin_2.12 | 0.4.15-10 |
| com.databricks.scalapb | scalapb-runtime_2.12 | 0.4.15-10 |
| com.esotericsoftware | kryo-shaded | 4.0.2 |
| com.esotericsoftware | minlog | 1.3.0 |
| com.fasterxml | classmate | 1.3.4 |
| com.fasterxml.jackson.core | jackson-annotations | 2.12.2 |
| com.fasterxml.jackson.core | jackson-core | 2.12.2 |
| com.fasterxml.jackson.core | jackson-databind | 2.12.2 |
| com.fasterxml.jackson.dataformat | jackson-dataformat-cbor | 2.12.2 |
| com.fasterxml.jackson.datatype | jackson-datatype-joda | 2.12.2 |
| com.fasterxml.jackson.module | jackson-module-paranamer | 2.12.2 |
| com.fasterxml.jackson.module | jackson-module-scala_2.12 | 2.12.2 |
| com.github.ben-manes.caffeine | caffeine | 2.3.4 |
| com.github.fommil | jniloader | 1.1 |
| com.github.fommil.netlib | core | 1.1.2 |
| com.github.fommil.netlib | native_ref-java | 1.1 |
| com.github.fommil.netlib | native_ref-java-natives | 1.1 |
| com.github.fommil.netlib | native_system-java | 1.1 |
| com.github.fommil.netlib | native_system-java-natives | 1.1 |
| com.github.fommil.netlib | netlib-native_ref-linux-x86_64-natives | 1.1 |
| com.github.fommil.netlib | netlib-native_system-linux-x86_64-natives | 1.1 |
| com.github.luben | zstd-jni | 1.5.0-4 |
| com.github.wendykierp | JTransforms | 3.1 |
| com.google.code.findbugs | jsr305 | 3.0.0 |
| com.google.code.gson | gson | 2.8.6 |
| com.google.crypto.tink | tink | 1.6.0 |
| com.google.flatbuffers | flatbuffers-java | 1.9.0 |
| com.google.guava | guava | 15.0 |
| com.google.protobuf | protobuf-java | 2.6.1 |
| com.h2database | h2 | 1.4.195 |
| com.helger | profiler | 1.1.1 |
| com.jcraft | jsch | 0.1.50 |
| com.jolbox | bonecp | 0.8.0.RELEASE |
| com.lihaoyi | sourcecode_2.12 | 0.1.9 |
| com.microsoft.azure | azure-data-lake-store-sdk | 2.3.9 |
| com.microsoft.sqlserver | mssql-jdbc | 9.2.1.jre8 |
| com.ning | compress-lzf | 1.0.3 |
| com.sun.istack | istack-commons-runtime | 3.0.8 |
| com.sun.mail | javax.mail | 1.5.2 |
| com.tdunning | json | 1.8 |
| com.thoughtworks.paranamer | paranamer | 2.8 |
| com.trueaccord.lenses | lenses_2.12 | 0.4.12 |
| com.twitter | chill-java | 0.10.0 |
| com.twitter | chill_2.12 | 0.10.0 |
| com.twitter | util-app_2.12 | 7.1.0 |
| com.twitter | util-core_2.12 | 7.1.0 |
| com.twitter | util-function_2.12 | 7.1.0 |
| com.twitter | util-jvm_2.12 | 7.1.0 |
| com.twitter | util-lint_2.12 | 7.1.0 |
| com.twitter | util-registry_2.12 | 7.1.0 |
| com.twitter | util-stats_2.12 | 7.1.0 |
| com.typesafe | config | 1.2.1 |
| com.typesafe.scala-logging | scala-logging_2.12 | 3.7.2 |
| com.univocity | univocity-parsers | 2.9.1 |
| com.zaxxer | HikariCP | 4.0.3 |
| commons-cli | commons-cli | 1.2 |
| commons-codec | commons-codec | 1.15 |
| commons-collections | commons-collections | 3.2.2 |
| commons-dbcp | commons-dbcp | 1.4 |
| commons-fileupload | commons-fileupload | 1.3.3 |
| commons-httpclient | commons-httpclient | 3.1 |
| commons-io | commons-io | 2.8.0 |
| commons-lang | commons-lang | 2.6 |
| commons-logging | commons-logging | 1.1.3 |
| commons-net | commons-net | 3.1 |
| commons-pool | commons-pool | 1.5.4 |
| dev.ludovic.netlib | arpack | 1.3.2 |
| dev.ludovic.netlib | blas | 1.3.2 |
| dev.ludovic.netlib | lapack | 1.3.2 |
| hive-2.3__hadoop-3.2 | jets3t-0.7 | liball_deps_2.12 |
| info.ganglia.gmetric4j | gmetric4j | 1.0.10 |
| io.airlift | aircompressor | 0.19 |
| io.delta | delta-sharing-spark_2.12 | 0.2.0 |
| io.dropwizard.metrics | metrics-core | 4.1.1 |
| io.dropwizard.metrics | metrics-graphite | 4.1.1 |
| io.dropwizard.metrics | metrics-healthchecks | 4.1.1 |
| io.dropwizard.metrics | metrics-jetty9 | 4.1.1 |
| io.dropwizard.metrics | metrics-jmx | 4.1.1 |
| io.dropwizard.metrics | metrics-json | 4.1.1 |
| io.dropwizard.metrics | metrics-jvm | 4.1.1 |
| io.dropwizard.metrics | metrics-servlets | 4.1.1 |
| io.netty | netty-all | 4.1.63.Final |
| io.prometheus | simpleclient | 0.7.0 |
| io.prometheus | simpleclient_common | 0.7.0 |
| io.prometheus | simpleclient_dropwizard | 0.7.0 |
| io.prometheus | simpleclient_pushgateway | 0.7.0 |
| io.prometheus | simpleclient_servlet | 0.7.0 |
| io.prometheus.jmx | collector | 0.12.0 |
| jakarta.annotation | jakarta.annotation-api | 1.3.5 |
| jakarta.servlet | jakarta.servlet-api | 4.0.3 |
| jakarta.validation | jakarta.validation-api | 2.0.2 |
| jakarta.ws.rs | jakarta.ws.rs-api | 2.1.6 |
| javax.activation | activation | 1.1.1 |
| javax.annotation | javax.annotation-api | 1.3.2 |
| javax.el | javax.el-api | 2.2.4 |
| javax.jdo | jdo-api | 3.0.1 |
| javax.transaction | jta | 1.1 |
| javax.transaction | transaction-api | 1.1 |
| javax.xml.bind | jaxb-api | 2.2.2 |
| javax.xml.stream | stax-api | 1.0-2 |
| javolution | javolution | 5.5.1 |
| jline | jline | 2.14.6 |
| joda-time | joda-time | 2.10.5 |
| log4j | apache-log4j-extras | 1.2.17 |
| log4j | log4j | 1.2.17 |
| maven-trees | hive-2.3__hadoop-3.2 | liball_deps_2.12 |
| net.java.dev.jna | jna | 5.8.0 |
| net.razorvine | pyrolite | 4.30 |
| net.sf.jpam | jpam | 1.1 |
| net.sf.opencsv | opencsv | 2.3 |
| net.sf.supercsv | super-csv | 2.2.0 |
| net.snowflake | snowflake-ingest-sdk | 0.9.6 |
| net.snowflake | snowflake-jdbc | 3.13.3 |
| net.snowflake | spark-snowflake_2.12 | 2.9.0-spark_3.1 |
| net.sourceforge.f2j | arpack_combined_all | 0.1 |
| org.acplt.remotetea | remotetea-oncrpc | 1.1.2 |
| org.antlr | ST4 | 4.0.4 |
| org.antlr | antlr-runtime | 3.5.2 |
| org.antlr | antlr4-runtime | 4.8 |
| org.antlr | stringtemplate | 3.2.1 |
| org.apache.ant | ant | 1.9.2 |
| org.apache.ant | ant-jsch | 1.9.2 |
| org.apache.ant | ant-launcher | 1.9.2 |
| org.apache.arrow | arrow-format | 2.0.0 |
| org.apache.arrow | arrow-memory-core | 2.0.0 |
| org.apache.arrow | arrow-memory-netty | 2.0.0 |
| org.apache.arrow | arrow-vector | 2.0.0 |
| org.apache.avro | avro | 1.10.2 |
| org.apache.avro | avro-ipc | 1.10.2 |
| org.apache.avro | avro-mapred | 1.10.2 |
| org.apache.commons | commons-compress | 1.20 |
| org.apache.commons | commons-crypto | 1.1.0 |
| org.apache.commons | commons-lang3 | 3.12.0 |
| org.apache.commons | commons-math3 | 3.4.1 |
| org.apache.commons | commons-text | 1.6 |
| org.apache.curator | curator-client | 2.13.0 |
| org.apache.curator | curator-framework | 2.13.0 |
| org.apache.curator | curator-recipes | 2.13.0 |
| org.apache.derby | derby | 10.14.2.0 |
| org.apache.hadoop | hadoop-client-api | 3.3.1-databricks |
| org.apache.hadoop | hadoop-client-runtime | 3.3.1 |
| org.apache.hive | hive-beeline | 2.3.9 |
| org.apache.hive | hive-cli | 2.3.9 |
| org.apache.hive | hive-jdbc | 2.3.9 |
| org.apache.hive | hive-llap-client | 2.3.9 |
| org.apache.hive | hive-llap-common | 2.3.9 |
| org.apache.hive | hive-serde | 2.3.9 |
| org.apache.hive | hive-shims | 2.3.9 |
| org.apache.hive | hive-storage-api | 2.7.2 |
| org.apache.hive.shims | hive-shims-0.23 | 2.3.9 |
| org.apache.hive.shims | hive-shims-common | 2.3.9 |
| org.apache.hive.shims | hive-shims-scheduler | 2.3.9 |
| org.apache.htrace | htrace-core4 | 4.1.0-incubating |
| org.apache.httpcomponents | httpclient | 4.5.13 |
| org.apache.httpcomponents | httpcore | 4.4.12 |
| org.apache.ivy | ivy | 2.5.0 |
| org.apache.mesos | mesos-shaded-protobuf | 1.4.0 |
| org.apache.orc | orc-core | 1.6.10 |
| org.apache.orc | orc-mapreduce | 1.6.10 |
| org.apache.orc | orc-shims | 1.6.10 |
| org.apache.parquet | parquet-column | 1.12.0-databricks-0003 |
| org.apache.parquet | parquet-common | 1.12.0-databricks-0003 |
| org.apache.parquet | parquet-encoding | 1.12.0-databricks-0003 |
| org.apache.parquet | parquet-format-structures | 1.12.0-databricks-0003 |
| org.apache.parquet | parquet-hadoop | 1.12.0-databricks-0003 |
| org.apache.parquet | parquet-jackson | 1.12.0-databricks-0003 |
| org.apache.thrift | libfb303 | 0.9.3 |
| org.apache.thrift | libthrift | 0.12.0 |
| org.apache.xbean | xbean-asm9-shaded | 4.20 |
| org.apache.yetus | audience-annotations | 0.5.0 |
| org.apache.zookeeper | zookeeper | 3.6.2 |
| org.apache.zookeeper | zookeeper-jute | 3.6.2 |
| org.checkerframework | checker-qual | 3.5.0 |
| org.codehaus.jackson | jackson-core-asl | 1.9.13 |
| org.codehaus.jackson | jackson-mapper-asl | 1.9.13 |
| org.codehaus.janino | commons-compiler | 3.0.16 |
| org.codehaus.janino | janino | 3.0.16 |
| org.datanucleus | datanucleus-api-jdo | 4.2.4 |
| org.datanucleus | datanucleus-core | 4.1.17 |
| org.datanucleus | datanucleus-rdbms | 4.1.19 |
| org.datanucleus | javax.jdo | 3.2.0-m3 |
| org.eclipse.jetty | jetty-client | 9.4.40.v20210413 |
| org.eclipse.jetty | jetty-continuation | 9.4.40.v20210413 |
| org.eclipse.jetty | jetty-http | 9.4.40.v20210413 |
| org.eclipse.jetty | jetty-io | 9.4.40.v20210413 |
| org.eclipse.jetty | jetty-jndi | 9.4.40.v20210413 |
| org.eclipse.jetty | jetty-plus | 9.4.40.v20210413 |
| org.eclipse.jetty | jetty-proxy | 9.4.40.v20210413 |
| org.eclipse.jetty | jetty-security | 9.4.40.v20210413 |
| org.eclipse.jetty | jetty-server | 9.4.40.v20210413 |
| org.eclipse.jetty | jetty-servlet | 9.4.40.v20210413 |
| org.eclipse.jetty | jetty-servlets | 9.4.40.v20210413 |
| org.eclipse.jetty | jetty-util | 9.4.40.v20210413 |
| org.eclipse.jetty | jetty-util-ajax | 9.4.40.v20210413 |
| org.eclipse.jetty | jetty-webapp | 9.4.40.v20210413 |
| org.eclipse.jetty | jetty-xml | 9.4.40.v20210413 |
| org.eclipse.jetty.websocket | websocket-api | 9.4.40.v20210413 |
| org.eclipse.jetty.websocket | websocket-client | 9.4.40.v20210413 |
| org.eclipse.jetty.websocket | websocket-common | 9.4.40.v20210413 |
| org.eclipse.jetty.websocket | websocket-server | 9.4.40.v20210413 |
| org.eclipse.jetty.websocket | websocket-servlet | 9.4.40.v20210413 |
| org.fusesource.leveldbjni | leveldbjni-all | 1.8 |
| org.glassfish.hk2 | hk2-api | 2.6.1 |
| org.glassfish.hk2 | hk2-locator | 2.6.1 |
| org.glassfish.hk2 | hk2-utils | 2.6.1 |
| org.glassfish.hk2 | osgi-resource-locator | 1.0.3 |
| org.glassfish.hk2.external | aopalliance-repackaged | 2.6.1 |
| org.glassfish.hk2.external | jakarta.inject | 2.6.1 |
| org.glassfish.jaxb | jaxb-runtime | 2.3.2 |
| org.glassfish.jersey.containers | jersey-container-servlet | 2.34 |
| org.glassfish.jersey.containers | jersey-container-servlet-core | 2.34 |
| org.glassfish.jersey.core | jersey-client | 2.34 |
| org.glassfish.jersey.core | jersey-common | 2.34 |
| org.glassfish.jersey.core | jersey-server | 2.34 |
| org.glassfish.jersey.inject | jersey-hk2 | 2.34 |
| org.hibernate.validator | hibernate-validator | 6.1.0.Final |
| org.javassist | javassist | 3.25.0-GA |
| org.jboss.logging | jboss-logging | 3.3.2.Final |
| org.jdbi | jdbi | 2.63.1 |
| org.jetbrains | annotations | 17.0.0 |
| org.joda | joda-convert | 1.7 |
| org.jodd | jodd-core | 3.5.2 |
| org.json4s | json4s-ast_2.12 | 3.7.0-M11 |
| org.json4s | json4s-core_2.12 | 3.7.0-M11 |
| org.json4s | json4s-jackson_2.12 | 3.7.0-M11 |
| org.json4s | json4s-scalap_2.12 | 3.7.0-M11 |
| org.lz4 | lz4-java | 1.7.1 |
| org.mariadb.jdbc | mariadb-java-client | 2.2.5 |
| org.objenesis | objenesis | 2.5.1 |
| org.postgresql | postgresql | 42.2.19 |
| org.roaringbitmap | RoaringBitmap | 0.9.14 |
| org.roaringbitmap | shims | 0.9.14 |
| org.rocksdb | rocksdbjni | 6.20.3 |
| org.rosuda.REngine | REngine | 2.1.0 |
| org.scala-lang | scala-compiler_2.12 | 2.12.14 |
| org.scala-lang | scala-library_2.12 | 2.12.14 |
| org.scala-lang | scala-reflect_2.12 | 2.12.14 |
| org.scala-lang.modules | scala-collection-compat_2.12 | 2.4.3 |
| org.scala-lang.modules | scala-parser-combinators_2.12 | 1.1.2 |
| org.scala-lang.modules | scala-xml_2.12 | 1.2.0 |
| org.scala-sbt | test-interface | 1.0 |
| org.scalacheck | scalacheck_2.12 | 1.14.2 |
| org.scalactic | scalactic_2.12 | 3.0.8 |
| org.scalanlp | breeze-macros_2.12 | 1.0 |
| org.scalanlp | breeze_2.12 | 1.0 |
| org.scalatest | scalatest_2.12 | 3.0.8 |
| org.slf4j | jcl-over-slf4j | 1.7.30 |
| org.slf4j | jul-to-slf4j | 1.7.30 |
| org.slf4j | slf4j-api | 1.7.30 |
| org.slf4j | slf4j-log4j12 | 1.7.30 |
| org.spark-project.spark | unused | 1.0.0 |
| org.springframework | spring-core | 4.1.4.RELEASE |
| org.springframework | spring-test | 4.1.4.RELEASE |
| org.threeten | threeten-extra | 1.5.0 |
| org.tukaani | xz | 1.8 |
| org.typelevel | algebra_2.12 | 2.0.0-M2 |
| org.typelevel | cats-kernel_2.12 | 2.0.0-M4 |
| org.typelevel | machinist_2.12 | 0.6.8 |
| org.typelevel | macro-compat_2.12 | 1.1.1 |
| org.typelevel | spire-macros_2.12 | 0.17.0-M1 |
| org.typelevel | spire-platform_2.12 | 0.17.0-M1 |
| org.typelevel | spire-util_2.12 | 0.17.0-M1 |
| org.typelevel | spire_2.12 | 0.17.0-M1 |
| org.wildfly.openssl | wildfly-openssl | 1.0.7.Final |
| org.xerial | sqlite-jdbc | 3.8.11.2 |
| org.xerial.snappy | snappy-java | 1.1.8.4 |
| org.yaml | snakeyaml | 1.24 |
| oro | oro | 2.0.8 |
| pl.edu.icm | JLargeArrays | 1.5 |
| software.amazon.ion | ion-java | 1.0.2 |
| stax | stax-api | 1.0.1 |