Databricks Runtime 10.5 (unsupported)
The following release notes provide information about Databricks Runtime 10.5, powered by Apache Spark 3.2.1. Databricks released these images in May 2022.
New features and improvements
- SQL: New functions
to_numberandtry_to_number - SQL: New function
try_sum - SQL: Update to
ceil,ceiling, andfloorfunctions - SQL: New aggregate functions
regr_avgxandregr_avgy - SQL: Update to
contains,startswith, andendswithfunctions - SQL: New function
array_size - Unity Catalog: SQL
LISToutput removesis_directorycolumn and adds trailing/for directory paths (Public Preview) - Auto Loader: new SQL function
CLOUD_FILES_STATE - Auto Loader: file notification support is now GA
- Delta Lake: new
maxRecordsPerFileoption for maximum records written to a single file - Deprecation of Koalas
- New
_metadatacolumn for all file formats - Test your Delta Live Tables pipeline with Databricks Runtime 10.5
SQL: New functions to_number and try_to_number
The to_number function and try_to_number function allow you to convert character representations of numbers in various formats to DECIMAL.
SQL: New function try_sum
Unlike sum, the try_sum aggregate function returns NULL instead of an error when an overflow occurs.
SQL: Update to ceil, ceiling, and floor functions
The ceil function, ceiling function, and floor function now support a second optional argument to specify a target scale.
SQL: New aggregate functions regr_avgx and regr_avgy
The new regr_avgx aggregate function and regr_avgy aggregate function support linear regressions.
SQL: Update to contains, startswith, and endswith functions
The contains function, startswith function, and endswith function now support binary data types as input in addition to strings.
SQL: New function array_size
The new array_size function calculates the size of an array.
Unity Catalog: SQL LIST output removes is_directory column and adds trailing / for directory paths (Public Preview)
The output of the SQL LIST statement for listing paths governed by Unity Catalog has changed. The is_directory column is no longer displayed. Instead, paths that represent directories now end with a trailing slash (/) for the path and name columns. This change is for consistency with the output of the dbutils.fs.ls command.
Auto Loader: new SQL function CLOUD_FILES_STATE
You can use the new CLOUD_FILES_STATE function to query the internal state of an Auto Loader stream; see Querying files discovered by Auto Loader.
Auto Loader: file notification support is now GA
File notification support for What is Auto Loader? on Google Cloud is now generally available.
Delta Lake: new maxRecordsPerFile option for maximum records written to a single file
When you use the DataFrame APIs to write to a Delta table, you can use the maxRecordsPerFile option to specify the maximum number of records to write to a single file. Specifying a value of zero or a negative value represents no limit. See Configure Delta Lake to control data file size.
Deprecation of Koalas
Koalas is now deprecated. Koalas provides pandas APIs on top of Apache Spark. Koalas was contributed to by the community and has been available as pandas API on Spark beginning in Apache Spark 3.2. Koalas continues to be preinstalled for Databricks Runtime 10.5, but Koalas will no longer be preinstalled beginning with the next Databricks Runtime version.
To learn more about pandas API on Spark, see:
- Pandas API on Spark in the Azure Databricks documentation
- Pandas API on Spark in the Apache Spark documentation
- Migrating from Koalas to pandas API on Spark in the Apache Spark documentation
New _metadata column for all file formats
You can get metadata information for input files with the _metadata column. The _metadata column is hidden. This column is available for all input file formats. To include the _metadata column in a returned DataFrame, you must explicitly reference this column in your query. See File metadata column.
Test your Delta Live Tables pipeline with Databricks Runtime 10.5
Databricks Runtime 10.5 is now the supported preview version for Delta Live Tables. To test your pipeline with Databricks Runtime 10.5, set the channel setting to preview in your pipeline settings.
Bug fixes
- Fixed the display of
numUpdatedRowsandnumCopiedRowsmetric results for update operations on Delta tables that have change data capture (CDC) enabled.
Library upgrades
- Upgraded Python libraries:
- plotly from 5.5.0 to 5.6.0
- Upgraded R libraries:
- base from 4.1.2 to 4.1.3
- caret from 6.0-90 to 6.0-91
- cluster from 2.1.2 to 2.1.3
- commonmark from 1.7 to 1.8.0
- compiler from 4.1.2 to 4.1.3
- crayon from 1.5.0 to 1.5.1
- datasets from 4.1.2 to 4.1.3
- desc from 1.4.0 to 1.4.1
- fansi from 1.0.2 to 1.0.3
- glue from 1.6.1 to 1.6.2
- graphics from 4.1.2 to 4.1.3
- grDevices from 4.1.2 to 4.1.3
- grid from 4.1.2 to 4.1.3
- knitr from 1.37 to 1.38
- MASS from 7.3-55 to 7.3-56
- Matrix from 1.4-0 to 1.4-1
- methods from 4.1.2 to 4.1.3
- mgcv from 1.8-39 to 1.8-40
- nlme from 3.1-155 to 3.1-157
- openssl from 1.4.6 to 2.0.0
- parallel from 4.1.2 to 4.1.3
- plyr from 1.8.6 to 1.8.7
- processx from 3.5.2 to 3.5.3
- r2d3 from 0.2.5 to 0.2.6
- Rcpp from 1.0.8 to 1.0.8.3
- rlang from 1.0.1 to 1.0.2
- rmarkdown from 2.11 to 2.13
- RSQLite from 2.2.10 to 2.2.11
- sass from 0.4.0 to 0.4.1
- splines from 4.1.2 to 4.1.3
- stats from 4.1.2 to 4.1.3
- stats4 from 4.1.2 to 4.1.3
- tcltk from 4.1.2 to 4.1.3
- tools from 4.1.2 to 4.1.3
- utils from 4.1.2 to 4.1.3
- uuid from 1.0-3 to 1.0-4
- waldo from 0.3.1 to 0.4.0
- withr from 2.4.3 to 2.5.0
- xfun from 0.29 to 0.30
- Upgraded Java libraries:
- com.h2database.h2 from 1.4.195 to 2.1.210
- org.apache.hadoop.hadoop-client-api from 3.3.1-databricks to 3.3.2-databricks
- org.apache.hadoop.hadoop-client-runtime from 3.3.1 to 3.3.2
- org.rocksdb.rocksdbjni from 6.24.2 to 6.20.3
Apache Spark
Databricks Runtime 10.5 includes Apache Spark 3.2.1. This release includes all Spark fixes and improvements included in Databricks Runtime 10.4 LTS, as well as the following additional bug fixes and improvements made to Spark:
- [SPARK-38354] [SC-97886] Revert "[SQL] Add hash probes metric for shuffled hash join"
- [SPARK-38484] [SAS-103][SAS-111][SAS-113][SC-98102][Cherry-Pick] Instrument PySpark SQL module with general usage logger
- [SPARK-38333] [SQL] PlanExpression expression should skip addExprTree function in Executor
- [SPARK-38646] [PYTHON][10.x] Pull a trait out for Python functions
- [SPARK-38674] [PHOTON] Add a metric to PhotonSubqueryBroadcastExec
- [SPARK-38787] [SS] Replace found value with non-null element in the remaining list for key and remove remaining null elements from values in keyWithIndexToValue store for stream-stream joins
- [SPARK-38446] [Core] Fix deadlock between ExecutorClassLoader and FileDownloadCallback caused by Log4j
- [SPARK-38705] [SQL] Use function identifier in create and drop function command
- [SPARK-38684] [SS] Fix correctness issue on stream-stream outer join with RocksDB state store provider
- [SPARK-38655] [SQL]
OffsetWindowFunctionFrameBasecannot find the offset row whose input is not-null - [SPARK-38204] [SS] Use StatefulOpClusteredDistribution for stateful operators with respecting backward compatibility
- [SPARK-38616] [SQL] Keep track of SQL query text in Catalyst TreeNode
- [SPARK-38600] [SQL] Include
unitinto the sql string ofTIMESTAMPADD/DIFF - [SPARK-34805] [SQL] Propagate metadata from nested columns in Alias
- [SPARK-38535] [SQL] Add the
datetimeUnitenum and use it inTIMESTAMPADD/DIFF - [SPARK-38415] [SQL] Update the histogram_numeric (x, y) result type to make x == the input type
- [SPARK-38148] [SQL] Do not add dynamic partition pruning if there exists static partition pruning
- [SPARK-37527] [SQL] Compile
COVAR_POP,COVAR_SAMPandCORRinH2Dialet - [SPARK-38510] [SQL] Retry ClassSymbol.selfType to work around cyclic references
- [SPARK-38509] [SQL] Unregister the
TIMESTAMPADD/DIFFfunctions and removeDATE_ADD/DIFF - [SPARK-38504] [SQL] Cannot read TimestampNTZ as TimestampLTZ
- [SPARK-38526] [SQL] Fix misleading function alias name for RuntimeReplaceable
- [SPARK-38335] [SQL] Implement parser support for DEFAULT column values
- [SPARK-38593] [SS] Carry over the metric of the number of dropped late events into SessionWindowStateStoreSaveExec
- [SPARK-38549] [SS] Add
numRowsDroppedByWatermarktoSessionWindowStateStoreRestoreExec - [SPARK-38583] [SQL] Restore the behavior of
to_timestampthat allows numeric types - [SPARK-38481] [SQL] Substitute Java overflow exception from
TIMESTAMPADDby Spark exception - [SPARK-37753] [SQL] Fine tune logic to demote Broadcast hash join in DynamicJoinSelection
- [SPARK-37582] [SPARK-37583][SQL] CONTAINS, STARTSWITH, ENDSWITH should support both string and binary types
- [SPARK-38345] [SQL] Introduce SQL function ARRAY_SIZE
- [SPARK-38385] [SQL] Improve error messages of empty statement and
<EOF>in ParseException - [SPARK-37614] [SQL] Support ANSI Aggregate Function: regr_avgx & regr_avgy
- [SPARK-38560] [SQL] If
Sum,Count,Anyaccompany with distinct, cannot do partial agg push down. - [SPARK-38360] [SQL][SS][PYTHON] Introduce a
existsfunction forTreeNodeto eliminate duplicate code patterns - [SPARK-38385] [SQL] Improve error messages of 'mismatched input' cases from ANTLR
- [SPARK-38240] [SQL] Improve RuntimeReplaceable and add a guideline for adding new functions
- [SPARK-38542] [SQL]UnsafeHashedRelation should serialize numKeys out
- [SPARK-38521] [SQL] Change
partitionOverwriteModefrom string to variable in Scala - [SPARK-38558] [SQL] Remove unnecessary casts between IntegerType and IntDecimal
- [SPARK-38489] [SQL] Aggregate.groupOnly support foldable expressions
- [SPARK-38410] [SQL] Support specify initial partition number for rebalance
- [SPARK-38107] [SQL] Use error classes in the compilation errors of python/pandas UDFs
- [SPARK-38354] [SQL] Add hash probes metric for shuffled hash join
- [SPARK-38112] [SQL] Use error classes in the execution errors of date/timestamp handling
- [SPARK-38361] [SQL] Add factory method
getConnectionintoJDBCDialect. - [SPARK-37865] [SQL] Fix union deduplication correctness bug
- [SPARK-38298] [SQL][Tests] Fix DataExpressionSuite, NullExpressionsSuite, StringExpressionsSuite, complexTypesSuite, CastSuite under ANSI mode
- [SPARK-38407] [SQL] ANSI Cast: loosen the limitation of casting non-null complex types
- [SPARK-38352] [SQL] Fix DataFrameAggregateSuite/DataFrameSetOperationsSuite/DataFrameWindowFunctionsSuite under ANSI mode
- [SPARK-38054] [SQL] Supports list namespaces in JDBC v2 MySQL dialect
- [SPARK-37960] [SQL] A new framework to represent catalyst expressions in DS v2 APIs
- [SPARK-38103] [SQL] Migrate parsing errors of transform into the new error framework
- [SPARK-38519] [SQL] AQE throw exception should respect SparkFatalException
- [SPARK-38528] [SQL] Eagerly iterate over aggregate sequence when building project list in
ExtractGenerator - [SPARK-38320] [SS] Fix flatMapGroupsWithState timeout in batch with data for key
- [SPARK-38442] [SQL] Fix ConstantFoldingSuite/ColumnExpressionSuite/DataFrameSuite/AdaptiveQueryExecSuite under ANSI mode
- [SPARK-38268] [SQL] Hide the "failOnError" field in the toString method of Abs/CheckOverflow
- [SPARK-37949] [SQL] Improve Rebalance statistics estimation
- [SPARK-38450] [SQL] Fix HiveQuerySuite//PushFoldableIntoBranchesSuite/TransposeWindowSuite under ANSI mode
- [SPARK-38501] [SQL] Fix thriftserver test failures under ANSI mode
- [SPARK-37895] [SQL] Filter push down column with quoted columns
- [SPARK-38449] [SQL] Avoid call createTable when ignoreIfExists=true and table exists
- [SPARK-38406] [SQL] Improve performance of ShufflePartitionsUtil createSkewPartitionSpecs
- [SPARK-37947] [SQL] Extract generator from GeneratorOuter expression contained by a Generate operator.
- [SPARK-38309] [CORE] Fix SHS
shuffleTotalReadsandshuffleTotalBlockspercentile metrics - [SPARK-38412] [SS] Fix the swapped sequence of from and to in StateSchemaCompatibilityChecker
- [SPARK-38267] [CORE][SQL][SS] Replace pattern matches on boolean expressions with conditional statements
- [SPARK-38104] [SQL] Migrate parsing errors of window into the new error framework
- [SPARK-38434] [SQL] Correct semantic of CheckAnalysis.getDataTypesAreCompatibleFn method
- [SPARK-38033] [SS] The SS processing cannot be started because the com…
- [SPARK-38378] [SQL] Refactoring of the ANTLR grammar definition into separate Parser and Lexer files
- [SPARK-37475] [SQL] Add scale parameter to floor and ceil functions
- [SPARK-38411] [CORE] Use
UTF-8whendoMergeApplicationListingInternalreads event logs - [SPARK-36553] [ML] KMeans avoid compute auxiliary statistics for large K
- [SPARK-38393] [SQL] Clean up deprecated usage of
GenSeq/GenMap - [SPARK-38389] [SQL] Add the
DATEDIFF()andDATE_DIFF()aliases forTIMESTAMPDIFF() - [SPARK-33206] [CORE][3.2] Fix shuffle index cache weight calculation for small index files
- [SPARK-38353] [PYTHON] Instrument enter and exit magic methods for Pandas API on Spark
- [SPARK-38323] [SQL][Streaming] Support the hidden file metadata in Streaming
- [SPARK-38138] [SQL] Materialize QueryPlan subqueries
- [SPARK-38094] Enable matching schema column names by field ids
- [SPARK-37923] [SQL] Generate partition transforms for BucketSpec inside parser
- [SPARK-38363] [SQL] Avoid runtime error in Dataset.summary()/Dataset.describe() when ANSI mode is on
- [SPARK-38284] [SQL] Add the
TIMESTAMPDIFF()function - [SPARK-38251] [SQL] Change Cast.toString as "cast" instead of "ansi_cast" under ANSI mode
- [SPARK-38314] [SQL] Fix of failing to read parquet files after writing the hidden file metadata in
See Databricks Runtime 10.5 maintenance updates.
System environment
- Operating System: Ubuntu 20.04.4 LTS
- Java: Zulu 8.56.0.21-CA-linux64
- Scala: 2.12.14
- Python: 3.8.10
- R: 4.1.3
- Delta Lake: 1.2.1
Installed Python libraries
| Library | Version | Library | Version | Library | Version |
|---|---|---|---|---|---|
| Antergos Linux | 2015.10 (ISO-Rolling) | appdirs | 1.4.4 | argon2-cffi | 20.1.0 |
| async-generator | 1.10 | attrs | 20.3.0 | backcall | 0.2.0 |
| bidict | 0.21.4 | bleach | 3.3.0 | boto3 | 1.16.7 |
| botocore | 1.19.7 | certifi | 2020.12.5 | cffi | 1.14.5 |
| chardet | 4.0.0 | cycler | 0.10.0 | Cython | 0.29.23 |
| dbus-python | 1.2.16 | decorator | 5.0.6 | defusedxml | 0.7.1 |
| distlib | 0.3.4 | distro-info | 0.23ubuntu1 | entrypoints | 0.3 |
| facets-overview | 1.0.0 | filelock | 3.6.0 | idna | 2.10 |
| ipykernel | 5.3.4 | ipython | 7.22.0 | ipython-genutils | 0.2.0 |
| ipywidgets | 7.6.3 | jedi | 0.17.2 | Jinja2 | 2.11.3 |
| jmespath | 0.10.0 | joblib | 1.0.1 | jsonschema | 3.2.0 |
| jupyter-client | 6.1.12 | jupyter-core | 4.7.1 | jupyterlab-pygments | 0.1.2 |
| jupyterlab-widgets | 1.0.0 | kiwisolver | 1.3.1 | koalas | 1.8.2 |
| MarkupSafe | 2.0.1 | matplotlib | 3.4.2 | mistune | 0.8.4 |
| nbclient | 0.5.3 | nbconvert | 6.0.7 | nbformat | 5.1.3 |
| nest-asyncio | 1.5.1 | notebook | 6.3.0 | numpy | 1.20.1 |
| packaging | 20.9 | pandas | 1.2.4 | pandocfilters | 1.4.3 |
| parso | 0.7.0 | patsy | 0.5.1 | pexpect | 4.8.0 |
| pickleshare | 0.7.5 | Pillow | 8.2.0 | pip | 21.0.1 |
| plotly | 5.6.0 | prometheus-client | 0.10.1 | prompt-toolkit | 3.0.17 |
| protobuf | 3.17.2 | psycopg2 | 2.8.5 | ptyprocess | 0.7.0 |
| pyarrow | 4.0.0 | pycparser | 2.20 | Pygments | 2.8.1 |
| PyGObject | 3.36.0 | pyparsing | 2.4.7 | pyrsistent | 0.17.3 |
| python-apt | 2.0.0+ubuntu0.20.4.7 | python-dateutil | 2.8.1 | python-engineio | 4.3.0 |
| python-socketio | 5.4.1 | pytz | 2020.5 | pyzmq | 20.0.0 |
| requests | 2.25.1 | requests-unixsocket | 0.2.0 | s3transfer | 0.3.7 |
| scikit-learn | 0.24.1 | scipy | 1.6.2 | seaborn | 0.11.1 |
| Send2Trash | 1.5.0 | setuptools | 52.0.0 | six | 1.15.0 |
| ssh-import-id | 5.10 | statsmodels | 0.12.2 | tenacity | 8.0.1 |
| terminado | 0.9.4 | testpath | 0.4.4 | threadpoolctl | 2.1.0 |
| tornado | 6.1 | traitlets | 5.0.5 | unattended-upgrades | 0.1 |
| urllib3 | 1.25.11 | virtualenv | 20.4.1 | wcwidth | 0.2.5 |
| webencodings | 0.5.1 | wheel | 0.36.2 | widgetsnbextension | 3.5.1 |
Installed R libraries
R libraries are installed from the Microsoft CRAN snapshot on 2022-03-28.
| Library | Version | Library | Version | Library | Version |
|---|---|---|---|---|---|
| askpass | 1.1 | assertthat | 0.2.1 | backports | 1.4.1 |
| base | 4.1.3 | base64enc | 0.1-3 | bit | 4.0.4 |
| bit64 | 4.0.5 | blob | 1.2.2 | boot | 1.3-28 |
| brew | 1.0-7 | brio | 1.1.3 | broom | 0.7.12 |
| bslib | 0.3.1 | cachem | 1.0.6 | callr | 3.7.0 |
| caret | 6.0-91 | cellranger | 1.1.0 | chron | 2.3-56 |
| class | 7.3-20 | cli | 3.2.0 | clipr | 0.8.0 |
| cluster | 2.1.3 | codetools | 0.2-18 | colorspace | 2.0-3 |
| commonmark | 1.8.0 | compiler | 4.1.3 | config | 0.3.1 |
| cpp11 | 0.4.2 | crayon | 1.5.1 | credentials | 1.3.2 |
| curl | 4.3.2 | data.table | 1.14.2 | datasets | 4.1.3 |
| DBI | 1.1.2 | dbplyr | 2.1.1 | desc | 1.4.1 |
| devtools | 2.4.3 | diffobj | 0.3.5 | digest | 0.6.29 |
| dplyr | 1.0.8 | dtplyr | 1.2.1 | e1071 | 1.7-9 |
| ellipsis | 0.3.2 | evaluate | 0.15 | fansi | 1.0.3 |
| farver | 2.1.0 | fastmap | 1.1.0 | fontawesome | 0.2.2 |
| forcats | 0.5.1 | foreach | 1.5.2 | foreign | 0.8-82 |
| forge | 0.2.0 | fs | 1.5.2 | future | 1.24.0 |
| future.apply | 1.8.1 | gargle | 1.2.0 | generics | 0.1.2 |
| gert | 1.5.0 | ggplot2 | 3.3.5 | gh | 1.3.0 |
| gitcreds | 0.1.1 | glmnet | 4.1-3 | globals | 0.14.0 |
| glue | 1.6.2 | googledrive | 2.0.0 | googlesheets4 | 1.0.0 |
| gower | 1.0.0 | graphics | 4.1.3 | grDevices | 4.1.3 |
| grid | 4.1.3 | gridExtra | 2.3 | gsubfn | 0.7 |
| gtable | 0.3.0 | hardhat | 0.2.0 | haven | 2.4.3 |
| highr | 0.9 | hms | 1.1.1 | htmltools | 0.5.2 |
| htmlwidgets | 1.5.4 | httpuv | 1.6.5 | httr | 1.4.2 |
| hwriter | 1.3.2 | hwriterPlus | 1.0-3 | ids | 1.0.1 |
| ini | 0.3.1 | ipred | 0.9-12 | isoband | 0.2.5 |
| iterators | 1.0.14 | jquerylib | 0.1.4 | jsonlite | 1.8.0 |
| KernSmooth | 2.23-20 | knitr | 1.38 | labeling | 0.4.2 |
| later | 1.3.0 | lattice | 0.20-45 | lava | 1.6.10 |
| lifecycle | 1.0.1 | listenv | 0.8.0 | lubridate | 1.8.0 |
| magrittr | 2.0.2 | markdown | 1.1 | MASS | 7.3-56 |
| Matrix | 1.4-1 | memoise | 2.0.1 | methods | 4.1.3 |
| mgcv | 1.8-40 | mime | 0.12 | ModelMetrics | 1.2.2.2 |
| modelr | 0.1.8 | munsell | 0.5.0 | nlme | 3.1-157 |
| nnet | 7.3-17 | numDeriv | 2016.8-1.1 | openssl | 2.0.0 |
| parallel | 4.1.3 | parallelly | 1.30.0 | pillar | 1.7.0 |
| pkgbuild | 1.3.1 | pkgconfig | 2.0.3 | pkgload | 1.2.4 |
| plogr | 0.2.0 | plyr | 1.8.7 | praise | 1.0.0 |
| prettyunits | 1.1.1 | pROC | 1.18.0 | processx | 3.5.3 |
| prodlim | 2019.11.13 | progress | 1.2.2 | progressr | 0.10.0 |
| promises | 1.2.0.1 | proto | 1.0.0 | proxy | 0.4-26 |
| ps | 1.6.0 | purrr | 0.3.4 | r2d3 | 0.2.6 |
| R6 | 2.5.1 | randomForest | 4.7-1 | rappdirs | 0.3.3 |
| rcmdcheck | 1.4.0 | RColorBrewer | 1.1-2 | Rcpp | 1.0.8.3 |
| RcppEigen | 0.3.3.9.1 | readr | 2.1.2 | readxl | 1.3.1 |
| recipes | 0.2.0 | rematch | 1.0.1 | rematch2 | 2.1.2 |
| remotes | 2.4.2 | reprex | 2.0.1 | reshape2 | 1.4.4 |
| rlang | 1.0.2 | rmarkdown | 2.13 | RODBC | 1.3-19 |
| roxygen2 | 7.1.2 | rpart | 4.1.16 | rprojroot | 2.0.2 |
| Rserve | 1.8-10 | RSQLite | 2.2.11 | rstudioapi | 0.13 |
| rversions | 2.1.1 | rvest | 1.0.2 | sass | 0.4.1 |
| scales | 1.1.1 | selectr | 0.4-2 | sessioninfo | 1.2.2 |
| shape | 1.4.6 | shiny | 1.7.1 | sourcetools | 0.1.7 |
| sparklyr | 1.7.5 | SparkR | 3.2.0 | spatial | 7.3-11 |
| splines | 4.1.3 | sqldf | 0.4-11 | SQUAREM | 2021.1 |
| stats | 4.1.3 | stats4 | 4.1.3 | stringi | 1.7.6 |
| stringr | 1.4.0 | survival | 3.2-13 | sys | 3.4 |
| tcltk | 4.1.3 | TeachingDemos | 2.10 | testthat | 3.1.2 |
| tibble | 3.1.6 | tidyr | 1.2.0 | tidyselect | 1.1.2 |
| tidyverse | 1.3.1 | timeDate | 3043.102 | tinytex | 0.37 |
| tools | 4.1.3 | tzdb | 0.2.0 | usethis | 2.1.5 |
| utf8 | 1.2.2 | utils | 4.1.3 | uuid | 1.0-4 |
| vctrs | 0.3.8 | viridisLite | 0.4.0 | vroom | 1.5.7 |
| waldo | 0.4.0 | whisker | 0.4 | withr | 2.5.0 |
| xfun | 0.30 | xml2 | 1.3.3 | xopen | 1.0.0 |
| xtable | 1.8-4 | yaml | 2.3.5 | zip | 2.2.0 |
Installed Java and Scala libraries (Scala 2.12 cluster version)
| Group ID | Artifact ID | Version |
|---|---|---|
| antlr | antlr | 2.7.7 |
| com.amazonaws | amazon-kinesis-client | 1.12.0 |
| com.amazonaws | aws-java-sdk-autoscaling | 1.12.189 |
| com.amazonaws | aws-java-sdk-cloudformation | 1.12.189 |
| com.amazonaws | aws-java-sdk-cloudfront | 1.12.189 |
| com.amazonaws | aws-java-sdk-cloudhsm | 1.12.189 |
| com.amazonaws | aws-java-sdk-cloudsearch | 1.12.189 |
| com.amazonaws | aws-java-sdk-cloudtrail | 1.12.189 |
| com.amazonaws | aws-java-sdk-cloudwatch | 1.12.189 |
| com.amazonaws | aws-java-sdk-cloudwatchmetrics | 1.12.189 |
| com.amazonaws | aws-java-sdk-codedeploy | 1.12.189 |
| com.amazonaws | aws-java-sdk-cognitoidentity | 1.12.189 |
| com.amazonaws | aws-java-sdk-cognitosync | 1.12.189 |
| com.amazonaws | aws-java-sdk-config | 1.12.189 |
| com.amazonaws | aws-java-sdk-core | 1.12.189 |
| com.amazonaws | aws-java-sdk-datapipeline | 1.12.189 |
| com.amazonaws | aws-java-sdk-directconnect | 1.12.189 |
| com.amazonaws | aws-java-sdk-directory | 1.12.189 |
| com.amazonaws | aws-java-sdk-dynamodb | 1.12.189 |
| com.amazonaws | aws-java-sdk-ec2 | 1.12.189 |
| com.amazonaws | aws-java-sdk-ecs | 1.12.189 |
| com.amazonaws | aws-java-sdk-efs | 1.12.189 |
| com.amazonaws | aws-java-sdk-elasticache | 1.12.189 |
| com.amazonaws | aws-java-sdk-elasticbeanstalk | 1.12.189 |
| com.amazonaws | aws-java-sdk-elasticloadbalancing | 1.12.189 |
| com.amazonaws | aws-java-sdk-elastictranscoder | 1.12.189 |
| com.amazonaws | aws-java-sdk-emr | 1.12.189 |
| com.amazonaws | aws-java-sdk-glacier | 1.12.189 |
| com.amazonaws | aws-java-sdk-glue | 1.12.189 |
| com.amazonaws | aws-java-sdk-iam | 1.12.189 |
| com.amazonaws | aws-java-sdk-importexport | 1.12.189 |
| com.amazonaws | aws-java-sdk-kinesis | 1.12.189 |
| com.amazonaws | aws-java-sdk-kms | 1.12.189 |
| com.amazonaws | aws-java-sdk-lambda | 1.12.189 |
| com.amazonaws | aws-java-sdk-logs | 1.12.189 |
| com.amazonaws | aws-java-sdk-machinelearning | 1.12.189 |
| com.amazonaws | aws-java-sdk-opsworks | 1.12.189 |
| com.amazonaws | aws-java-sdk-rds | 1.12.189 |
| com.amazonaws | aws-java-sdk-redshift | 1.12.189 |
| com.amazonaws | aws-java-sdk-route53 | 1.12.189 |
| com.amazonaws | aws-java-sdk-s3 | 1.12.189 |
| com.amazonaws | aws-java-sdk-ses | 1.12.189 |
| com.amazonaws | aws-java-sdk-simpledb | 1.12.189 |
| com.amazonaws | aws-java-sdk-simpleworkflow | 1.12.189 |
| com.amazonaws | aws-java-sdk-sns | 1.12.189 |
| com.amazonaws | aws-java-sdk-sqs | 1.12.189 |
| com.amazonaws | aws-java-sdk-ssm | 1.12.189 |
| com.amazonaws | aws-java-sdk-storagegateway | 1.12.189 |
| com.amazonaws | aws-java-sdk-sts | 1.12.189 |
| com.amazonaws | aws-java-sdk-support | 1.12.189 |
| com.amazonaws | aws-java-sdk-swf-libraries | 1.11.22 |
| com.amazonaws | aws-java-sdk-workspaces | 1.12.189 |
| com.amazonaws | jmespath-java | 1.12.189 |
| com.chuusai | shapeless_2.12 | 2.3.3 |
| com.clearspring.analytics | stream | 2.9.6 |
| com.databricks | Rserve | 1.8-3 |
| com.databricks | jets3t | 0.7.1-0 |
| com.databricks.scalapb | compilerplugin_2.12 | 0.4.15-10 |
| com.databricks.scalapb | scalapb-runtime_2.12 | 0.4.15-10 |
| com.esotericsoftware | kryo-shaded | 4.0.2 |
| com.esotericsoftware | minlog | 1.3.0 |
| com.fasterxml | classmate | 1.3.4 |
| com.fasterxml.jackson.core | jackson-annotations | 2.12.3 |
| com.fasterxml.jackson.core | jackson-core | 2.12.3 |
| com.fasterxml.jackson.core | jackson-databind | 2.12.3 |
| com.fasterxml.jackson.dataformat | jackson-dataformat-cbor | 2.12.3 |
| com.fasterxml.jackson.datatype | jackson-datatype-joda | 2.12.3 |
| com.fasterxml.jackson.module | jackson-module-paranamer | 2.12.3 |
| com.fasterxml.jackson.module | jackson-module-scala_2.12 | 2.12.3 |
| com.github.ben-manes.caffeine | caffeine | 2.3.4 |
| com.github.fommil | jniloader | 1.1 |
| com.github.fommil.netlib | core | 1.1.2 |
| com.github.fommil.netlib | native_ref-java | 1.1 |
| com.github.fommil.netlib | native_ref-java-natives | 1.1 |
| com.github.fommil.netlib | native_system-java | 1.1 |
| com.github.fommil.netlib | native_system-java-natives | 1.1 |
| com.github.fommil.netlib | netlib-native_ref-linux-x86_64-natives | 1.1 |
| com.github.fommil.netlib | netlib-native_system-linux-x86_64-natives | 1.1 |
| com.github.luben | zstd-jni | 1.5.0-4 |
| com.github.wendykierp | JTransforms | 3.1 |
| com.google.code.findbugs | jsr305 | 3.0.0 |
| com.google.code.gson | gson | 2.8.6 |
| com.google.crypto.tink | tink | 1.6.0 |
| com.google.flatbuffers | flatbuffers-java | 1.9.0 |
| com.google.guava | guava | 15.0 |
| com.google.protobuf | protobuf-java | 2.6.1 |
| com.h2database | h2 | 2.1.210 |
| com.helger | profiler | 1.1.1 |
| com.jcraft | jsch | 0.1.50 |
| com.jolbox | bonecp | 0.8.0.RELEASE |
| com.lihaoyi | sourcecode_2.12 | 0.1.9 |
| com.microsoft.azure | azure-data-lake-store-sdk | 2.3.9 |
| com.ning | compress-lzf | 1.0.3 |
| com.sun.istack | istack-commons-runtime | 3.0.8 |
| com.sun.mail | javax.mail | 1.5.2 |
| com.tdunning | json | 1.8 |
| com.thoughtworks.paranamer | paranamer | 2.8 |
| com.trueaccord.lenses | lenses_2.12 | 0.4.12 |
| com.twitter | chill-java | 0.10.0 |
| com.twitter | chill_2.12 | 0.10.0 |
| com.twitter | util-app_2.12 | 7.1.0 |
| com.twitter | util-core_2.12 | 7.1.0 |
| com.twitter | util-function_2.12 | 7.1.0 |
| com.twitter | util-jvm_2.12 | 7.1.0 |
| com.twitter | util-lint_2.12 | 7.1.0 |
| com.twitter | util-registry_2.12 | 7.1.0 |
| com.twitter | util-stats_2.12 | 7.1.0 |
| com.typesafe | config | 1.2.1 |
| com.typesafe.scala-logging | scala-logging_2.12 | 3.7.2 |
| com.univocity | univocity-parsers | 2.9.1 |
| com.zaxxer | HikariCP | 4.0.3 |
| commons-cli | commons-cli | 1.2 |
| commons-codec | commons-codec | 1.15 |
| commons-collections | commons-collections | 3.2.2 |
| commons-dbcp | commons-dbcp | 1.4 |
| commons-fileupload | commons-fileupload | 1.3.3 |
| commons-httpclient | commons-httpclient | 3.1 |
| commons-io | commons-io | 2.8.0 |
| commons-lang | commons-lang | 2.6 |
| commons-logging | commons-logging | 1.1.3 |
| commons-net | commons-net | 3.1 |
| commons-pool | commons-pool | 1.5.4 |
| dev.ludovic.netlib | arpack | 2.2.1 |
| dev.ludovic.netlib | blas | 2.2.1 |
| dev.ludovic.netlib | lapack | 2.2.1 |
| hive-2.3__hadoop-3.2 | jets3t-0.7 | liball_deps_2.12 |
| info.ganglia.gmetric4j | gmetric4j | 1.0.10 |
| io.airlift | aircompressor | 0.21 |
| io.delta | delta-sharing-spark_2.12 | 0.4.0 |
| io.dropwizard.metrics | metrics-core | 4.1.1 |
| io.dropwizard.metrics | metrics-graphite | 4.1.1 |
| io.dropwizard.metrics | metrics-healthchecks | 4.1.1 |
| io.dropwizard.metrics | metrics-jetty9 | 4.1.1 |
| io.dropwizard.metrics | metrics-jmx | 4.1.1 |
| io.dropwizard.metrics | metrics-json | 4.1.1 |
| io.dropwizard.metrics | metrics-jvm | 4.1.1 |
| io.dropwizard.metrics | metrics-servlets | 4.1.1 |
| io.netty | netty-all | 4.1.68.Final |
| io.prometheus | simpleclient | 0.7.0 |
| io.prometheus | simpleclient_common | 0.7.0 |
| io.prometheus | simpleclient_dropwizard | 0.7.0 |
| io.prometheus | simpleclient_pushgateway | 0.7.0 |
| io.prometheus | simpleclient_servlet | 0.7.0 |
| io.prometheus.jmx | collector | 0.12.0 |
| jakarta.annotation | jakarta.annotation-api | 1.3.5 |
| jakarta.servlet | jakarta.servlet-api | 4.0.3 |
| jakarta.validation | jakarta.validation-api | 2.0.2 |
| jakarta.ws.rs | jakarta.ws.rs-api | 2.1.6 |
| javax.activation | activation | 1.1.1 |
| javax.annotation | javax.annotation-api | 1.3.2 |
| javax.el | javax.el-api | 2.2.4 |
| javax.jdo | jdo-api | 3.0.1 |
| javax.transaction | jta | 1.1 |
| javax.transaction | transaction-api | 1.1 |
| javax.xml.bind | jaxb-api | 2.2.2 |
| javax.xml.stream | stax-api | 1.0-2 |
| javolution | javolution | 5.5.1 |
| jline | jline | 2.14.6 |
| joda-time | joda-time | 2.10.10 |
| log4j | apache-log4j-extras | 1.2.17 |
| log4j | log4j | 1.2.17 |
| maven-trees | hive-2.3__hadoop-3.2 | liball_deps_2.12 |
| net.java.dev.jna | jna | 5.8.0 |
| net.razorvine | pyrolite | 4.30 |
| net.sf.jpam | jpam | 1.1 |
| net.sf.opencsv | opencsv | 2.3 |
| net.sf.supercsv | super-csv | 2.2.0 |
| net.snowflake | snowflake-ingest-sdk | 0.9.6 |
| net.snowflake | snowflake-jdbc | 3.13.3 |
| net.snowflake | spark-snowflake_2.12 | 2.9.0-spark_3.1 |
| net.sourceforge.f2j | arpack_combined_all | 0.1 |
| org.acplt.remotetea | remotetea-oncrpc | 1.1.2 |
| org.antlr | ST4 | 4.0.4 |
| org.antlr | antlr-runtime | 3.5.2 |
| org.antlr | antlr4-runtime | 4.8 |
| org.antlr | stringtemplate | 3.2.1 |
| org.apache.ant | ant | 1.9.2 |
| org.apache.ant | ant-jsch | 1.9.2 |
| org.apache.ant | ant-launcher | 1.9.2 |
| org.apache.arrow | arrow-format | 2.0.0 |
| org.apache.arrow | arrow-memory-core | 2.0.0 |
| org.apache.arrow | arrow-memory-netty | 2.0.0 |
| org.apache.arrow | arrow-vector | 2.0.0 |
| org.apache.avro | avro | 1.10.2 |
| org.apache.avro | avro-ipc | 1.10.2 |
| org.apache.avro | avro-mapred | 1.10.2 |
| org.apache.commons | commons-compress | 1.21 |
| org.apache.commons | commons-crypto | 1.1.0 |
| org.apache.commons | commons-lang3 | 3.12.0 |
| org.apache.commons | commons-math3 | 3.4.1 |
| org.apache.commons | commons-text | 1.6 |
| org.apache.curator | curator-client | 2.13.0 |
| org.apache.curator | curator-framework | 2.13.0 |
| org.apache.curator | curator-recipes | 2.13.0 |
| org.apache.derby | derby | 10.14.2.0 |
| org.apache.hadoop | hadoop-client-api | 3.3.2-databricks |
| org.apache.hadoop | hadoop-client-runtime | 3.3.2 |
| org.apache.hive | hive-beeline | 2.3.9 |
| org.apache.hive | hive-cli | 2.3.9 |
| org.apache.hive | hive-jdbc | 2.3.9 |
| org.apache.hive | hive-llap-client | 2.3.9 |
| org.apache.hive | hive-llap-common | 2.3.9 |
| org.apache.hive | hive-serde | 2.3.9 |
| org.apache.hive | hive-shims | 2.3.9 |
| org.apache.hive | hive-storage-api | 2.7.2 |
| org.apache.hive.shims | hive-shims-0.23 | 2.3.9 |
| org.apache.hive.shims | hive-shims-common | 2.3.9 |
| org.apache.hive.shims | hive-shims-scheduler | 2.3.9 |
| org.apache.httpcomponents | httpclient | 4.5.13 |
| org.apache.httpcomponents | httpcore | 4.4.12 |
| org.apache.ivy | ivy | 2.5.0 |
| org.apache.mesos | mesos-shaded-protobuf | 1.4.0 |
| org.apache.orc | orc-core | 1.6.13 |
| org.apache.orc | orc-mapreduce | 1.6.13 |
| org.apache.orc | orc-shims | 1.6.13 |
| org.apache.parquet | parquet-column | 1.12.0-databricks-0004 |
| org.apache.parquet | parquet-common | 1.12.0-databricks-0004 |
| org.apache.parquet | parquet-encoding | 1.12.0-databricks-0004 |
| org.apache.parquet | parquet-format-structures | 1.12.0-databricks-0004 |
| org.apache.parquet | parquet-hadoop | 1.12.0-databricks-0004 |
| org.apache.parquet | parquet-jackson | 1.12.0-databricks-0004 |
| org.apache.thrift | libfb303 | 0.9.3 |
| org.apache.thrift | libthrift | 0.12.0 |
| org.apache.xbean | xbean-asm9-shaded | 4.20 |
| org.apache.yetus | audience-annotations | 0.5.0 |
| org.apache.zookeeper | zookeeper | 3.6.2 |
| org.apache.zookeeper | zookeeper-jute | 3.6.2 |
| org.checkerframework | checker-qual | 3.5.0 |
| org.codehaus.jackson | jackson-core-asl | 1.9.13 |
| org.codehaus.jackson | jackson-mapper-asl | 1.9.13 |
| org.codehaus.janino | commons-compiler | 3.0.16 |
| org.codehaus.janino | janino | 3.0.16 |
| org.datanucleus | datanucleus-api-jdo | 4.2.4 |
| org.datanucleus | datanucleus-core | 4.1.17 |
| org.datanucleus | datanucleus-rdbms | 4.1.19 |
| org.datanucleus | javax.jdo | 3.2.0-m3 |
| org.eclipse.jetty | jetty-client | 9.4.43.v20210629 |
| org.eclipse.jetty | jetty-continuation | 9.4.43.v20210629 |
| org.eclipse.jetty | jetty-http | 9.4.43.v20210629 |
| org.eclipse.jetty | jetty-io | 9.4.43.v20210629 |
| org.eclipse.jetty | jetty-jndi | 9.4.43.v20210629 |
| org.eclipse.jetty | jetty-plus | 9.4.43.v20210629 |
| org.eclipse.jetty | jetty-proxy | 9.4.43.v20210629 |
| org.eclipse.jetty | jetty-security | 9.4.43.v20210629 |
| org.eclipse.jetty | jetty-server | 9.4.43.v20210629 |
| org.eclipse.jetty | jetty-servlet | 9.4.43.v20210629 |
| org.eclipse.jetty | jetty-servlets | 9.4.43.v20210629 |
| org.eclipse.jetty | jetty-util | 9.4.43.v20210629 |
| org.eclipse.jetty | jetty-util-ajax | 9.4.43.v20210629 |
| org.eclipse.jetty | jetty-webapp | 9.4.43.v20210629 |
| org.eclipse.jetty | jetty-xml | 9.4.43.v20210629 |
| org.eclipse.jetty.websocket | websocket-api | 9.4.43.v20210629 |
| org.eclipse.jetty.websocket | websocket-client | 9.4.43.v20210629 |
| org.eclipse.jetty.websocket | websocket-common | 9.4.43.v20210629 |
| org.eclipse.jetty.websocket | websocket-server | 9.4.43.v20210629 |
| org.eclipse.jetty.websocket | websocket-servlet | 9.4.43.v20210629 |
| org.fusesource.leveldbjni | leveldbjni-all | 1.8 |
| org.glassfish.hk2 | hk2-api | 2.6.1 |
| org.glassfish.hk2 | hk2-locator | 2.6.1 |
| org.glassfish.hk2 | hk2-utils | 2.6.1 |
| org.glassfish.hk2 | osgi-resource-locator | 1.0.3 |
| org.glassfish.hk2.external | aopalliance-repackaged | 2.6.1 |
| org.glassfish.hk2.external | jakarta.inject | 2.6.1 |
| org.glassfish.jaxb | jaxb-runtime | 2.3.2 |
| org.glassfish.jersey.containers | jersey-container-servlet | 2.34 |
| org.glassfish.jersey.containers | jersey-container-servlet-core | 2.34 |
| org.glassfish.jersey.core | jersey-client | 2.34 |
| org.glassfish.jersey.core | jersey-common | 2.34 |
| org.glassfish.jersey.core | jersey-server | 2.34 |
| org.glassfish.jersey.inject | jersey-hk2 | 2.34 |
| org.hibernate.validator | hibernate-validator | 6.1.0.Final |
| org.javassist | javassist | 3.25.0-GA |
| org.jboss.logging | jboss-logging | 3.3.2.Final |
| org.jdbi | jdbi | 2.63.1 |
| org.jetbrains | annotations | 17.0.0 |
| org.joda | joda-convert | 1.7 |
| org.jodd | jodd-core | 3.5.2 |
| org.json4s | json4s-ast_2.12 | 3.7.0-M11 |
| org.json4s | json4s-core_2.12 | 3.7.0-M11 |
| org.json4s | json4s-jackson_2.12 | 3.7.0-M11 |
| org.json4s | json4s-scalap_2.12 | 3.7.0-M11 |
| org.lz4 | lz4-java | 1.7.1 |
| org.mariadb.jdbc | mariadb-java-client | 2.2.5 |
| org.objenesis | objenesis | 2.5.1 |
| org.postgresql | postgresql | 42.2.19 |
| org.roaringbitmap | RoaringBitmap | 0.9.14 |
| org.roaringbitmap | shims | 0.9.14 |
| org.rocksdb | rocksdbjni | 6.20.3 |
| org.rosuda.REngine | REngine | 2.1.0 |
| org.scala-lang | scala-compiler_2.12 | 2.12.14 |
| org.scala-lang | scala-library_2.12 | 2.12.14 |
| org.scala-lang | scala-reflect_2.12 | 2.12.14 |
| org.scala-lang.modules | scala-collection-compat_2.12 | 2.4.3 |
| org.scala-lang.modules | scala-parser-combinators_2.12 | 1.1.2 |
| org.scala-lang.modules | scala-xml_2.12 | 1.2.0 |
| org.scala-sbt | test-interface | 1.0 |
| org.scalacheck | scalacheck_2.12 | 1.14.2 |
| org.scalactic | scalactic_2.12 | 3.0.8 |
| org.scalanlp | breeze-macros_2.12 | 1.2 |
| org.scalanlp | breeze_2.12 | 1.2 |
| org.scalatest | scalatest_2.12 | 3.0.8 |
| org.slf4j | jcl-over-slf4j | 1.7.30 |
| org.slf4j | jul-to-slf4j | 1.7.30 |
| org.slf4j | slf4j-api | 1.7.30 |
| org.slf4j | slf4j-log4j12 | 1.7.30 |
| org.spark-project.spark | unused | 1.0.0 |
| org.threeten | threeten-extra | 1.5.0 |
| org.tukaani | xz | 1.8 |
| org.typelevel | algebra_2.12 | 2.0.1 |
| org.typelevel | cats-kernel_2.12 | 2.1.1 |
| org.typelevel | macro-compat_2.12 | 1.1.1 |
| org.typelevel | spire-macros_2.12 | 0.17.0 |
| org.typelevel | spire-platform_2.12 | 0.17.0 |
| org.typelevel | spire-util_2.12 | 0.17.0 |
| org.typelevel | spire_2.12 | 0.17.0 |
| org.wildfly.openssl | wildfly-openssl | 1.0.7.Final |
| org.xerial | sqlite-jdbc | 3.8.11.2 |
| org.xerial.snappy | snappy-java | 1.1.8.4 |
| org.yaml | snakeyaml | 1.24 |
| oro | oro | 2.0.8 |
| pl.edu.icm | JLargeArrays | 1.5 |
| software.amazon.ion | ion-java | 1.0.2 |
| stax | stax-api | 1.0.1 |