Databricks Runtime 11.3 LTS

  • Article

The following release notes provide information about Databricks Runtime 11.3 LTS, powered by Apache Spark 3.3.0. Databricks released these images in October 2022.

Note

LTS means this version is under long-term support. See Databricks Runtime LTS version lifecycle.

New features and improvements

Structured Streaming trigger once is deprecated

The Trigger.Once setting has been deprecated. Databricks recommends you use Trigger.AvailableNow. See Configure Structured Streaming trigger intervals.

Change source path for Auto Loader

You can now change the directory input path for Auto Loader configured with directory listing mode without having to choose a new checkpoint directory. See Change source path for Auto Loader.

Databricks Kinesis connector now supports reading from Kinesis Data streams in EFO mode

You can now use the Databricks Kinesis structured streaming source in Databricks Runtime 11.3 LTS to run queries that read from Kinesis Data streams in enhanced fan-out mode. This allows dedicated throughput per shard, per consumer and record delivery in push mode.

New H3 geospatial functions and added Photon support for all H3 functions

Introducing 4 new H3 functions, h3_maxchild, h3_minchild, h3_pointash3, and h3_pointash3string. These functions are available in SQL, Scala, and Python. All H3 expressions are now supported in Photon. See H3 geospatial functions.

New features for Predictive I/O

Photon supports range mode for running frames, using RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW. Photon also supports range mode for growing frames, using RANGE BETWEEN UNBOUNDED PRECEDING AND offset_stop { PRECEDING | FOLLOWING }.

Increasing initial partitions to scan for selective queries

The value of initial partitions to scan has been increased to 10 for selective query with take/tail/limit in Photon-enabled clusters and LIMIT in Databricks SQL. With 10 partitions, you can avoid the overhead of launching multiple small jobs and a slow scale-up. You can also configure this through spark.sql.limit.selectiveInitialNumPartitions.

New AQE plan versions visualization

Introducing AQE plan versions that allows you to visualize your runtime plan updates from adaptive query execution (AQE).

New asynchronous progress tracking and log purging modes

Introducing Structured Streaming modes called asynchronous progress tracking and asynchronous log purging. Asynchronous log purging mode lowers the latency of streaming queries by removing logs used for progress tracking in the background.

Structured Streaming on Unity Catalog now supports display()

You can now use display() when you use Structured Streaming to work with tables registered in Unity Catalog.

Pipeline events are now logged in JSON format

Azure Databricks now writes pipeline events to the driver log in JSON format. While each event will be JSON-parseable, large events may not contain all fields, or the fields may be truncated. Each event is logged in a single line with the prefix Event received: . The following is an example event.

Event received: {"id":"some-event-id","origin":{"pipeline_id":"some-pipeline-id","cluster_id":"some-cluster id"},"message":"simple [truncated] message","level":"WARN"}

Arbitrary Stateful Processing in Structured Streaming with Python

Introducing the applyInPandasWithState function that can be used to perform arbitrary stateful processing in PySpark. This is equivalent to the flatMapGroupsWithState function in the Java API.

Date inference in CSV files

Introducing improved inference of date type columns in CSV files. When the date format is consistent across the records for a column, those columns can be inferred as DateType. You can also have a combination of date formats across different columns. Azure Databricks can automatically infer the date format for each column. Date columns in CSV files prior to Databricks Runtime 11.3 LTS are left as StringType.

Clone support for Apache Parquet and Apache Iceberg tables (Public Preview)

Clone can now be used to create and incrementally update Delta tables that mirror Apache Parquet and Apache Iceberg tables. You can update your source Parquet table and incrementally apply the changes to their cloned Delta table with the clone command. See Incrementally clone Parquet and Iceberg tables to Delta Lake.

Use SQL to specify schema- and catalog-level storage locations for Unity Catalog managed tables

You can now use the MANAGED LOCATION SQL command to specify a cloud storage location for managed tables at the catalog and schema levels. See CREATE CATALOG and CREATE SCHEMA.

Behavior changes

Databricks Connect 11.3.2

Databricks Connect client update 11.3.2 is now supported. See Databricks Connect and Databricks Connect release notes.

Updated Azure Databricks Snowflake connector

The Azure Databricks Snowflake connector has been updated to the latest version of code from the open-source repository, Snowflake Data Source for Apache Spark. It is now fully compatible with Databricks Runtime 11.3 LTS, including predicate pushdown and internal query plan pushdown while maintaining all of the features of the open-source version.

Hadoop cache for S3A is now disabled

The Hadoop cache (FileSystem Apache Hadoop Main 3.3.4 API) for S3A is now disabled. This is to align with other cloud storage connectors. For workloads that rely on file system caching, make sure that newly created file systems are supplied with the correct Hadoop configurations, including credential providers.

Delta Lake stats collection schema now matches column order in table schema definition

This change addresses a bug in the Delta Lake protocol where stats were not collected for columns due to a mismatch in DataFrame and table column ordering. In some cases, you might encounter write performance degradation due to stats collection on previously untracked fields. See Data skipping for Delta Lake.

applyInPandasWithState throws an error if the query has a shuffle after the operator

The operator applyInPandasWithState throws an error if the query has shuffle after the operator. This happens when either the user adds shuffle after the operation, or the optimizer or sink adds shuffle implicitly.

Library upgrades

  • Upgraded Python libraries:
    • distlib from 0.3.5 to 0.3.6
  • Upgraded R libraries:
    • broom from 1.0.0 to 1.0.1
    • callr from 3.7.1 to 3.7.2
    • dplyr from 1.0.9 to 1.0.10
    • dtplyr from 1.2.1 to 1.2.2
    • forcats from 0.5.1 to 0.5.2
    • future from 1.27.0 to 1.28.0
    • future.apply from 1.9.0 to 1.9.1
    • gert from 1.7.0 to 1.8.0
    • globals from 0.16.0 to 0.16.1
    • gtable from 0.3.0 to 0.3.1
    • haven from 2.5.0 to 2.5.1
    • hms from 1.1.1 to 1.1.2
    • httr from 1.4.3 to 1.4.4
    • knitr from 1.39 to 1.40
    • modelr from 0.1.8 to 0.1.9
    • pillar from 1.8.0 to 1.8.1
    • progressr from 0.10.1 to 0.11.0
    • readxl from 1.4.0 to 1.4.1
    • reprex from 2.0.1 to 2.0.2
    • rlang from 1.0.4 to 1.0.5
    • rmarkdown from 2.14 to 2.16
    • RSQLite from 2.2.15 to 2.2.16
    • rstudioapi from 0.13 to 0.14
    • rversions from 2.1.1 to 2.1.2
    • rvest from 1.0.2 to 1.0.3
    • scales from 1.2.0 to 1.2.1
    • sparklyr from 1.7.7 to 1.7.8
    • stringr from 1.4.0 to 1.4.1
    • survival from 3.2-13 to 3.4-0
    • tinytex from 0.40 to 0.41
    • viridisLite from 0.4.0 to 0.4.1
  • Upgraded Java libraries:
    • com.fasterxml.jackson.core.jackson-annotations from 2.13.3 to 2.13.4
    • com.fasterxml.jackson.core.jackson-core from 2.13.3 to 2.13.4
    • com.fasterxml.jackson.core.jackson-databind from 2.13.3 to 2.13.4
    • com.fasterxml.jackson.dataformat.jackson-dataformat-cbor from 2.13.3 to 2.13.4
    • com.fasterxml.jackson.datatype.jackson-datatype-joda from 2.13.3 to 2.13.4
    • com.fasterxml.jackson.datatype.jackson-datatype-jsr310 from 2.13.3 to 2.13.4
    • com.fasterxml.jackson.module.jackson-module-paranamer from 2.13.3 to 2.13.4
    • com.fasterxml.jackson.module.jackson-module-scala_2.12 from 2.13.3 to 2.13.4
    • org.apache.hadoop.hadoop-client-api from 3.3.2-databricks to 3.3.4-databricks
    • org.apache.hadoop.hadoop-client-runtime from 3.3.2 to 3.3.4
    • org.apache.orc.orc-core from 1.7.5 to 1.7.6
    • org.apache.orc.orc-mapreduce from 1.7.5 to 1.7.6
    • org.apache.orc.orc-shims from 1.7.5 to 1.7.6
    • org.apache.parquet.parquet-column from 1.12.0-databricks-0004 to 1.12.0-databricks-0007
    • org.apache.parquet.parquet-common from 1.12.0-databricks-0004 to 1.12.0-databricks-0007
    • org.apache.parquet.parquet-encoding from 1.12.0-databricks-0004 to 1.12.0-databricks-0007
    • org.apache.parquet.parquet-format-structures from 1.12.0-databricks-0004 to 1.12.0-databricks-0007
    • org.apache.parquet.parquet-hadoop from 1.12.0-databricks-0004 to 1.12.0-databricks-0007
    • org.apache.parquet.parquet-jackson from 1.12.0-databricks-0004 to 1.12.0-databricks-0007
    • org.glassfish.jersey.containers.jersey-container-servlet from 2.34 to 2.36
    • org.glassfish.jersey.containers.jersey-container-servlet-core from 2.34 to 2.36
    • org.glassfish.jersey.core.jersey-client from 2.34 to 2.36
    • org.glassfish.jersey.core.jersey-common from 2.34 to 2.36
    • org.glassfish.jersey.core.jersey-server from 2.34 to 2.36
    • org.glassfish.jersey.inject.jersey-hk2 from 2.34 to 2.36

Apache Spark

Databricks Runtime 11.3 LTS includes Apache Spark 3.3.0. This release includes all Spark fixes and improvements included in Databricks Runtime 11.2 (unsupported), as well as the following additional bug fixes and improvements made to Spark:

  • [SPARK-39957] [WARMFIX][SC-111425][CORE] Delay onDisconnected to enable Driver receives ExecutorExitCode
  • [SPARK-39955] [WARMFIX][SC-111424][CORE] Improve LaunchTask process to avoid Stage failures caused by fail-to-send LaunchTask messages
  • [SPARK-40474] [SC-106248][Cherry-Pick] Correct CSV schema inference behavior for datetime columns and introduce auto detection for Date fields
  • [SPARK-40535] [SC-111243][SQL] Fix bug the buffer of AggregatingAccumulator will not be created if the input rows is empty
  • [SPARK-40434] [SC-111125][SC-111144][SC-111138][SPARK-40435][11.3][SS][PYTHON] Implement applyInPandasWithState in PySpark
  • [SPARK-40460] [SC-110832][SS] Fix streaming metrics when selecting _metadata
  • [SPARK-40324] [SC-109943][SQL] Provide a query context of ParseException
  • [SPARK-40466] [SC-110899][SS] Improve the error message when DSv2 is disabled while DSv1 is not available
  • [SPARK-40456] [SC-110848][SQL] PartitionIterator.hasNext should be cheap to call repeatedly
  • [SPARK-40169] [SC-110772][SQL] Don't pushdown Parquet filters with no reference to data schema
  • [SPARK-40467] [SC-110759][SS] Split FlatMapGroupsWithState down to multiple test suites
  • [SPARK-40468] [SC-110813][SQL] Fix column pruning in CSV when _corrupt_record is selected
  • [SPARK-40291] [SC-110085][SQL] Improve the message for column not in group by clause error
  • [SPARK-40398] [SC-110762][CORE][SQL] Use Loop instead of Arrays.stream api
  • [SPARK-40433] [SC-110684][SS][PYTHON] Add toJVMRow in PythonSQLUtils to convert pickled PySpark Row to JVM Row
  • [SPARK-40414] [SC-110568][SQL][PYTHON] More generic type on PythonArrowInput and PythonArrowOutput
  • [SPARK-40352] [SC-109945][SQL] Add function aliases: len, datepart, dateadd, date_diff and curdate
  • [SPARK-40470] [SC-110761][SQL] Handle GetArrayStructFields and GetMapValue in "arrays_zip" function
  • [SPARK-40387] [SC-110685][SQL] Improve the implementation of Spark Decimal
  • [SPARK-40429] [SC-110675][SQL] Only set KeyGroupedPartitioning when the referenced column is in the output
  • [SPARK-40432] [SC-110716][SS][PYTHON] Introduce GroupStateImpl and GroupStateTimeout in PySpark
  • [SPARK-39915] [SC-110496][SQL] Ensure the output partitioning is user-specified in AQE
  • [SPARK-29260] [SQL] Support ALTER DATABASE SET LOCATION if HMS supports
  • [SPARK-40185] [SC-110056][SQL] Remove column suggestion when the candidate list is empty
  • [SPARK-40362] [SC-110401][SQL] Fix BinaryComparison canonicalization
  • [SPARK-40411] [SC-110381][SS] Refactor FlatMapGroupsWithStateExec to have a parent trait
  • [SPARK-40293] [SC-110084][SQL] Make the V2 table error message more meaningful
  • [SPARK-38734] [SC-110383][SQL] Remove the error class INDEX_OUT_OF_BOUNDS
  • [SPARK-40292] [SC-110300][SQL] Fix column names in "arrays_zip" function when arrays are referenced from nested structs
  • [SPARK-40276] [SC-109674][CORE] Reduce the result size of RDD.takeOrdered
  • [SPARK-40197] [SC-109176][SQL] Replace query plan with context for MULTI_VALUE_SUBQUERY_ERROR
  • [SPARK-40300] [SC-109942][SQL] Migrate onto the DATATYPE_MISMATCH error class
  • [SPARK-40149] [SC-110055][SQL] Propagate metadata columns through Project
  • [SPARK-40280] [SC-110146][SQL] Add support for parquet push down for annotated int and long
  • [SPARK-40220] [SC-110143][SC-109175][SQL] Don't output the empty map of error message parameters
  • [SPARK-40295] [SC-110070][SQL] Allow v2 functions with literal args in write distribution/ordering
  • [SPARK-40156] [SC-109264][SQL] url_decode() should the return an error class
  • [SPARK-39195] [SQL] Spark OutputCommitCoordinator should abort stage when committed file not consistent with task status
  • [SPARK-40260] [SC-109424][SQL] Use error classes in the compilation errors of GROUP BY a position
  • [SPARK-40205] [SC-110144][SC-109082][SQL] Provide a query context of ELEMENT_AT_BY_INDEX_ZERO
  • [SPARK-40112] [SC-109676][SQL] Improve the TO_BINARY() function
  • [SPARK-40209] [SC-109081][SQL] Don't change the interval value of Decimal in changePrecision() on errors
  • [SPARK-40319] [SC-109873][SQL] Remove duplicated query execution error method for PARSE_DATETIME_BY_NEW_PARSER
  • [SPARK-40222] [SC-109209][SQL] Numeric try_add/try_divide/try_subtract/try_multiply should throw error from their children
  • [SPARK-40183] [SC-108907][SQL] Use error class NUMERIC_VALUE_OUT_OF_RANGE for overflow in decimal conversion
  • [SPARK-40180] [SC-109069][SQL] Format error messages by spark-sql
  • [SPARK-40153] [SC-109165][SQL] Unify resolve functions and table-valued functions
  • [SPARK-40308] [SC-109880][SQL] Allow non-foldable delimiter arguments to str_to_map function
  • [SPARK-40219] [SC-110052][SC-109663][SQL] Resolved view logical plan should hold the schema to avoid redundant lookup
  • [SPARK-40098] [SC-109939][SC-108693][SQL] Format error messages in the Thrift Server
  • [SPARK-39917] [SC-109038][SQL] Use different error classes for numeric/interval arithmetic overflow
  • [SPARK-40033] [SC-109875][SQL] Nested schema pruning support through element_at
  • [SPARK-40194] [SC-109660][SQL] SPLIT function on empty regex should truncate trailing empty string.
  • [SPARK-40228] [SC-109835][SQL] Do not simplify multiLike if child is not a cheap expression
  • [SPARK-40039] [SC-109896][SC-109260][SS] Introducing a streaming checkpoint file manager based on Hadoop's Abortable interface
  • [SPARK-40285] [SC-109679][SQL] Simplify the roundTo[Numeric] for Spark Decimal
  • [SPARK-39896] [SC-109658][SQL] UnwrapCastInBinaryComparison should work when the literal of In/InSet downcast failed
  • [SPARK-40040] [SC-109662][SQL] Push local limit to both sides if join condition is empty
  • [SPARK-40055] [SC-109075][SQL] listCatalogs should also return spark_catalog even when spark_catalog implementation is defaultSessionCatalog
  • [SPARK-39915] [SC-109391][SQL] Dataset.repartition(N) may not create N partitions Non-AQE part
  • [SPARK-40207] [SC-109401][SQL] Specify the column name when the data type is not supported by datasource
  • [SPARK-40245] [SC-109295][SQL] Fix FileScan equality check when partition or data filter columns are not read
  • [SPARK-40113] [SC-109405][SQL] Reactor ParquetScanBuilder DataSourceV2 interface implementations
  • [SPARK-40211] [SC-109226][CORE][SQL] Allow customize initial partitions number in take() behavior
  • [SPARK-40252] [SC-109379][SQL] Replace Stream.collect(Collectors.joining) with StringJoiner Api
  • [SPARK-40247] [SC-109272][SQL] Fix BitSet equality check
  • [SPARK-40067] [SQL] Use Table#name() instead of Scan#name() to populate the table name in the BatchScan node in SparkUI
  • [SPARK-39966] [SQL] Use V2 Filter in SupportsDelete
  • [SPARK-39607] [SC-109268][SQL][DSV2] Distribution and ordering support V2 function in writing
  • [SPARK-40224] [SC-109271][SQL] Make ObjectHashAggregateExec release memory eagerly when fallback to sort-based
  • [SPARK-40013] [SQL] DS V2 expressions should have the default toString
  • [SPARK-40214] [SC-109079][PYTHON][SQL] add 'get' to functions
  • [SPARK-40192] [SC-109089][SQL][ML] Remove redundant groupby
  • [SPARK-40146] [SC-108694][SQL] Simply the codegen of getting map value
  • [SPARK-40109] [SQL] New SQL function: get()
  • [SPARK-39929] [SQL] DS V2 supports push down string functions(non ANSI)
  • [SPARK-39819] [SQL] DS V2 aggregate push down can work with Top N or Paging (Sort with expressions)
  • [SPARK-40213] [SC-109077][SQL] Support ASCII value conversion for Latin-1 characters
  • [SPARK-39887] [SQL] RemoveRedundantAliases should keep aliases that make the output of projection nodes unique
  • [SPARK-39764] [SQL] Make PhysicalOperation the same as ScanOperation
  • [SPARK-39964] [SQL] DS V2 pushdown should unify the translate path
  • [SPARK-39528] [SQL] Use V2 Filter in SupportsRuntimeFiltering
  • [SPARK-40066] [SQL] ANSI mode: always return null on invalid access to map column
  • [SPARK-39912] [SPARK-39828][SQL] Refine CatalogImpl
  • [SPARK-39833] [SC-108736][SQL] Disable Parquet column index in DSv1 to fix a correctness issue in the case of overlapping partition and data columns
  • [SPARK-39880] [SQL] V2 SHOW FUNCTIONS command should print qualified function name like v1
  • [SPARK-39767] [SQL] Remove UnresolvedDBObjectName and add UnresolvedIdentifier
  • [SPARK-40163] [SC-108740][SQL] feat: SparkSession.config(Map)
  • [SPARK-40136] [SQL] Fix the fragment of SQL query contexts
  • [SPARK-40107] [SC-108689][SQL] Pull out empty2null conversion from FileFormatWriter
  • [SPARK-40121] [PYTHON][SQL] Initialize projection used for Python UDF
  • [SPARK-40128] [SQL] Make the VectorizedColumnReader recognize DELTA_LENGTH_BYTE_ARRAY as a standalone column encoding
  • [SPARK-40132] [ML] Restore rawPredictionCol to MultilayerPerceptronClassifier.setParams
  • [SPARK-40050] [SC-108696][SQL] Enhance EliminateSorts to support removing sorts via LocalLimit
  • [SPARK-39629] [SQL] Support v2 SHOW FUNCTIONS
  • [SPARK-39925] [SC-108734][SQL] Add array_sort(column, comparator) overload to DataFrame operations
  • [SPARK-40117] [PYTHON][SQL] Convert condition to java in DataFrameWriterV2.overwrite
  • [SPARK-40105] [SQL] Improve repartition in ReplaceCTERefWithRepartition
  • [SPARK-39503] [SQL] Add session catalog name for v1 database table and function
  • [SPARK-39889] [SQL] Use different error classes for numeric/interval divided by 0
  • [SPARK-39741] [SQL] Support url encode/decode as built-in function and tidy up url-related functions
  • [SPARK-40102] [SQL] Use SparkException instead of IllegalStateException in SparkPlan
  • [SPARK-40014] [SQL] Support cast of decimals to ANSI intervals
  • [SPARK-39776] [SQL][FOLLOW] Update UT of PlanStabilitySuite in ANSI mode
  • [SPARK-39963] [SQL] Simplify SimplifyCasts.isWiderCast

Maintenance updates

See Databricks Runtime 11.3 maintenance updates.

System environment

  • Operating System: Ubuntu 20.04.5 LTS
  • Java: Zulu 8.56.0.21-CA-linux64
  • Scala: 2.12.14
  • Python: 3.9.5
  • R: 4.1.3
  • Delta Lake: 2.1.0

Installed Python libraries

Library Version Library Version Library Version
argon2-cffi 20.1.0 async-generator 1.10 attrs 21.2.0
backcall 0.2.0 backports.entry-points-selectable 1.1.1 black 22.3.0
bleach 4.0.0 boto3 1.21.18 botocore 1.24.18
certifi 2021.10.8 cffi 1.14.6 chardet 4.0.0
charset-normalizer 2.0.4 click 8.0.3 cryptography 3.4.8
cycler 0.10.0 Cython 0.29.24 dbus-python 1.2.16
debugpy 1.4.1 decorator 5.1.0 defusedxml 0.7.1
distlib 0.3.6 entrypoints 0.3 facets-overview 1.0.0
filelock 3.8.0 idna 3.2 ipykernel 6.12.1
ipython 7.32.0 ipython-genutils 0.2.0 ipywidgets 7.7.0
jedi 0.18.0 Jinja2 2.11.3 jmespath 0.10.0
joblib 1.0.1 jsonschema 3.2.0 jupyter-client 6.1.12
jupyter-core 4.8.1 jupyterlab-pygments 0.1.2 jupyterlab-widgets 1.0.0
kiwisolver 1.3.1 MarkupSafe 2.0.1 matplotlib 3.4.3
matplotlib-inline 0.1.2 mistune 0.8.4 mypy-extensions 0.4.3
nbclient 0.5.3 nbconvert 6.1.0 nbformat 5.1.3
nest-asyncio 1.5.1 notebook 6.4.5 numpy 1.20.3
packaging 21.0 pandas 1.3.4 pandocfilters 1.4.3
parso 0.8.2 pathspec 0.9.0 patsy 0.5.2
pexpect 4.8.0 pickleshare 0.7.5 Pillow 8.4.0
pip 21.2.4 platformdirs 2.5.2 plotly 5.9.0
prometheus-client 0.11.0 prompt-toolkit 3.0.20 protobuf 4.21.5
psutil 5.8.0 psycopg2 2.9.3 ptyprocess 0.7.0
pyarrow 7.0.0 pycparser 2.20 Pygments 2.10.0
PyGObject 3.36.0 pyodbc 4.0.31 pyparsing 3.0.4
pyrsistent 0.18.0 python-dateutil 2.8.2 pytz 2021.3
pyzmq 22.2.1 requests 2.26.0 requests-unixsocket 0.2.0
s3transfer 0.5.2 scikit-learn 0.24.2 scipy 1.7.1
seaborn 0.11.2 Send2Trash 1.8.0 setuptools 58.0.4
six 1.16.0 ssh-import-id 5.10 statsmodels 0.12.2
tenacity 8.0.1 terminado 0.9.4 testpath 0.5.0
threadpoolctl 2.2.0 tokenize-rt 4.2.1 tomli 2.0.1
tornado 6.1 traitlets 5.1.0 typing-extensions 3.10.0.2
unattended-upgrades 0.1 urllib3 1.26.7 virtualenv 20.8.0
wcwidth 0.2.5 webencodings 0.5.1 wheel 0.37.0
widgetsnbextension 3.6.0

Installed R libraries

R libraries are installed from the Microsoft CRAN snapshot on 2022-09-08 .

Library Version Library Version Library Version
askpass 1.1 assertthat 0.2.1 backports 1.4.1
base 4.1.3 base64enc 0.1-3 bit 4.0.4
bit64 4.0.5 blob 1.2.3 boot 1.3-28
brew 1.0-7 brio 1.1.3 broom 1.0.1
bslib 0.4.0 cachem 1.0.6 callr 3.7.2
caret 6.0-93 cellranger 1.1.0 chron 2.3-57
class 7.3-20 cli 3.3.0 clipr 0.8.0
cluster 2.1.3 codetools 0.2-18 colorspace 2.0-3
commonmark 1.8.0 compiler 4.1.3 config 0.3.1
cpp11 0.4.2 crayon 1.5.1 credentials 1.3.2
curl 4.3.2 data.table 1.14.2 datasets 4.1.3
DBI 1.1.3 dbplyr 2.2.1 desc 1.4.1
devtools 2.4.4 diffobj 0.3.5 digest 0.6.29
downlit 0.4.2 dplyr 1.0.10 dtplyr 1.2.2
e1071 1.7-11 ellipsis 0.3.2 evaluate 0.16
fansi 1.0.3 farver 2.1.1 fastmap 1.1.0
fontawesome 0.3.0 forcats 0.5.2 foreach 1.5.2
foreign 0.8-82 forge 0.2.0 fs 1.5.2
future 1.28.0 future.apply 1.9.1 gargle 1.2.0
generics 0.1.3 gert 1.8.0 ggplot2 3.3.6
gh 1.3.0 gitcreds 0.1.1 glmnet 4.1-4
globals 0.16.1 glue 1.6.2 googledrive 2.0.0
googlesheets4 1.0.1 gower 1.0.0 graphics 4.1.3
grDevices 4.1.3 grid 4.1.3 gridExtra 2.3
gsubfn 0.7 gtable 0.3.1 hardhat 1.2.0
haven 2.5.1 highr 0.9 hms 1.1.2
htmltools 0.5.3 htmlwidgets 1.5.4 httpuv 1.6.5
httr 1.4.4 ids 1.0.1 ini 0.3.1
ipred 0.9-13 isoband 0.2.5 iterators 1.0.14
jquerylib 0.1.4 jsonlite 1.8.0 KernSmooth 2.23-20
knitr 1.40 labeling 0.4.2 later 1.3.0
lattice 0.20-45 lava 1.6.10 lifecycle 1.0.1
listenv 0.8.0 lubridate 1.8.0 magrittr 2.0.3
markdown 1.1 MASS 7.3-56 Matrix 1.4-1
memoise 2.0.1 methods 4.1.3 mgcv 1.8-40
mime 0.12 miniUI 0.1.1.1 ModelMetrics 1.2.2.2
modelr 0.1.9 munsell 0.5.0 nlme 3.1-157
nnet 7.3-17 numDeriv 2016.8-1.1 openssl 2.0.2
parallel 4.1.3 parallelly 1.32.1 pillar 1.8.1
pkgbuild 1.3.1 pkgconfig 2.0.3 pkgdown 2.0.6
pkgload 1.3.0 plogr 0.2.0 plyr 1.8.7
praise 1.0.0 prettyunits 1.1.1 pROC 1.18.0
processx 3.7.0 prodlim 2019.11.13 profvis 0.3.7
progress 1.2.2 progressr 0.11.0 promises 1.2.0.1
proto 1.0.0 proxy 0.4-27 ps 1.7.1
purrr 0.3.4 r2d3 0.2.6 R6 2.5.1
ragg 1.2.2 randomForest 4.7-1.1 rappdirs 0.3.3
rcmdcheck 1.4.0 RColorBrewer 1.1-3 Rcpp 1.0.9
RcppEigen 0.3.3.9.2 readr 2.1.2 readxl 1.4.1
recipes 1.0.1 rematch 1.0.1 rematch2 2.1.2
remotes 2.4.2 reprex 2.0.2 reshape2 1.4.4
rlang 1.0.5 rmarkdown 2.16 RODBC 1.3-19
roxygen2 7.2.1 rpart 4.1.16 rprojroot 2.0.3
Rserve 1.8-11 RSQLite 2.2.16 rstudioapi 0.14
rversions 2.1.2 rvest 1.0.3 sass 0.4.2
scales 1.2.1 selectr 0.4-2 sessioninfo 1.2.2
shape 1.4.6 shiny 1.7.2 sourcetools 0.1.7
sparklyr 1.7.8 SparkR 3.3.0 spatial 7.3-11
splines 4.1.3 sqldf 0.4-11 SQUAREM 2021.1
stats 4.1.3 stats4 4.1.3 stringi 1.7.8
stringr 1.4.1 survival 3.4-0 sys 3.4
systemfonts 1.0.4 tcltk 4.1.3 testthat 3.1.4
textshaping 0.3.6 tibble 3.1.8 tidyr 1.2.0
tidyselect 1.1.2 tidyverse 1.3.2 timeDate 4021.104
tinytex 0.41 tools 4.1.3 tzdb 0.3.0
urlchecker 1.0.1 usethis 2.1.6 utf8 1.2.2
utils 4.1.3 uuid 1.1-0 vctrs 0.4.1
viridisLite 0.4.1 vroom 1.5.7 waldo 0.4.0
whisker 0.4 withr 2.5.0 xfun 0.32
xml2 1.3.3 xopen 1.0.0 xtable 1.8-4
yaml 2.3.5 zip 2.2.0

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID Artifact ID Version
antlr antlr 2.7.7
com.amazonaws amazon-kinesis-client 1.12.0
com.amazonaws aws-java-sdk-autoscaling 1.12.189
com.amazonaws aws-java-sdk-cloudformation 1.12.189
com.amazonaws aws-java-sdk-cloudfront 1.12.189
com.amazonaws aws-java-sdk-cloudhsm 1.12.189
com.amazonaws aws-java-sdk-cloudsearch 1.12.189
com.amazonaws aws-java-sdk-cloudtrail 1.12.189
com.amazonaws aws-java-sdk-cloudwatch 1.12.189
com.amazonaws aws-java-sdk-cloudwatchmetrics 1.12.189
com.amazonaws aws-java-sdk-codedeploy 1.12.189
com.amazonaws aws-java-sdk-cognitoidentity 1.12.189
com.amazonaws aws-java-sdk-cognitosync 1.12.189
com.amazonaws aws-java-sdk-config 1.12.189
com.amazonaws aws-java-sdk-core 1.12.189
com.amazonaws aws-java-sdk-datapipeline 1.12.189
com.amazonaws aws-java-sdk-directconnect 1.12.189
com.amazonaws aws-java-sdk-directory 1.12.189
com.amazonaws aws-java-sdk-dynamodb 1.12.189
com.amazonaws aws-java-sdk-ec2 1.12.189
com.amazonaws aws-java-sdk-ecs 1.12.189
com.amazonaws aws-java-sdk-efs 1.12.189
com.amazonaws aws-java-sdk-elasticache 1.12.189
com.amazonaws aws-java-sdk-elasticbeanstalk 1.12.189
com.amazonaws aws-java-sdk-elasticloadbalancing 1.12.189
com.amazonaws aws-java-sdk-elastictranscoder 1.12.189
com.amazonaws aws-java-sdk-emr 1.12.189
com.amazonaws aws-java-sdk-glacier 1.12.189
com.amazonaws aws-java-sdk-glue 1.12.189
com.amazonaws aws-java-sdk-iam 1.12.189
com.amazonaws aws-java-sdk-importexport 1.12.189
com.amazonaws aws-java-sdk-kinesis 1.12.189
com.amazonaws aws-java-sdk-kms 1.12.189
com.amazonaws aws-java-sdk-lambda 1.12.189
com.amazonaws aws-java-sdk-logs 1.12.189
com.amazonaws aws-java-sdk-machinelearning 1.12.189
com.amazonaws aws-java-sdk-opsworks 1.12.189
com.amazonaws aws-java-sdk-rds 1.12.189
com.amazonaws aws-java-sdk-redshift 1.12.189
com.amazonaws aws-java-sdk-route53 1.12.189
com.amazonaws aws-java-sdk-s3 1.12.189
com.amazonaws aws-java-sdk-ses 1.12.189
com.amazonaws aws-java-sdk-simpledb 1.12.189
com.amazonaws aws-java-sdk-simpleworkflow 1.12.189
com.amazonaws aws-java-sdk-sns 1.12.189
com.amazonaws aws-java-sdk-sqs 1.12.189
com.amazonaws aws-java-sdk-ssm 1.12.189
com.amazonaws aws-java-sdk-storagegateway 1.12.189
com.amazonaws aws-java-sdk-sts 1.12.189
com.amazonaws aws-java-sdk-support 1.12.189
com.amazonaws aws-java-sdk-swf-libraries 1.11.22
com.amazonaws aws-java-sdk-workspaces 1.12.189
com.amazonaws jmespath-java 1.12.189
com.chuusai shapeless_2.12 2.3.3
com.clearspring.analytics stream 2.9.6
com.databricks Rserve 1.8-3
com.databricks jets3t 0.7.1-0
com.databricks.scalapb compilerplugin_2.12 0.4.15-10
com.databricks.scalapb scalapb-runtime_2.12 0.4.15-10
com.esotericsoftware kryo-shaded 4.0.2
com.esotericsoftware minlog 1.3.0
com.fasterxml classmate 1.3.4
com.fasterxml.jackson.core jackson-annotations 2.13.4
com.fasterxml.jackson.core jackson-core 2.13.4
com.fasterxml.jackson.core jackson-databind 2.13.4
com.fasterxml.jackson.dataformat jackson-dataformat-cbor 2.13.4
com.fasterxml.jackson.datatype jackson-datatype-joda 2.13.4
com.fasterxml.jackson.datatype jackson-datatype-jsr310 2.13.4
com.fasterxml.jackson.module jackson-module-paranamer 2.13.4
com.fasterxml.jackson.module jackson-module-scala_2.12 2.13.4
com.github.ben-manes.caffeine caffeine 2.3.4
com.github.fommil jniloader 1.1
com.github.fommil.netlib core 1.1.2
com.github.fommil.netlib native_ref-java 1.1
com.github.fommil.netlib native_ref-java-natives 1.1
com.github.fommil.netlib native_system-java 1.1
com.github.fommil.netlib native_system-java-natives 1.1
com.github.fommil.netlib netlib-native_ref-linux-x86_64-natives 1.1
com.github.fommil.netlib netlib-native_system-linux-x86_64-natives 1.1
com.github.luben zstd-jni 1.5.2-1
com.github.wendykierp JTransforms 3.1
com.google.code.findbugs jsr305 3.0.0
com.google.code.gson gson 2.8.6
com.google.crypto.tink tink 1.6.1
com.google.flatbuffers flatbuffers-java 1.12.0
com.google.guava guava 15.0
com.google.protobuf protobuf-java 2.6.1
com.h2database h2 2.0.204
com.helger profiler 1.1.1
com.jcraft jsch 0.1.50
com.jolbox bonecp 0.8.0.RELEASE
com.lihaoyi sourcecode_2.12 0.1.9
com.microsoft.azure azure-data-lake-store-sdk 2.3.9
com.microsoft.sqlserver mssql-jdbc 9.2.1.jre8
com.ning compress-lzf 1.1
com.sun.mail javax.mail 1.5.2
com.tdunning json 1.8
com.thoughtworks.paranamer paranamer 2.8
com.trueaccord.lenses lenses_2.12 0.4.12
com.twitter chill-java 0.10.0
com.twitter chill_2.12 0.10.0
com.twitter util-app_2.12 7.1.0
com.twitter util-core_2.12 7.1.0
com.twitter util-function_2.12 7.1.0
com.twitter util-jvm_2.12 7.1.0
com.twitter util-lint_2.12 7.1.0
com.twitter util-registry_2.12 7.1.0
com.twitter util-stats_2.12 7.1.0
com.typesafe config 1.2.1
com.typesafe.scala-logging scala-logging_2.12 3.7.2
com.uber h3 3.7.0
com.univocity univocity-parsers 2.9.1
com.zaxxer HikariCP 4.0.3
commons-cli commons-cli 1.5.0
commons-codec commons-codec 1.15
commons-collections commons-collections 3.2.2
commons-dbcp commons-dbcp 1.4
commons-fileupload commons-fileupload 1.3.3
commons-httpclient commons-httpclient 3.1
commons-io commons-io 2.11.0
commons-lang commons-lang 2.6
commons-logging commons-logging 1.1.3
commons-pool commons-pool 1.5.4
dev.ludovic.netlib arpack 2.2.1
dev.ludovic.netlib blas 2.2.1
dev.ludovic.netlib lapack 2.2.1
info.ganglia.gmetric4j gmetric4j 1.0.10
io.airlift aircompressor 0.21
io.delta delta-sharing-spark_2.12 0.5.1
io.dropwizard.metrics metrics-core 4.1.1
io.dropwizard.metrics metrics-graphite 4.1.1
io.dropwizard.metrics metrics-healthchecks 4.1.1
io.dropwizard.metrics metrics-jetty9 4.1.1
io.dropwizard.metrics metrics-jmx 4.1.1
io.dropwizard.metrics metrics-json 4.1.1
io.dropwizard.metrics metrics-jvm 4.1.1
io.dropwizard.metrics metrics-servlets 4.1.1
io.netty netty-all 4.1.74.Final
io.netty netty-buffer 4.1.74.Final
io.netty netty-codec 4.1.74.Final
io.netty netty-common 4.1.74.Final
io.netty netty-handler 4.1.74.Final
io.netty netty-resolver 4.1.74.Final
io.netty netty-tcnative-classes 2.0.48.Final
io.netty netty-transport 4.1.74.Final
io.netty netty-transport-classes-epoll 4.1.74.Final
io.netty netty-transport-classes-kqueue 4.1.74.Final
io.netty netty-transport-native-epoll-linux-aarch_64 4.1.74.Final
io.netty netty-transport-native-epoll-linux-x86_64 4.1.74.Final
io.netty netty-transport-native-kqueue-osx-aarch_64 4.1.74.Final
io.netty netty-transport-native-kqueue-osx-x86_64 4.1.74.Final
io.netty netty-transport-native-unix-common 4.1.74.Final
io.prometheus simpleclient 0.7.0
io.prometheus simpleclient_common 0.7.0
io.prometheus simpleclient_dropwizard 0.7.0
io.prometheus simpleclient_pushgateway 0.7.0
io.prometheus simpleclient_servlet 0.7.0
io.prometheus.jmx collector 0.12.0
jakarta.annotation jakarta.annotation-api 1.3.5
jakarta.servlet jakarta.servlet-api 4.0.3
jakarta.validation jakarta.validation-api 2.0.2
jakarta.ws.rs jakarta.ws.rs-api 2.1.6
javax.activation activation 1.1.1
javax.annotation javax.annotation-api 1.3.2
javax.el javax.el-api 2.2.4
javax.jdo jdo-api 3.0.1
javax.transaction jta 1.1
javax.transaction transaction-api 1.1
javax.xml.bind jaxb-api 2.2.11
javolution javolution 5.5.1
jline jline 2.14.6
joda-time joda-time 2.10.13
net.java.dev.jna jna 5.8.0
net.razorvine pickle 1.2
net.sf.jpam jpam 1.1
net.sf.opencsv opencsv 2.3
net.sf.supercsv super-csv 2.2.0
net.snowflake snowflake-ingest-sdk 0.9.6
net.snowflake snowflake-jdbc 3.13.14
net.sourceforge.f2j arpack_combined_all 0.1
org.acplt.remotetea remotetea-oncrpc 1.1.2
org.antlr ST4 4.0.4
org.antlr antlr-runtime 3.5.2
org.antlr antlr4-runtime 4.8
org.antlr stringtemplate 3.2.1
org.apache.ant ant 1.9.2
org.apache.ant ant-jsch 1.9.2
org.apache.ant ant-launcher 1.9.2
org.apache.arrow arrow-format 7.0.0
org.apache.arrow arrow-memory-core 7.0.0
org.apache.arrow arrow-memory-netty 7.0.0
org.apache.arrow arrow-vector 7.0.0
org.apache.avro avro 1.11.0
org.apache.avro avro-ipc 1.11.0
org.apache.avro avro-mapred 1.11.0
org.apache.commons commons-collections4 4.4
org.apache.commons commons-compress 1.21
org.apache.commons commons-crypto 1.1.0
org.apache.commons commons-lang3 3.12.0
org.apache.commons commons-math3 3.6.1
org.apache.commons commons-text 1.9
org.apache.curator curator-client 2.13.0
org.apache.curator curator-framework 2.13.0
org.apache.curator curator-recipes 2.13.0
org.apache.derby derby 10.14.2.0
org.apache.hadoop hadoop-client-api 3.3.4-databricks
org.apache.hadoop hadoop-client-runtime 3.3.4
org.apache.hive hive-beeline 2.3.9
org.apache.hive hive-cli 2.3.9
org.apache.hive hive-jdbc 2.3.9
org.apache.hive hive-llap-client 2.3.9
org.apache.hive hive-llap-common 2.3.9
org.apache.hive hive-serde 2.3.9
org.apache.hive hive-shims 2.3.9
org.apache.hive hive-storage-api 2.7.2
org.apache.hive.shims hive-shims-0.23 2.3.9
org.apache.hive.shims hive-shims-common 2.3.9
org.apache.hive.shims hive-shims-scheduler 2.3.9
org.apache.httpcomponents httpclient 4.5.13
org.apache.httpcomponents httpcore 4.4.14
org.apache.ivy ivy 2.5.0
org.apache.logging.log4j log4j-1.2-api 2.18.0
org.apache.logging.log4j log4j-api 2.18.0
org.apache.logging.log4j log4j-core 2.18.0
org.apache.logging.log4j log4j-slf4j-impl 2.18.0
org.apache.mesos mesos-shaded-protobuf 1.4.0
org.apache.orc orc-core 1.7.6
org.apache.orc orc-mapreduce 1.7.6
org.apache.orc orc-shims 1.7.6
org.apache.parquet parquet-column 1.12.0-databricks-0007
org.apache.parquet parquet-common 1.12.0-databricks-0007
org.apache.parquet parquet-encoding 1.12.0-databricks-0007
org.apache.parquet parquet-format-structures 1.12.0-databricks-0007
org.apache.parquet parquet-hadoop 1.12.0-databricks-0007
org.apache.parquet parquet-jackson 1.12.0-databricks-0007
org.apache.thrift libfb303 0.9.3
org.apache.thrift libthrift 0.12.0
org.apache.xbean xbean-asm9-shaded 4.20
org.apache.yetus audience-annotations 0.5.0
org.apache.zookeeper zookeeper 3.6.2
org.apache.zookeeper zookeeper-jute 3.6.2
org.checkerframework checker-qual 3.5.0
org.codehaus.jackson jackson-core-asl 1.9.13
org.codehaus.jackson jackson-mapper-asl 1.9.13
org.codehaus.janino commons-compiler 3.0.16
org.codehaus.janino janino 3.0.16
org.datanucleus datanucleus-api-jdo 4.2.4
org.datanucleus datanucleus-core 4.1.17
org.datanucleus datanucleus-rdbms 4.1.19
org.datanucleus javax.jdo 3.2.0-m3
org.eclipse.jetty jetty-client 9.4.46.v20220331
org.eclipse.jetty jetty-continuation 9.4.46.v20220331
org.eclipse.jetty jetty-http 9.4.46.v20220331
org.eclipse.jetty jetty-io 9.4.46.v20220331
org.eclipse.jetty jetty-jndi 9.4.46.v20220331
org.eclipse.jetty jetty-plus 9.4.46.v20220331
org.eclipse.jetty jetty-proxy 9.4.46.v20220331
org.eclipse.jetty jetty-security 9.4.46.v20220331
org.eclipse.jetty jetty-server 9.4.46.v20220331
org.eclipse.jetty jetty-servlet 9.4.46.v20220331
org.eclipse.jetty jetty-servlets 9.4.46.v20220331
org.eclipse.jetty jetty-util 9.4.46.v20220331
org.eclipse.jetty jetty-util-ajax 9.4.46.v20220331
org.eclipse.jetty jetty-webapp 9.4.46.v20220331
org.eclipse.jetty jetty-xml 9.4.46.v20220331
org.eclipse.jetty.websocket websocket-api 9.4.46.v20220331
org.eclipse.jetty.websocket websocket-client 9.4.46.v20220331
org.eclipse.jetty.websocket websocket-common 9.4.46.v20220331
org.eclipse.jetty.websocket websocket-server 9.4.46.v20220331
org.eclipse.jetty.websocket websocket-servlet 9.4.46.v20220331
org.fusesource.leveldbjni leveldbjni-all 1.8
org.glassfish.hk2 hk2-api 2.6.1
org.glassfish.hk2 hk2-locator 2.6.1
org.glassfish.hk2 hk2-utils 2.6.1
org.glassfish.hk2 osgi-resource-locator 1.0.3
org.glassfish.hk2.external aopalliance-repackaged 2.6.1
org.glassfish.hk2.external jakarta.inject 2.6.1
org.glassfish.jersey.containers jersey-container-servlet 2.36
org.glassfish.jersey.containers jersey-container-servlet-core 2.36
org.glassfish.jersey.core jersey-client 2.36
org.glassfish.jersey.core jersey-common 2.36
org.glassfish.jersey.core jersey-server 2.36
org.glassfish.jersey.inject jersey-hk2 2.36
org.hibernate.validator hibernate-validator 6.1.0.Final
org.javassist javassist 3.25.0-GA
org.jboss.logging jboss-logging 3.3.2.Final
org.jdbi jdbi 2.63.1
org.jetbrains annotations 17.0.0
org.joda joda-convert 1.7
org.jodd jodd-core 3.5.2
org.json4s json4s-ast_2.12 3.7.0-M11
org.json4s json4s-core_2.12 3.7.0-M11
org.json4s json4s-jackson_2.12 3.7.0-M11
org.json4s json4s-scalap_2.12 3.7.0-M11
org.lz4 lz4-java 1.8.0
org.mariadb.jdbc mariadb-java-client 2.7.4
org.mlflow mlflow-spark 1.27.0
org.objenesis objenesis 2.5.1
org.postgresql postgresql 42.3.3
org.roaringbitmap RoaringBitmap 0.9.25
org.roaringbitmap shims 0.9.25
org.rocksdb rocksdbjni 6.24.2
org.rosuda.REngine REngine 2.1.0
org.scala-lang scala-compiler_2.12 2.12.14
org.scala-lang scala-library_2.12 2.12.14
org.scala-lang scala-reflect_2.12 2.12.14
org.scala-lang.modules scala-collection-compat_2.12 2.4.3
org.scala-lang.modules scala-parser-combinators_2.12 1.1.2
org.scala-lang.modules scala-xml_2.12 1.2.0
org.scala-sbt test-interface 1.0
org.scalacheck scalacheck_2.12 1.14.2
org.scalactic scalactic_2.12 3.0.8
org.scalanlp breeze-macros_2.12 1.2
org.scalanlp breeze_2.12 1.2
org.scalatest scalatest_2.12 3.0.8
org.slf4j jcl-over-slf4j 1.7.36
org.slf4j jul-to-slf4j 1.7.36
org.slf4j slf4j-api 1.7.36
org.spark-project.spark unused 1.0.0
org.threeten threeten-extra 1.5.0
org.tukaani xz 1.8
org.typelevel algebra_2.12 2.0.1
org.typelevel cats-kernel_2.12 2.1.1
org.typelevel macro-compat_2.12 1.1.1
org.typelevel spire-macros_2.12 0.17.0
org.typelevel spire-platform_2.12 0.17.0
org.typelevel spire-util_2.12 0.17.0
org.typelevel spire_2.12 0.17.0
org.wildfly.openssl wildfly-openssl 1.0.7.Final
org.xerial sqlite-jdbc 3.8.11.2
org.xerial.snappy snappy-java 1.1.8.4
org.yaml snakeyaml 1.24
oro oro 2.0.8
pl.edu.icm JLargeArrays 1.5
software.amazon.ion ion-java 1.0.2
stax stax-api 1.0.1