Databricks Runtime 11.0 (unsupported)

  • Article

The following release notes provide information about Databricks Runtime 11.0, powered by Apache Spark 3.3.0. Databricks released these images in June 2022.

New features and improvements

New version of Apache Spark

Databricks Runtime 11.0 and Databricks Runtime 11.0 Photon include Apache Spark 3.3.0. For details, see Apache Spark.

Python notebooks now use the IPython kernel

In Databricks Runtime 11.0 and above, Python notebooks use the IPython kernel to execute Python code. See IPython kernel.

Support for ipywidgets

You can now use ipywidgets to make your Databricks Python notebooks interactive. See ipywidgets.

Synapse connector now writes Parquet data in non-legacy mode

The Azure Synapse connector now writes Parquet data in non-legacy mode. It preserves the INT96 timestamp format when using PolyBase and COPY commands for both batch and streaming workloads.

HTTPS scheme now enforced when ABFS client uses SAS token

When the Azure Blob File System (ABFS) client uses a Shared Access Signature (SAS) token, the HTTPS scheme is now enforced.

SQL: DESC is now an alias for DESCRIBE

You can now use DESC as an alias for DESCRIBE when you describe external locations or storage credentials. For example:

-- Describe an external location.
DESC EXTERNAL LOCATION location_name;

-- Describe a storage credential.
DESC STORAGE CREDENTIAL credential_name;

SQL: New current_version function outputs version details

The new current_version function outputs the current Databricks Runtime version if available, the current Databricks SQL version if available, and other related version details. Use this new function to query version-related information. See current_version function.

Dropping a missing Delta table constraint now raises an error

If you now try to drop a Delta table constraint by name, and that constraint does not exist, you will get an error. To get the previous behavior, which does not throw an error if the constraint does not exist, you must now use the IF EXISTS statement. See ALTER TABLE.

SQL: New EXCEPT clause in SELECT statement excludes columns from selection

SELECT statements now support the EXCEPT clause to exclude columns from selection. For example, SELECT * EXCEPT (x) FROM table returns all of table's columns except x. Nested columns are also allowed. For example, SELECT * EXCEPT (x.a) FROM table returns all of table's columns but omits field a from struct x.

Support for dropping columns in Delta tables (Public Preview)

You can use ALTER TABLE <table-name> DROP COLUMN [IF EXISTS] <column-name> or ALTER TABLE <table-name> DROP COLUMNS [IF EXISTS] (<column-name>, *) to drop a column or a list of columns, respectively, from a Delta table as a metadata-only operation. The columns are effectively "soft-deleted," as they are still in the underlying Parquet files but are no longer visible to the Delta table.

You can use REORG TABLE <table-name> APPLY (PURGE) to trigger a file rewrite on the files that contain any soft-deleted data such as dropped columns.

You can use VACUUM to remove the deleted files from physical storage, including the old files that contain dropped columns and have been rewritten by REORG TABLE.

COPY INTO improvements

You can now create empty placeholder Delta tables so that the schema is later inferred during a COPY INTO command:

CREATE TABLE IF NOT EXISTS my_table
[COMMENT <table-description>]
[TBLPROPERTIES (<table-properties>)];

COPY INTO my_table
FROM '/path/to/files'
FILEFORMAT = <format>
FORMAT_OPTIONS ('mergeSchema' = 'true')
COPY_OPTIONS ('mergeSchema' = 'true');

The preceding SQL statement is idempotent and can be scheduled to run to ingest data exactly once into a Delta table.

Note

The empty Delta table is not usable outside of COPY INTO. You cannot use INSERT INTO and MERGE INTO to write data into schema-less Delta tables. After data is inserted into the table with COPY INTO, the table is queryable.

If the data you are ingesting cannot be read due to some corruption issue, you can skip corrupted files by setting ignoreCorruptFiles to true in the FORMAT_OPTIONS:

COPY INTO my_table
FROM '/path/to/files'
FILEFORMAT = <format>
FORMAT_OPTIONS ('ignoreCorruptFiles' = 'true')

The COPY INTO command returns the number of files that were skipped due to corruption in the num_skipped_corrupt_files column. This metric also appears in the operationMetrics column under numSkippedCorruptFiles after you run DESCRIBE HISTORY on the Delta table.

Corrupt files are not tracked by COPY INTO, so they can be reloaded in a subsequent run if the corruption is fixed. You can see which files are corrupt by running COPY INTO in VALIDATE mode.

CONVERT TO DELTA is now supported in Unity Catalog enabled environments (Public Preview)

In Unity-Catalog-enabled environments, with CONVERT TO DELTA you can now:

  • Convert Parquet files in external locations to Delta Lake.
  • Convert Parquet external tables to Delta tables.

Behavior changes

SQL: lpad and rpad functions now support byte sequences

The lpad and rpad functions have been updated to add support for byte sequences in addition to strings.

String format in format_string and printf no longer allows %0$

Specifying the format %0$ in the format_string and printf functions now throws an error by default. This change is to preserve expected behavior with previous versions of the Databricks Runtime and common third-party databases. The first argument should always reference %1$ when you use an argument index to indicate the position of the argument in the argument list.

Null values in CSV files are now written as unquoted empty strings by default

Null values in CSV files were previously written as quoted empty strings. With this release, null values in CSV files are now written as unquoted empty strings by default. To change back to the previous behavior, set the nullValue option to "" for write operations.

The table property external is now reserved

The property external is now a reserved table property by default. Exceptions are now thrown when you use the external property with the CREATE TABLE ... TBLPROPERTIES and ALTER TABLE ... SET TBLPROPERTIES clauses.

Log4j is upgraded from Log4j 1 to Log4j 2

Log4j 1 is being upgraded to Log4j 2. Legacy Log4j 1 dependencies are being removed.

If you depend on the Log4j 1 classes that were previously included inside the Databricks Runtime, those classes no longer exist. You should upgrade your dependencies to Log4j 2.

If you have custom plugins or configuration files that depend on Log4j 2, they may no longer work with the version of Log4j 2 in this release. For assistance, contact your Azure Databricks account team.

Libraries installed from Maven are now resolved in your compute plane by default

Maven libraries now resolve in your compute plane by default when you install libraries on a cluster. You cluster must have access to Maven Central. Alternatively, you can revert to the previous behavior by setting the Spark configuration property:

spark.databricks.libraries.enableMavenResolution false

Bug fixes

  • Binary compatibility for LeafNode, UnaryNode, and BinaryNode between Apache Spark and the Databricks Runtime has been fixed and the classes are now compatible with Apache Spark 3.3.0 and above. If you encounter the following or a similar message when using a third-party package with the Databricks Runtime, rebuild the package with Apache Spark 3.3.0 or above: Found interface org.apache.spark.sql.catalyst.plans.logical.UnaryNode, but class was expected.

Library upgrades

  • Upgraded Python libraries:
    • platformdirs from 2.5.1 to 2.5.2
    • protobuf from 3.20.0 to 3.20.1
  • Upgraded R libraries:
    • blob from 1.2.2 to 1.2.3
    • broom from 0.7.12 to 0.8.0
    • caret from 6.0-91 to 6.0-92
    • cli from 3.2.0 to 3.3.0
    • dplyr from 1.0.8 to 1.0.9
    • future from 1.24.0 to 1.25.0
    • future.apply from 1.8.1 to 1.9.0
    • gert from 1.5.0 to 1.6.0
    • ggplot2 from 3.3.5 to 3.3.6
    • glmnet from 4.1-3 to 4.1-4
    • haven from 2.4.3 to 2.5.0
    • httr from 1.4.2 to 1.4.3
    • knitr from 1.38 to 1.39
    • magrittr from 2.0.2 to 2.0.3
    • parallelly from 1.30.0 to 1.31.1
    • ps from 1.6.0 to 1.7.0
    • RColorBrewer from 1.1-2 to 1.1-3
    • RcppEigen from 0.3.3.9.1 to 0.3.3.9.2
    • readxl from 1.3.1 to 1.4.0
    • rmarkdown from 2.13 to 2.14
    • rprojroot from 2.0.2 to 2.0.3
    • RSQLite from 2.2.11 to 2.2.13
    • scales from 1.1.1 to 1.2.0
    • testthat from 3.1.2 to 3.1.4
    • tibble from 3.1.6 to 3.1.7
    • tinytex from 0.37 to 0.38
    • tzdb from 0.2.0 to 0.3.0
    • uuid from 1.0-4 to 1.1-0
    • vctrs from 0.3.8 to 0.4.1
  • Upgraded Java libraries:
    • com.fasterxml.jackson.core.jackson-annotations from 2.13.0 to 2.13.3
    • com.fasterxml.jackson.core.jackson-core from 2.13.0 to 2.13.3
    • com.fasterxml.jackson.core.jackson-databind from 2.13.0 to 2.13.3
    • com.fasterxml.jackson.dataformat.jackson-dataformat-cbor from 2.13.0 to 2.13.3
    • com.fasterxml.jackson.datatype.jackson-datatype-joda from 2.13.0 to 2.13.3
    • com.fasterxml.jackson.module.jackson-module-paranamer from 2.13.0 to 2.13.3
    • com.fasterxml.jackson.module.jackson-module-scala_2.12 from 2.13.0 to 2.13.3
    • com.google.crypto.tink.tink from 1.6.0 to 1.6.1
    • com.ning.compress-lzf from 1.0.3 to 1.1
    • dev.ludovic.netlib.arpack from 2.2.0 to 2.2.1
    • dev.ludovic.netlib.blas from 2.2.0 to 2.2.1
    • dev.ludovic.netlib.lapack from 2.2.0 to 2.2.1
    • io.netty.netty-all from 4.1.73.Final to 4.1.74.Final
    • io.netty.netty-buffer from 4.1.73.Final to 4.1.74.Final
    • io.netty.netty-codec from 4.1.73.Final to 4.1.74.Final
    • io.netty.netty-common from 4.1.73.Final to 4.1.74.Final
    • io.netty.netty-handler from 4.1.73.Final to 4.1.74.Final
    • io.netty.netty-resolver from 4.1.73.Final to 4.1.74.Final
    • io.netty.netty-tcnative-classes from 2.0.46.Final to 2.0.48.Final
    • io.netty.netty-transport from 4.1.73.Final to 4.1.74.Final
    • io.netty.netty-transport-classes-epoll from 4.1.73.Final to 4.1.74.Final
    • io.netty.netty-transport-classes-kqueue from 4.1.73.Final to 4.1.74.Final
    • io.netty.netty-transport-native-epoll-linux-aarch_64 from 4.1.73.Final to 4.1.74.Final
    • io.netty.netty-transport-native-epoll-linux-x86_64 from 4.1.73.Final to 4.1.74.Final
    • io.netty.netty-transport-native-kqueue-osx-aarch_64 from 4.1.73.Final to 4.1.74.Final
    • io.netty.netty-transport-native-kqueue-osx-x86_64 from 4.1.73.Final to 4.1.74.Final
    • io.netty.netty-transport-native-unix-common from 4.1.73.Final to 4.1.74.Final
    • joda-time.joda-time from 2.10.12 to 2.10.13
    • org.apache.commons.commons-math3 from 3.4.1 to 3.6.1
    • org.apache.httpcomponents.httpcore from 4.4.12 to 4.4.14
    • org.apache.orc.orc-core from 1.7.3 to 1.7.4
    • org.apache.orc.orc-mapreduce from 1.7.3 to 1.7.4
    • org.apache.orc.orc-shims from 1.7.3 to 1.7.4
    • org.eclipse.jetty.jetty-client from 9.4.43.v20210629 to 9.4.46.v20220331
    • org.eclipse.jetty.jetty-continuation from 9.4.43.v20210629 to 9.4.46.v20220331
    • org.eclipse.jetty.jetty-http from 9.4.43.v20210629 to 9.4.46.v20220331
    • org.eclipse.jetty.jetty-io from 9.4.43.v20210629 to 9.4.46.v20220331
    • org.eclipse.jetty.jetty-jndi from 9.4.43.v20210629 to 9.4.46.v20220331
    • org.eclipse.jetty.jetty-plus from 9.4.43.v20210629 to 9.4.46.v20220331
    • org.eclipse.jetty.jetty-proxy from 9.4.43.v20210629 to 9.4.46.v20220331
    • org.eclipse.jetty.jetty-security from 9.4.43.v20210629 to 9.4.46.v20220331
    • org.eclipse.jetty.jetty-server from 9.4.43.v20210629 to 9.4.46.v20220331
    • org.eclipse.jetty.jetty-servlet from 9.4.43.v20210629 to 9.4.46.v20220331
    • org.eclipse.jetty.jetty-servlets from 9.4.43.v20210629 to 9.4.46.v20220331
    • org.eclipse.jetty.jetty-util from 9.4.43.v20210629 to 9.4.46.v20220331
    • org.eclipse.jetty.jetty-util-ajax from 9.4.43.v20210629 to 9.4.46.v20220331
    • org.eclipse.jetty.jetty-webapp from 9.4.43.v20210629 to 9.4.46.v20220331
    • org.eclipse.jetty.jetty-xml from 9.4.43.v20210629 to 9.4.46.v20220331
    • org.eclipse.jetty.websocket.websocket-api from 9.4.43.v20210629 to 9.4.46.v20220331
    • org.eclipse.jetty.websocket.websocket-client from 9.4.43.v20210629 to 9.4.46.v20220331
    • org.eclipse.jetty.websocket.websocket-common from 9.4.43.v20210629 to 9.4.46.v20220331
    • org.eclipse.jetty.websocket.websocket-server from 9.4.43.v20210629 to 9.4.46.v20220331
    • org.eclipse.jetty.websocket.websocket-servlet from 9.4.43.v20210629 to 9.4.46.v20220331
    • org.mariadb.jdbc.mariadb-java-client from 2.2.5 to 2.7.4
    • org.postgresql.postgresql from 42.2.19 to 42.3.3
    • org.roaringbitmap.RoaringBitmap from 0.9.23 to 0.9.25
    • org.roaringbitmap.shims from 0.9.23 to 0.9.25
    • org.rocksdb.rocksdbjni from 6.20.3 to 6.24.2
    • org.slf4j.jcl-over-slf4j from 1.7.32 to 1.7.36
    • org.slf4j.jul-to-slf4j from 1.7.32 to 1.7.36
    • org.slf4j.slf4j-api from 1.7.30 to 1.7.36

Apache Spark

Databricks Runtime 11.0 includes Apache Spark 3.3.0.

In this section:

Spark SQL and Core

ANSI mode

  • New explicit cast syntax rules in ANSI mode (SPARK-33354)
  • Elt() should return null if index is null under ANSI mode (SPARK-38304)
  • Optionally return null result if element not exists in array/map (SPARK-37750)
  • Allow casting between numeric type and timestamp type (SPARK-37714)
  • Disable ANSI reserved keywords by default (SPARK-37724)
  • Use store assignment rules for resolving function invocation (SPARK-37438)
  • Add a config to allow casting between Datetime and Numeric (SPARK-37179)
  • Add a config to optionally enforce ANSI reserved keywords (SPARK-37133)
  • Disallow binary operations between Interval and String literal (SPARK-36508)

Feature enhancements

Performance enhancements

  • Whole-stage code generation
    • Add code-gen for sort aggregate without grouping keys (SPARK-37564)
    • Add code-gen for full outer sort merge join (SPARK-35352)
    • Add code-gen for full outer shuffled hash join (SPARK-32567)
    • Add code-gen for existence sort merge join (SPARK-37316)
  • Push down (filters)
    • Push down filters through RebalancePartitions (SPARK-37828)
    • Push down boolean column filter (SPARK-36644)
    • Push down limit 1 for right side of left semi/anti join if join condition is empty (SPARK-37917)
    • Translate more standard aggregate functions for pushdown (SPARK-37527)
    • Support propagate empty relation through aggregate/union (SPARK-35442)
    • Row-level Runtime Filtering (SPARK-32268)
    • Support Left Semi join in row level runtime filters (SPARK-38565)
    • Support predicate pushdown and column pruning for de-duped CTEs (SPARK-37670)
  • Vectorization
    • Implement a ConstantColumnVector and improve performance of the hidden file metadata (SPARK-37896)
    • Enable vectorized read for VectorizedPlainValuesReader.readBooleans (SPARK-35867)
  • Combine/remove/replace nodes
    • Combine unions if there is a project between them (SPARK-37915)
    • Combine to one cast if we can safely up-cast two casts (SPARK-37922)
    • Remove the Sort if it is the child of RepartitionByExpression (SPARK-36703)
    • Removes outer join if it only has DISTINCT on streamed side with alias (SPARK-37292)
    • Replace hash with sort aggregate if child is already sorted (SPARK-37455)
    • Only collapse projects if we don't duplicate expensive expressions (SPARK-36718)
    • Remove redundant aliases after RewritePredicateSubquery (SPARK-36280)
    • Merge non-correlated scalar subqueries (SPARK-34079)
  • Partitioning
    • Do not add dynamic partition pruning if there exists static partition pruning (SPARK-38148)
    • Improve RebalancePartitions in rules of Optimizer (SPARK-37904)
    • Add small partition factor for rebalance partitions (SPARK-37357)
  • Join
    • Fine tune logic to demote Broadcast hash join in DynamicJoinSelection (SPARK-37753)
    • Ignore duplicated join keys when building relation for SEMI/ANTI shuffled hash join (SPARK-36794)
    • Support optimize skewed join even if introduce extra shuffle (SPARK-33832)
  • AQE
    • Support eliminate limits in AQE Optimizer (SPARK-36424)
    • Optimize one row plan in normal and AQE Optimizer (SPARK-38162)
  • Aggregate.groupOnly support foldable expressions (SPARK-38489)
  • ByteArrayMethods arrayEquals should fast skip the check of aligning with unaligned platform (SPARK-37796)
  • Add tree pattern pruning to CTESubstitution rule (SPARK-37379)
  • Add more Not operator simplifications (SPARK-36665)
  • Support BooleanType in UnwrapCastInBinaryComparison (SPARK-36607)
  • Coalesce drop all expressions after the first non nullable expression (SPARK-36359)
  • Add a logical plan visitor to propagate the distinct attributes (SPARK-36194)

Built-in connector enhancements

  • General
    • Lenient serialization of datetime from datasource (SPARK-38437)
    • Treat table location as absolute when the first letter of its path is slash in create/alter table (SPARK-38236)
    • Remove leading zeros from empty static number type partition (SPARK-35561)
    • Support ignoreCorruptFiles and ignoreMissingFiles in Data Source options (SPARK-38767)
    • Add command SHOW CATALOGS (SPARK-35973)
  • Parquet
    • Enable matching schema column names by field ids (SPARK-38094)
    • Remove check field name when reading/writing data in parquet (SPARK-27442)
    • Support vectorized read boolean values use RLE encoding with Parquet DataPage V2 (SPARK-37864)
    • Support Parquet v2 data page encoding (DELTA_BINARY_PACKED) for the vectorized path (SPARK-36879)
    • Rebase timestamps in the session time zone saved in Parquet/Avro metadata (SPARK-37705)
    • Push down group by partition column for aggregate (SPARK-36646)
    • Aggregate (Min/Max/Count) push down for Parquet (SPARK-36645)
    • Parquet: enable matching schema columns by field id (SPARK-38094)
    • Reduce default page size by LONG_ARRAY_OFFSET if G1GC and ON_HEAP are used (SPARK-37593)
    • Implement vectorized DELTA_BYTE_ARRAY and DELTA_LENGTH_BYTE_ARRAY encodings for Parquet V2 support (SPARK-37974)
    • Support complex types for Parquet vectorized reader (SPARK-34863)
  • ORC
    • Remove check field name when reading/writing existing data in Orc (SPARK-37965)
    • Aggregate push down for ORC (SPARK-34960)
    • Support reading and writing ANSI intervals from/to ORC datasources (SPARK-36931)
    • Support number-only column names in ORC data sources (SPARK-36663)
  • JSON
    • Respect allowNonNumericNumbers when parsing quoted NaN and Infinity values in JSON reader (SPARK-38060)
    • Use CAST for datetime in CSV/JSON by default (SPARK-36536)
    • Align error message for unsupported key types in MapType in Json reader (SPARK-35320)
  • CSV
    • Fix referring to the corrupt record column from CSV (SPARK-38534)
    • null values should be saved as nothing instead of quoted empty Strings "" by default (SPARK-37575)
  • JDBC
    • Add the IMMEDIATE statement to the DB2 dialect truncate implementation (SPARK-30062)
  • Hive
    • Support writing Hive bucketed table (Hive file formats with Hive hash) (SPARK-32712)
    • Use expressions to filter Hive partitions at client side (SPARK-35437)
    • Support Dynamic Partition pruning for HiveTableScanExec (SPARK-36876)
    • InsertIntoHiveDir should use data source if it's convertible (SPARK-38215)
    • Support writing Hive bucketed table (Parquet/ORC format with Hive hash) (SPARK-32709)

Node decommission

  • FallbackStorage shouldn't attempt to resolve arbitrary "remote" hostname (SPARK-38062)
  • ExecutorMonitor.onExecutorRemoved should handle ExecutorDecommission as finished (SPARK-38023)

Other notable changes

  • Add fine grained locking to BlockInfoManager (SPARK-37356)
  • Support mapping Spark gpu/fpga resource types to custom YARN resource type (SPARK-37208)
  • Report accurate shuffle block size if its skewed (SPARK-36967)
  • Supporting Netty Logging at the network layer (SPARK-36719)

Structured Streaming

Major features

  • Introduce Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches (SPARK-36533)

Other notable changes

  • Use StatefulOpClusteredDistribution for stateful operators with respecting backward compatibility (SPARK-38204)
  • Fix flatMapGroupsWithState timeout in batch with data for key (SPARK-38320)
  • Fix correctness issue on stream-stream outer join with RocksDB state store provider (SPARK-38684)
  • Support Trigger.AvailableNow on Kafka data source (SPARK-36649)
  • Optimize write path on RocksDB state store provider (SPARK-37224)
  • Introduce a new data source for providing a consistent set of rows per microbatch (SPARK-37062)
  • Use HashClusteredDistribution for stateful operators with respecting backward compatibility (SPARK-38204)

PySpark

Pandas API on Spark

Major improvements
  • 'distributed-sequence' index optimization with being default (SPARK-36559, SPARK-36338)
    • Support to specify index type and name in pandas API on Spark (SPARK-36709)
    • Show default index type in SQL plans for pandas API on Spark (SPARK-38654)
Major features
  • Implement SparkSQL native ps.merge_asof (SPARK-36813)
  • Support TimedeltaIndex in pandas API on Spark (SPARK-37525)
  • Support Python's timedelta (SPARK-37275, SPARK-37510)
  • Implement functions in CategoricalAccessor/CategoricalIndex (SPARK-36185)
  • Uses Python's standard string formatter for SQL API in pandas API on Spark (SPARK-37436)
  • Support basic operations of timedelta Series/Index (SPARK-37510)
  • Support ps.MultiIndex.dtypes (SPARK-36930)
  • Implement Index.map (SPARK-36469)
  • Implement Series.xor and Series.rxor (SPARK-36653)
  • Implement unary operator invert of integral ps.Series/Index (SPARK-36003)
  • Implement DataFrame.cov (SPARK-36396)
  • Support str and timestamp for (Series|DataFrame).describe() (SPARK-37657)
  • Support lambda column parameter of DataFrame.rename(SPARK-38763)

Other notable changes

Breaking changes
  • Drop references to Python 3.6 support in docs and python/docs (SPARK-36977)
  • Remove namedtuple hack by replacing built-in pickle to cloudpickle (SPARK-32079)
  • Bump minimum pandas version to 1.0.5 (SPARK-37465)
  • Major improvements
    • Provide a profiler for Python/Pandas UDFs (SPARK-37443)
    • Uses Python's standard string formatter for SQL API in PySpark (SPARK-37516)
    • Expose SQL state and error class in PySpark exceptions (SPARK-36953)
    • Try to capture faulthanlder when a Python worker crashes (SPARK-36062)
Major features
  • Implement DataFrame.mapInArrow in Python (SPARK-37228)
  • Uses Python's standard string formatter for SQL API in PySpark (SPARK-37516)
  • Add df.withMetadata pyspark API (SPARK-36642)
  • Support Python's timedelta (SPARK-37275)
  • Expose tableExists in pyspark.sql.catalog (SPARK-36176)
  • Expose databaseExists in pyspark.sql.catalog (SPARK-36207)
  • Exposing functionExists in pyspark sql catalog (SPARK-36258)
  • Add Dataframe.observation to PySpark (SPARK-36263)
  • Add max_by/min_by API to PySpark (SPARK-36972)
  • Support to infer nested dict as a struct when creating a DataFrame (SPARK-35929)
  • Add bit/octet_length APIs to Scala, Python and R (SPARK-36751)
  • Support ILIKE API on Python (SPARK-36882)
  • Add isEmpty method for the Python DataFrame API (SPARK-37207)
  • Add multiple columns adding support (SPARK-35173)
  • Add SparkContext.addArchive in PySpark (SPARK-38278)
  • Make sql type reprs eval-able (SPARK-18621)
  • Inline type hints for fpm.py in python/pyspark/mllib (SPARK-37396)
  • Implement dropna parameter of SeriesGroupBy.value_counts (SPARK-38837)

MLLIB

Major features

  • Add distanceMeasure param to trainKMeansModel (SPARK-37118)
  • Expose LogisticRegression.setInitialModel, like KMeans et al do (SPARK-36481)
  • Support CrossValidatorModel get standard deviation of metrics for each paramMap (SPARK-36425)

Major improvements

  • Optimize some treeAggregates in MLlib by delaying allocations (SPARK-35848)
  • Rewrite _shared_params_code_gen.py to inline type hints for ml/param/shared.py (SPARK-37419)

Other notable changes

SparkR

UI

  • Speculation metrics summary at stage level (SPARK-36038)
  • Unified shuffle read block time to shuffle read fetch wait time in StagePage (SPARK-37469)
  • Add modified configs for SQL execution in UI (SPARK-34735)
  • Make ThriftServer recognize spark.sql.redaction.string.regex (SPARK-36400)
  • Attach and start handler after application started in UI (SPARK-36237)
  • Add commit duration to SQL tab's graph node (SPARK-34399)
  • Support RocksDB backend in Spark History Server (SPARK-37680)
  • Show options for Pandas API on Spark in UI (SPARK-38656)
  • Rename 'SQL' to 'SQL / DataFrame' in SQL UI page (SPARK-38657)

Build

Maintenance updates

See Databricks Runtime 11.0 (unsupported).

System environment

  • Operating System: Ubuntu 20.04.4 LTS
  • Java: Zulu 8.56.0.21-CA-linux64
  • Scala: 2.12.14
  • Python: 3.9.5
  • R: 4.1.3
  • Delta Lake: 1.2.1

Installed Python libraries

Library Version Library Version Library Version
Antergos Linux 2015.10 (ISO-Rolling) argon2-cffi 20.1.0 async-generator 1.10
attrs 21.2.0 backcall 0.2.0 backports.entry-points-selectable 1.1.1
bleach 4.0.0 boto3 1.21.18 botocore 1.24.18
certifi 2021.10.8 cffi 1.14.6 chardet 4.0.0
charset-normalizer 2.0.4 cycler 0.10.0 Cython 0.29.24
dbus-python 1.2.16 debugpy 1.4.1 decorator 5.1.0
defusedxml 0.7.1 distlib 0.3.4 distro-info 0.23ubuntu1
entrypoints 0.3 facets-overview 1.0.0 filelock 3.6.0
idna 3.2 ipykernel 6.12.1 ipython 7.32.0
ipython-genutils 0.2.0 ipywidgets 7.7.0 jedi 0.18.0
Jinja2 2.11.3 jmespath 0.10.0 joblib 1.0.1
jsonschema 3.2.0 jupyter-client 6.1.12 jupyter-core 4.8.1
jupyterlab-pygments 0.1.2 jupyterlab-widgets 1.0.0 kiwisolver 1.3.1
MarkupSafe 2.0.1 matplotlib 3.4.3 matplotlib-inline 0.1.2
mistune 0.8.4 nbclient 0.5.3 nbconvert 6.1.0
nbformat 5.1.3 nest-asyncio 1.5.1 notebook 6.4.5
numpy 1.20.3 packaging 21.0 pandas 1.3.4
pandocfilters 1.4.3 parso 0.8.2 patsy 0.5.2
pexpect 4.8.0 pickleshare 0.7.5 Pillow 8.4.0
pip 21.2.4 platformdirs 2.5.2 plotly 5.6.0
prometheus-client 0.11.0 prompt-toolkit 3.0.20 protobuf 3.20.1
psutil 5.8.0 psycopg2 2.9.3 ptyprocess 0.7.0
pyarrow 7.0.0 pycparser 2.20 Pygments 2.10.0
PyGObject 3.36.0 pyodbc 4.0.31 pyparsing 3.0.4
pyrsistent 0.18.0 python-apt 2.0.0+ubuntu0.20.4.7 python-dateutil 2.8.2
pytz 2021.3 pyzmq 22.2.1 requests 2.26.0
requests-unixsocket 0.2.0 s3transfer 0.5.2 scikit-learn 0.24.2
scipy 1.7.1 seaborn 0.11.2 Send2Trash 1.8.0
setuptools 58.0.4 six 1.16.0 ssh-import-id 5.10
statsmodels 0.12.2 tenacity 8.0.1 terminado 0.9.4
testpath 0.5.0 threadpoolctl 2.2.0 tornado 6.1
traitlets 5.1.0 unattended-upgrades 0.1 urllib3 1.26.7
virtualenv 20.8.0 wcwidth 0.2.5 webencodings 0.5.1
wheel 0.37.0 widgetsnbextension 3.6.0

Installed R libraries

R libraries are installed from the Microsoft CRAN snapshot on 2022-05-06.

Library Version Library Version Library Version
askpass 1.1 assertthat 0.2.1 backports 1.4.1
base 4.1.3 base64enc 0.1-3 bit 4.0.4
bit64 4.0.5 blob 1.2.3 boot 1.3-28
brew 1.0-7 brio 1.1.3 broom 0.8.0
bslib 0.3.1 cachem 1.0.6 callr 3.7.0
caret 6.0-92 cellranger 1.1.0 chron 2.3-56
class 7.3-20 cli 3.3.0 clipr 0.8.0
cluster 2.1.3 codetools 0.2-18 colorspace 2.0-3
commonmark 1.8.0 compiler 4.1.3 config 0.3.1
cpp11 0.4.2 crayon 1.5.1 credentials 1.3.2
curl 4.3.2 data.table 1.14.2 datasets 4.1.3
DBI 1.1.2 dbplyr 2.1.1 desc 1.4.1
devtools 2.4.3 diffobj 0.3.5 digest 0.6.29
dplyr 1.0.9 dtplyr 1.2.1 e1071 1.7-9
ellipsis 0.3.2 evaluate 0.15 fansi 1.0.3
farver 2.1.0 fastmap 1.1.0 fontawesome 0.2.2
forcats 0.5.1 foreach 1.5.2 foreign 0.8-82
forge 0.2.0 fs 1.5.2 future 1.25.0
future.apply 1.9.0 gargle 1.2.0 generics 0.1.2
gert 1.6.0 ggplot2 3.3.6 gh 1.3.0
gitcreds 0.1.1 glmnet 4.1-4 globals 0.14.0
glue 1.6.2 googledrive 2.0.0 googlesheets4 1.0.0
gower 1.0.0 graphics 4.1.3 grDevices 4.1.3
grid 4.1.3 gridExtra 2.3 gsubfn 0.7
gtable 0.3.0 hardhat 0.2.0 haven 2.5.0
highr 0.9 hms 1.1.1 htmltools 0.5.2
htmlwidgets 1.5.4 httpuv 1.6.5 httr 1.4.3
ids 1.0.1 ini 0.3.1 ipred 0.9-12
isoband 0.2.5 iterators 1.0.14 jquerylib 0.1.4
jsonlite 1.8.0 KernSmooth 2.23-20 knitr 1.39
labeling 0.4.2 later 1.3.0 lattice 0.20-45
lava 1.6.10 lifecycle 1.0.1 listenv 0.8.0
lubridate 1.8.0 magrittr 2.0.3 markdown 1.1
MASS 7.3-56 Matrix 1.4-1 memoise 2.0.1
methods 4.1.3 mgcv 1.8-40 mime 0.12
ModelMetrics 1.2.2.2 modelr 0.1.8 munsell 0.5.0
nlme 3.1-157 nnet 7.3-17 numDeriv 2016.8-1.1
openssl 2.0.0 parallel 4.1.3 parallelly 1.31.1
pillar 1.7.0 pkgbuild 1.3.1 pkgconfig 2.0.3
pkgload 1.2.4 plogr 0.2.0 plyr 1.8.7
praise 1.0.0 prettyunits 1.1.1 pROC 1.18.0
processx 3.5.3 prodlim 2019.11.13 progress 1.2.2
progressr 0.10.0 promises 1.2.0.1 proto 1.0.0
proxy 0.4-26 ps 1.7.0 purrr 0.3.4
r2d3 0.2.6 R6 2.5.1 randomForest 4.7-1
rappdirs 0.3.3 rcmdcheck 1.4.0 RColorBrewer 1.1-3
Rcpp 1.0.8.3 RcppEigen 0.3.3.9.2 readr 2.1.2
readxl 1.4.0 recipes 0.2.0 rematch 1.0.1
rematch2 2.1.2 remotes 2.4.2 reprex 2.0.1
reshape2 1.4.4 rlang 1.0.2 rmarkdown 2.14
RODBC 1.3-19 roxygen2 7.1.2 rpart 4.1.16
rprojroot 2.0.3 Rserve 1.8-10 RSQLite 2.2.13
rstudioapi 0.13 rversions 2.1.1 rvest 1.0.2
sass 0.4.1 scales 1.2.0 selectr 0.4-2
sessioninfo 1.2.2 shape 1.4.6 shiny 1.7.1
sourcetools 0.1.7 sparklyr 1.7.5 SparkR 3.3.0
spatial 7.3-11 splines 4.1.3 sqldf 0.4-11
SQUAREM 2021.1 stats 4.1.3 stats4 4.1.3
stringi 1.7.6 stringr 1.4.0 survival 3.2-13
sys 3.4 tcltk 4.1.3 testthat 3.1.4
tibble 3.1.7 tidyr 1.2.0 tidyselect 1.1.2
tidyverse 1.3.1 timeDate 3043.102 tinytex 0.38
tools 4.1.3 tzdb 0.3.0 usethis 2.1.5
utf8 1.2.2 utils 4.1.3 uuid 1.1-0
vctrs 0.4.1 viridisLite 0.4.0 vroom 1.5.7
waldo 0.4.0 whisker 0.4 withr 2.5.0
xfun 0.30 xml2 1.3.3 xopen 1.0.0
xtable 1.8-4 yaml 2.3.5 zip 2.2.0

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID Artifact ID Version
antlr antlr 2.7.7
com.amazonaws amazon-kinesis-client 1.12.0
com.amazonaws aws-java-sdk-autoscaling 1.12.189
com.amazonaws aws-java-sdk-cloudformation 1.12.189
com.amazonaws aws-java-sdk-cloudfront 1.12.189
com.amazonaws aws-java-sdk-cloudhsm 1.12.189
com.amazonaws aws-java-sdk-cloudsearch 1.12.189
com.amazonaws aws-java-sdk-cloudtrail 1.12.189
com.amazonaws aws-java-sdk-cloudwatch 1.12.189
com.amazonaws aws-java-sdk-cloudwatchmetrics 1.12.189
com.amazonaws aws-java-sdk-codedeploy 1.12.189
com.amazonaws aws-java-sdk-cognitoidentity 1.12.189
com.amazonaws aws-java-sdk-cognitosync 1.12.189
com.amazonaws aws-java-sdk-config 1.12.189
com.amazonaws aws-java-sdk-core 1.12.189
com.amazonaws aws-java-sdk-datapipeline 1.12.189
com.amazonaws aws-java-sdk-directconnect 1.12.189
com.amazonaws aws-java-sdk-directory 1.12.189
com.amazonaws aws-java-sdk-dynamodb 1.12.189
com.amazonaws aws-java-sdk-ec2 1.12.189
com.amazonaws aws-java-sdk-ecs 1.12.189
com.amazonaws aws-java-sdk-efs 1.12.189
com.amazonaws aws-java-sdk-elasticache 1.12.189
com.amazonaws aws-java-sdk-elasticbeanstalk 1.12.189
com.amazonaws aws-java-sdk-elasticloadbalancing 1.12.189
com.amazonaws aws-java-sdk-elastictranscoder 1.12.189
com.amazonaws aws-java-sdk-emr 1.12.189
com.amazonaws aws-java-sdk-glacier 1.12.189
com.amazonaws aws-java-sdk-glue 1.12.189
com.amazonaws aws-java-sdk-iam 1.12.189
com.amazonaws aws-java-sdk-importexport 1.12.189
com.amazonaws aws-java-sdk-kinesis 1.12.189
com.amazonaws aws-java-sdk-kms 1.12.189
com.amazonaws aws-java-sdk-lambda 1.12.189
com.amazonaws aws-java-sdk-logs 1.12.189
com.amazonaws aws-java-sdk-machinelearning 1.12.189
com.amazonaws aws-java-sdk-opsworks 1.12.189
com.amazonaws aws-java-sdk-rds 1.12.189
com.amazonaws aws-java-sdk-redshift 1.12.189
com.amazonaws aws-java-sdk-route53 1.12.189
com.amazonaws aws-java-sdk-s3 1.12.189
com.amazonaws aws-java-sdk-ses 1.12.189
com.amazonaws aws-java-sdk-simpledb 1.12.189
com.amazonaws aws-java-sdk-simpleworkflow 1.12.189
com.amazonaws aws-java-sdk-sns 1.12.189
com.amazonaws aws-java-sdk-sqs 1.12.189
com.amazonaws aws-java-sdk-ssm 1.12.189
com.amazonaws aws-java-sdk-storagegateway 1.12.189
com.amazonaws aws-java-sdk-sts 1.12.189
com.amazonaws aws-java-sdk-support 1.12.189
com.amazonaws aws-java-sdk-swf-libraries 1.11.22
com.amazonaws aws-java-sdk-workspaces 1.12.189
com.amazonaws jmespath-java 1.12.189
com.chuusai shapeless_2.12 2.3.3
com.clearspring.analytics stream 2.9.6
com.databricks Rserve 1.8-3
com.databricks jets3t 0.7.1-0
com.databricks.scalapb compilerplugin_2.12 0.4.15-10
com.databricks.scalapb scalapb-runtime_2.12 0.4.15-10
com.esotericsoftware kryo-shaded 4.0.2
com.esotericsoftware minlog 1.3.0
com.fasterxml classmate 1.3.4
com.fasterxml.jackson.core jackson-annotations 2.13.3
com.fasterxml.jackson.core jackson-core 2.13.3
com.fasterxml.jackson.core jackson-databind 2.13.3
com.fasterxml.jackson.dataformat jackson-dataformat-cbor 2.13.3
com.fasterxml.jackson.datatype jackson-datatype-joda 2.13.3
com.fasterxml.jackson.module jackson-module-paranamer 2.13.3
com.fasterxml.jackson.module jackson-module-scala_2.12 2.13.3
com.github.ben-manes.caffeine caffeine 2.3.4
com.github.fommil jniloader 1.1
com.github.fommil.netlib core 1.1.2
com.github.fommil.netlib native_ref-java 1.1
com.github.fommil.netlib native_ref-java-natives 1.1
com.github.fommil.netlib native_system-java 1.1
com.github.fommil.netlib native_system-java-natives 1.1
com.github.fommil.netlib netlib-native_ref-linux-x86_64-natives 1.1
com.github.fommil.netlib netlib-native_system-linux-x86_64-natives 1.1
com.github.luben zstd-jni 1.5.2-1
com.github.wendykierp JTransforms 3.1
com.google.code.findbugs jsr305 3.0.0
com.google.code.gson gson 2.8.6
com.google.crypto.tink tink 1.6.1
com.google.flatbuffers flatbuffers-java 1.12.0
com.google.guava guava 15.0
com.google.protobuf protobuf-java 2.6.1
com.h2database h2 2.0.204
com.helger profiler 1.1.1
com.jcraft jsch 0.1.50
com.jolbox bonecp 0.8.0.RELEASE
com.lihaoyi sourcecode_2.12 0.1.9
com.microsoft.azure azure-data-lake-store-sdk 2.3.9
com.ning compress-lzf 1.1
com.sun.mail javax.mail 1.5.2
com.tdunning json 1.8
com.thoughtworks.paranamer paranamer 2.8
com.trueaccord.lenses lenses_2.12 0.4.12
com.twitter chill-java 0.10.0
com.twitter chill_2.12 0.10.0
com.twitter util-app_2.12 7.1.0
com.twitter util-core_2.12 7.1.0
com.twitter util-function_2.12 7.1.0
com.twitter util-jvm_2.12 7.1.0
com.twitter util-lint_2.12 7.1.0
com.twitter util-registry_2.12 7.1.0
com.twitter util-stats_2.12 7.1.0
com.typesafe config 1.2.1
com.typesafe.scala-logging scala-logging_2.12 3.7.2
com.univocity univocity-parsers 2.9.1
com.zaxxer HikariCP 4.0.3
commons-cli commons-cli 1.5.0
commons-codec commons-codec 1.15
commons-collections commons-collections 3.2.2
commons-dbcp commons-dbcp 1.4
commons-fileupload commons-fileupload 1.3.3
commons-httpclient commons-httpclient 3.1
commons-io commons-io 2.11.0
commons-lang commons-lang 2.6
commons-logging commons-logging 1.1.3
commons-pool commons-pool 1.5.4
dev.ludovic.netlib arpack 2.2.1
dev.ludovic.netlib blas 2.2.1
dev.ludovic.netlib lapack 2.2.1
hive-2.3__hadoop-3.2 jets3t-0.7 liball_deps_2.12
info.ganglia.gmetric4j gmetric4j 1.0.10
io.airlift aircompressor 0.21
io.delta delta-sharing-spark_2.12 0.4.0
io.dropwizard.metrics metrics-core 4.1.1
io.dropwizard.metrics metrics-graphite 4.1.1
io.dropwizard.metrics metrics-healthchecks 4.1.1
io.dropwizard.metrics metrics-jetty9 4.1.1
io.dropwizard.metrics metrics-jmx 4.1.1
io.dropwizard.metrics metrics-json 4.1.1
io.dropwizard.metrics metrics-jvm 4.1.1
io.dropwizard.metrics metrics-servlets 4.1.1
io.netty netty-all 4.1.74.Final
io.netty netty-buffer 4.1.74.Final
io.netty netty-codec 4.1.74.Final
io.netty netty-common 4.1.74.Final
io.netty netty-handler 4.1.74.Final
io.netty netty-resolver 4.1.74.Final
io.netty netty-tcnative-classes 2.0.48.Final
io.netty netty-transport 4.1.74.Final
io.netty netty-transport-classes-epoll 4.1.74.Final
io.netty netty-transport-classes-kqueue 4.1.74.Final
io.netty netty-transport-native-epoll-linux-aarch_64 4.1.74.Final
io.netty netty-transport-native-epoll-linux-x86_64 4.1.74.Final
io.netty netty-transport-native-kqueue-osx-aarch_64 4.1.74.Final
io.netty netty-transport-native-kqueue-osx-x86_64 4.1.74.Final
io.netty netty-transport-native-unix-common 4.1.74.Final
io.prometheus simpleclient 0.7.0
io.prometheus simpleclient_common 0.7.0
io.prometheus simpleclient_dropwizard 0.7.0
io.prometheus simpleclient_pushgateway 0.7.0
io.prometheus simpleclient_servlet 0.7.0
io.prometheus.jmx collector 0.12.0
jakarta.annotation jakarta.annotation-api 1.3.5
jakarta.servlet jakarta.servlet-api 4.0.3
jakarta.validation jakarta.validation-api 2.0.2
jakarta.ws.rs jakarta.ws.rs-api 2.1.6
javax.activation activation 1.1.1
javax.annotation javax.annotation-api 1.3.2
javax.el javax.el-api 2.2.4
javax.jdo jdo-api 3.0.1
javax.transaction jta 1.1
javax.transaction transaction-api 1.1
javax.xml.bind jaxb-api 2.2.11
javolution javolution 5.5.1
jline jline 2.14.6
joda-time joda-time 2.10.13
maven-trees hive-2.3__hadoop-3.2 liball_deps_2.12
net.java.dev.jna jna 5.8.0
net.razorvine pickle 1.2
net.sf.jpam jpam 1.1
net.sf.opencsv opencsv 2.3
net.sf.supercsv super-csv 2.2.0
net.snowflake snowflake-ingest-sdk 0.9.6
net.snowflake snowflake-jdbc 3.13.14
net.snowflake spark-snowflake_2.12 2.10.0-spark_3.1
net.sourceforge.f2j arpack_combined_all 0.1
org.acplt.remotetea remotetea-oncrpc 1.1.2
org.antlr ST4 4.0.4
org.antlr antlr-runtime 3.5.2
org.antlr antlr4-runtime 4.8
org.antlr stringtemplate 3.2.1
org.apache.ant ant 1.9.2
org.apache.ant ant-jsch 1.9.2
org.apache.ant ant-launcher 1.9.2
org.apache.arrow arrow-format 7.0.0
org.apache.arrow arrow-memory-core 7.0.0
org.apache.arrow arrow-memory-netty 7.0.0
org.apache.arrow arrow-vector 7.0.0
org.apache.avro avro 1.11.0
org.apache.avro avro-ipc 1.11.0
org.apache.avro avro-mapred 1.11.0
org.apache.commons commons-collections4 4.4
org.apache.commons commons-compress 1.21
org.apache.commons commons-crypto 1.1.0
org.apache.commons commons-lang3 3.12.0
org.apache.commons commons-math3 3.6.1
org.apache.commons commons-text 1.9
org.apache.curator curator-client 2.13.0
org.apache.curator curator-framework 2.13.0
org.apache.curator curator-recipes 2.13.0
org.apache.derby derby 10.14.2.0
org.apache.hadoop hadoop-client-api 3.3.2-databricks
org.apache.hadoop hadoop-client-runtime 3.3.2
org.apache.hive hive-beeline 2.3.9
org.apache.hive hive-cli 2.3.9
org.apache.hive hive-jdbc 2.3.9
org.apache.hive hive-llap-client 2.3.9
org.apache.hive hive-llap-common 2.3.9
org.apache.hive hive-serde 2.3.9
org.apache.hive hive-shims 2.3.9
org.apache.hive hive-storage-api 2.7.2
org.apache.hive.shims hive-shims-0.23 2.3.9
org.apache.hive.shims hive-shims-common 2.3.9
org.apache.hive.shims hive-shims-scheduler 2.3.9
org.apache.httpcomponents httpclient 4.5.13
org.apache.httpcomponents httpcore 4.4.14
org.apache.ivy ivy 2.5.0
org.apache.logging.log4j log4j-1.2-api 2.17.2
org.apache.logging.log4j log4j-api 2.17.2
org.apache.logging.log4j log4j-core 2.17.2
org.apache.logging.log4j log4j-slf4j-impl 2.17.2
org.apache.mesos mesos-shaded-protobuf 1.4.0
org.apache.orc orc-core 1.7.4
org.apache.orc orc-mapreduce 1.7.4
org.apache.orc orc-shims 1.7.4
org.apache.parquet parquet-column 1.12.0-databricks-0004
org.apache.parquet parquet-common 1.12.0-databricks-0004
org.apache.parquet parquet-encoding 1.12.0-databricks-0004
org.apache.parquet parquet-format-structures 1.12.0-databricks-0004
org.apache.parquet parquet-hadoop 1.12.0-databricks-0004
org.apache.parquet parquet-jackson 1.12.0-databricks-0004
org.apache.thrift libfb303 0.9.3
org.apache.thrift libthrift 0.12.0
org.apache.xbean xbean-asm9-shaded 4.20
org.apache.yetus audience-annotations 0.5.0
org.apache.zookeeper zookeeper 3.6.2
org.apache.zookeeper zookeeper-jute 3.6.2
org.checkerframework checker-qual 3.5.0
org.codehaus.jackson jackson-core-asl 1.9.13
org.codehaus.jackson jackson-mapper-asl 1.9.13
org.codehaus.janino commons-compiler 3.0.16
org.codehaus.janino janino 3.0.16
org.datanucleus datanucleus-api-jdo 4.2.4
org.datanucleus datanucleus-core 4.1.17
org.datanucleus datanucleus-rdbms 4.1.19
org.datanucleus javax.jdo 3.2.0-m3
org.eclipse.jetty jetty-client 9.4.46.v20220331
org.eclipse.jetty jetty-continuation 9.4.46.v20220331
org.eclipse.jetty jetty-http 9.4.46.v20220331
org.eclipse.jetty jetty-io 9.4.46.v20220331
org.eclipse.jetty jetty-jndi 9.4.46.v20220331
org.eclipse.jetty jetty-plus 9.4.46.v20220331
org.eclipse.jetty jetty-proxy 9.4.46.v20220331
org.eclipse.jetty jetty-security 9.4.46.v20220331
org.eclipse.jetty jetty-server 9.4.46.v20220331
org.eclipse.jetty jetty-servlet 9.4.46.v20220331
org.eclipse.jetty jetty-servlets 9.4.46.v20220331
org.eclipse.jetty jetty-util 9.4.46.v20220331
org.eclipse.jetty jetty-util-ajax 9.4.46.v20220331
org.eclipse.jetty jetty-webapp 9.4.46.v20220331
org.eclipse.jetty jetty-xml 9.4.46.v20220331
org.eclipse.jetty.websocket websocket-api 9.4.46.v20220331
org.eclipse.jetty.websocket websocket-client 9.4.46.v20220331
org.eclipse.jetty.websocket websocket-common 9.4.46.v20220331
org.eclipse.jetty.websocket websocket-server 9.4.46.v20220331
org.eclipse.jetty.websocket websocket-servlet 9.4.46.v20220331
org.fusesource.leveldbjni leveldbjni-all 1.8
org.glassfish.hk2 hk2-api 2.6.1
org.glassfish.hk2 hk2-locator 2.6.1
org.glassfish.hk2 hk2-utils 2.6.1
org.glassfish.hk2 osgi-resource-locator 1.0.3
org.glassfish.hk2.external aopalliance-repackaged 2.6.1
org.glassfish.hk2.external jakarta.inject 2.6.1
org.glassfish.jersey.containers jersey-container-servlet 2.34
org.glassfish.jersey.containers jersey-container-servlet-core 2.34
org.glassfish.jersey.core jersey-client 2.34
org.glassfish.jersey.core jersey-common 2.34
org.glassfish.jersey.core jersey-server 2.34
org.glassfish.jersey.inject jersey-hk2 2.34
org.hibernate.validator hibernate-validator 6.1.0.Final
org.javassist javassist 3.25.0-GA
org.jboss.logging jboss-logging 3.3.2.Final
org.jdbi jdbi 2.63.1
org.jetbrains annotations 17.0.0
org.joda joda-convert 1.7
org.jodd jodd-core 3.5.2
org.json4s json4s-ast_2.12 3.7.0-M11
org.json4s json4s-core_2.12 3.7.0-M11
org.json4s json4s-jackson_2.12 3.7.0-M11
org.json4s json4s-scalap_2.12 3.7.0-M11
org.lz4 lz4-java 1.8.0
org.mariadb.jdbc mariadb-java-client 2.7.4
org.objenesis objenesis 2.5.1
org.postgresql postgresql 42.3.3
org.roaringbitmap RoaringBitmap 0.9.25
org.roaringbitmap shims 0.9.25
org.rocksdb rocksdbjni 6.24.2
org.rosuda.REngine REngine 2.1.0
org.scala-lang scala-compiler_2.12 2.12.14
org.scala-lang scala-library_2.12 2.12.14
org.scala-lang scala-reflect_2.12 2.12.14
org.scala-lang.modules scala-collection-compat_2.12 2.4.3
org.scala-lang.modules scala-parser-combinators_2.12 1.1.2
org.scala-lang.modules scala-xml_2.12 1.2.0
org.scala-sbt test-interface 1.0
org.scalacheck scalacheck_2.12 1.14.2
org.scalactic scalactic_2.12 3.0.8
org.scalanlp breeze-macros_2.12 1.2
org.scalanlp breeze_2.12 1.2
org.scalatest scalatest_2.12 3.0.8
org.slf4j jcl-over-slf4j 1.7.36
org.slf4j jul-to-slf4j 1.7.36
org.slf4j slf4j-api 1.7.36
org.spark-project.spark unused 1.0.0
org.threeten threeten-extra 1.5.0
org.tukaani xz 1.8
org.typelevel algebra_2.12 2.0.1
org.typelevel cats-kernel_2.12 2.1.1
org.typelevel macro-compat_2.12 1.1.1
org.typelevel spire-macros_2.12 0.17.0
org.typelevel spire-platform_2.12 0.17.0
org.typelevel spire-util_2.12 0.17.0
org.typelevel spire_2.12 0.17.0
org.wildfly.openssl wildfly-openssl 1.0.7.Final
org.xerial sqlite-jdbc 3.8.11.2
org.xerial.snappy snappy-java 1.1.8.4
org.yaml snakeyaml 1.24
oro oro 2.0.8
pl.edu.icm JLargeArrays 1.5
software.amazon.ion ion-java 1.0.2
stax stax-api 1.0.1