Databricks Runtime 14.2

  • Article

The following release notes provide information about Databricks Runtime 14.2, powered by Apache Spark 3.5.0.

Databricks released these images in November 2023.

New features and improvements

CACHE SELECT is now ignored

The SQL command CACHE SELECT is now ignored in favor of an improved caching algorithm. See Optimize performance with caching on Azure Databricks.

Improved caching behavior

DiskCache placement by the Spark scheduler now has faster partition assignment, better balancing during cluster autoscaling, and more consistent query execution.

Fixed corrupt file handling in DML commands

The DML commands DELETE, UPDATE, and MERGE INTO no longer respect the read options ignoreCorruptFiles and ignoreMissingFiles. When encountering an unreadable file in a table, these commands now fail even if these options are specified.

You can now use the from_avro, to_avro, from_protobuf, and to_protobuf Python functions with Schema Registry in shared clusters.

foreachBatch and StreamingListener support

You can now use the foreachBatch() and StreamingListener APIs with Structured Streaming in shared clusters. See Use foreachBatch to write to arbitrary data sinks and Monitoring Structured Streaming queries on Azure Databricks.

Row-level concurrency is Generally Available and on by default

Row-level concurrency reduces conflicts between concurrent write operations by detecting changes at the row-level. Row-level concurrency is only supported on tables without partitioning, which includes tables with liquid clustering. Row-level concurrency is enabled by default on Delta tables with deletion vectors enabled. See Write conflicts with row-level concurrency.

Delta Sharing: Recipients can perform batch, CDF, and streaming queries on shared tables with deletion vectors (Public Preview)

Delta Sharing recipients can now perform batch, CDF, and streaming queries on shared tables that use deletion vectors. On Databricks Runtime 14.1, they can only perform batch queries. See Add tables with deletion vectors or column mapping to a share, Read tables with deletion vectors or column mapping enabled, and Read tables with deletion vectors or column mapping enabled.

Shallow clone for Unity Catalog external tables (Public Preview)

You can now use shallow clone with Unity Catalog external tables. See Shallow clone for Unity Catalog tables.

New disk caching assignment algorithm

The Spark scheduler now uses a new disk caching algorithm. The algorithm improves disk usage and partition assignment across nodes, with faster assignment both initially and after cluster scaling events. Stickier cache assignment improves consistency across runs and reduces data moved during rebalancing operations.

from_avro with schema registry connector adds support for schema evolution

You can now enable your pipelines to restart when evolved records are detected. Previously, if schema evolution happened with the from_avro connector, the new columns would return null. See Read and write streaming Avro data.

Faster multi-threaded statistics collection

Statistics collection is up to 10 times faster on small clusters when running CONVERT TO DELTA or cloning from Iceberg and Parquet tables. See Convert to Delta Lake and Clone Parquet and Iceberg tables.

Pushdown filters in the DeltaSource on Delta files

For better utilization, partition filters on Delta tables streaming queries are now pushed down to Delta before rate limiting.

Support for Scala scalar user-defined functions on shared clusters (Public Preview)

You can now use Scala scalar user-defined functions on Unity Catalog-enabled clusters configured with shared access mode. See User-defined scalar functions - Scala.

Library upgrades

  • Upgraded Python libraries:
    • fastjsonschema from 2.18.0 to 2.19.0
    • filelock from 3.12.3 to 3.12.4
    • googleapis-common-protos from 1.60.0 to 1.61.0
  • Upgraded R libraries:
  • Upgraded Java libraries:

Apache Spark

Databricks Runtime 14.2 includes Apache Spark 3.5.0. This release includes all Spark fixes and improvements included in Databricks Runtime 14.1, as well as the following additional bug fixes and improvements made to Spark:

  • [SPARK-45592] [DBRRM-624] Revert "[SC-146977][SQL] Correctness issue in AQE with InMemoryTableScanExec"
  • [SPARK-45524] [DBRRM-625] Revert "[SC-146319][PYTHON][SQL] Initial support f…
  • [SPARK-45433] [DBRRM-621] Revert "[SC-145163][SQL] Fix CSV/JSON schema infer…
  • [SPARK-45592] [SC-146977][SQL] Correctness issue in AQE with InMemoryTableScanExec
  • [SPARK-45354] Revert "[SC-143991][SQL] Resolve functions bottom-up"
  • [SPARK-45680] [TEST-ONLY][SASP-2347][CONNECT] Fix tests
  • [SPARK-45680] [CONNECT] Release session
  • [SPARK-43380] [SC-146726][ES-897115][SQL] Fix slowdown in Avro read
  • [SPARK-45649] [SC-146742][SQL] Unify the prepare framework for OffsetWindowFunctionFrame
  • [SPARK-40820] [SC-146727][PYTHON][SQL] Creating StructType from Json
  • [SPARK-45620] [SC-146225][PYTHON] Fix user-facing APIs related to Python UDTF to use camelCase
  • [SPARK-45727] [SC-146978][SS] Remove unused map in watermark propagation simulation
  • [SPARK-45723] [SC-146890][PYTHON][CONNECT] Catalog methods avoid pandas conversion
  • [SPARK-45574] [SC-146106][SQL] Add :: syntax as a shorthand for casting
  • [SPARK-45670] [SC-146725][CORE][3.5] SparkSubmit does not support --total-executor-cores when deploying on K8s
  • [SPARK-45661] [SC-146594][SQL][PYTHON] Add toNullable in StructType, MapType and ArrayType
  • [SPARK-45524] [SC-146319][PYTHON][SQL] Initial support for Python data source read API
  • [SPARK-45542] [SC-145892][CORE] Replace setSafeMode(HdfsConstants.SafeModeAction, boolean) with setSafeMode(SafeModeAction, boolean)
  • [SPARK-45652] [SC-146641][SQL] SPJ: Handle empty input partitions after dynamic filtering
  • [SPARK-45454] [SC-144846][SQL] Set the table's default owner to current_user
  • [SPARK-45554] [SC-146875][PYTHON] Introduce flexible parameter to assertSchemaEqual
  • [SPARK-45242] [SC-143410][SQL] Use DataFrame ID to semantically validate CollectMetrics
  • [SPARK-45674] [SC-146862][CONNECT][PYTHON] Improve error message for JVM-dependent attributes on Spark Connect.
  • [SPARK-45646] [SC-146276][SQL] Remove hardcoding time variables prior to Hive 2.0
  • [SPARK-43380] [SC-146726][ES-897115][SQL] Fix slowdown in Avro read
  • [SPARK-45545] [SC-146484][CORE] Pass SSLOptions wherever we create a SparkTransportConf
  • [SPARK-45641] [SC-146260][UI] Display the application start time on AllJobsPage
  • [SPARK-40154] [SC-146491][Python][Docs] Correct storage level in Dataframe.cache docstring
  • [SPARK-45626] [SC-146259][SQL] Convert _LEGACY_ERROR_TEMP_1055 to REQUIRES_SINGLE_PART_NAMESPACE
  • [SPARK-45507] [SC-145900][SQL] Correctness fix for nested correlated scalar subqueries with COUNT aggregates
  • [SPARK-45619] [SC-146226][CONNECT][PYTHON] Apply the observed metrics to Observation object
  • [SPARK-45588] [SC-145909][PROTOBUF][CONNECT][MINOR] Scaladoc improvement for StreamingForeachBatchHelper
  • [SPARK-45616] [SC-146120][CORE] Avoid ParVector, which does not propagate ThreadLocals or SparkSession
  • [SPARK-45604] [SC-146105][SQL] Add LogicalType checking on INT64 -> DateTime conversion on Parquet Vectorized Reader
  • [SPARK-44649] [SC-145891][SQL] Runtime Filter supports passing equivalent creation side expressions
  • [SPARK-41674] [SC-128408][SQL] Runtime filter should supports multi level shuffle join side as filter creation side
  • [SPARK-45547] [SC-146228][ML] Validate Vectors with built-in function
  • [SPARK-45558] [SC-145890][SS] Introduce a metadata file for streaming stateful operator
  • [SPARK-45618] [SC-146252][CORE] Remove BaseErrorHandler
  • [SPARK-45638] [SC-146251][SQL][Avro] Improve test coverage of decimal conversion
  • [SPARK-44837] [SC-146020][SQL] Change ALTER TABLE ALTER PARTITION column error message
  • [SPARK-45561] [SC-146264][SQL] Add proper conversions for TINYINT in MySQLDialect
  • [SPARK-45632] [SC-146255][SQL] Table cache should avoid unnecessary ColumnarToRow when enable AQE
  • [SPARK-45569] [SC-145932][SQL] Assign name to the error _LEGACY_ERROR_TEMP_2152
  • [SPARK-44784] [SC-141015][CONNECT] Make SBT testing hermetic.
  • [SPARK-45452] [SC-144836][SQL] Improve InMemoryFileIndex to use FileSystem.listFiles API
  • [SPARK-44735] [SC-146186][SQL] Add warning msg when inserting columns with the same name by row that don't match up
  • [SPARK-45351] [SC-144791][CORE] Change spark.shuffle.service.db.backend default value to ROCKSDB
  • [SPARK-45553] [SC-145906][PS] Deprecate assertPandasOnSparkEqual
  • [SPARK-45613] [SC-146022][CORE] Expose DeterministicLevel as a DeveloperApi
  • [SPARK-45549] [SC-145895][CORE] Remove unused numExistingExecutors in CoarseGrainedSchedulerBackend
  • [SPARK-45609] [SC-146030][CONNECT] Include SqlState in SparkThrowable proto message
  • [SPARK-45595] [SC-146037] expose sqlstate in error message
  • [SPARK-45628] [SC-146123][ML] Improve vector_to_array for .mllib.linalg.SparseVector
  • [SPARK-45009] [SC-145884][SQL][FOLLOW UP] Turn off decorrelation in join conditions for AQE InSubquery test
  • [SPARK-45576] [SC-145883][CORE] Remove unnecessary debug logs in ReloadingX509TrustManagerSuite
  • [SPARK-45485] [SC-145719][CONNECT] User agent improvements: Use SPARK_CONNECT_USER_AGENT env variable and include environment specific attributes
  • [SPARK-45508] [SC-145513][CORE] Add "-add-opens=java.base/jdk.internal.ref=ALL-UNNAMED" so Platform can access Cleaner on Java 9+
  • [SPARK-45595] Revert "[SC-146037] Expose SQLSTATE in error message"
  • [SPARK-45595] [SC-146037] Expose SQLSTATE in error message
  • [SPARK-45112] Revert "[SC-143259][SQL] Use UnresolvedFunction ba…
  • [SPARK-45257] [SC-143411][CORE] Enable spark.eventLog.compress by default
  • [SPARK-45586] [SC-145899][SQL] Reduce compiler latency for plans with large expression trees
  • [SPARK-45517] [SC-145691][CONNECT][14.x] Expand more exception constructors to support error framework parameters
  • [SPARK-45581] [SC-145896] Make SQLSTATE mandatory.
  • [SPARK-45427] [SC-145590][CORE] Add RPC SSL settings to SSLOptions and SparkTransportConf
  • [SPARK-45261] [SC-143357][CORE] Fix EventLogFileWriters to handle none as a codec
  • [SPARK-45582] [SC-145897][SS] Ensure that store instance is not used after calling commit within output mode streaming aggregation
  • [SPARK-45562] [SC-145720][SC-144650][SQL] XML: Make 'rowTag' a required option
  • [SPARK-45392] [SC-144362][CORE][SQL][SS] Replace Class.newInstance() with Class.getDeclaredConstructor().newInstance()
  • [SPARK-45433] [SC-145163][SQL] Fix CSV/JSON schema inference when timestamps do not match specified timestampFormat
  • [SPARK-45458] [SC-145011][SQL] Convert IllegalArgumentException to SparkIllegalArgumentException in bitwiseExpressions
  • [SPARK-45564] [SC-145735][SQL] Simplify 'DataFrameStatFunctions.bloomFilter' with 'BloomFilterAggregate' expression
  • [SPARK-45566] [SC-145777][PS] Support Pandas-like testing utils for Pandas API on Spark
  • [SPARK-45498] [SC-145509][CORE] Followup: Ignore task completion from old stage a…
  • [SPARK-45483] [SC-145152][CONNECT] Correct the function groups in connect.functions
  • [SPARK-45577] [SC-145781][PYTHON] Fix UserDefinedPythonTableFunctionAnalyzeRunner to pass folded values from named arguments
  • [SPARK-45009] [SC-138223][FOLLOW UP] Turn off decorrelation in join conditions for AetherSQLQuerySuite
  • [SPARK-45415] [SC-145344] Allow selective disabling of "fallocate" in RocksDB statestore
  • [SPARK-45500] [SC-145134][CORE][WEBUI] Show the number of abnormally completed drivers in MasterPage
  • [SPARK-44120] [SC-144349][PYTHON] Support Python 3.12
  • [SPARK-45439] [SC-145589][SQL][UI] Reduce memory usage of LiveStageMetrics.accumIdsToMetricType
  • [SPARK-45565] [SC-145717][UI] Unnecessary JSON.stringify and JSON.parse loop for task list on stage detail
  • [SPARK-45567] [SC-145718][CONNECT] Remove redundant if in org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender#run
  • [SPARK-45112] [SC-143259][SQL] Use UnresolvedFunction based resolution in SQL Dataset functions
  • [SPARK-45486] [SASP-2457][SC-145612][CONNECT] Make add_artifact request idempotent
  • [SPARK-44913] [SC-144326][SQL] DS V2 supports push down V2 UDF that has magic method
  • [SPARK-45538] [SC-145635][PYTHON][CONNECT]pyspark connect overwrite_partitions bug
  • [SPARK-45491] [SC-145622] Add missing SQLSTATES 2/2
  • [SPARK-45009] [SC-145620][SC-138223][SQL] Decorrelate predicate subqueries in join condition
  • [SPARK-45377] [SC-144377][CORE] Handle InputStream in NettyLogger
  • [SPARK-45248] [SC-143759][CORE]Set the timeout for spark ui server
  • [SPARK-44594] [SC-145611][SS] Remove redundant method parameter in kafka connector
  • [SPARK-45539] [SC-145621][SS] Add assert and log to indicate watermark definition is required for streaming aggregation queries in append mode
  • [SPARK-44262] [SC-145639][SQL] Add dropTable and getInsertStatement to JdbcDialect
  • [SPARK-45516] [SC-145480][CONNECT] Include QueryContext in SparkThrowable proto message
  • [SPARK-45310] [SC-145127][CORE] Report shuffle block status should respect shuffle service during decommission migration
  • [SPARK-45521] [SC-145487][ML] Avoid re-computation of nnz in VectorAssembler
  • [SPARK-45418] [SC-145485][SQL][PYTHON][CONNECT] Change current_database() column alias to current_schema()
  • [SPARK-45116] [SC-145142][SQL] Add some comment for param of JdbcDialect createTable
  • [SPARK-45495] [SC-145529][core] Support stage level task resource profile for k8s cluster when dynamic allocation disabled
  • [SPARK-45487] [SC-145559] Fix SQLSTATEs and temp errors
  • [SPARK-45505] [SC-145550][PYTHON] Refactor analyzeInPython to make it reusable
  • [SPARK-45132] [SC-145425][SQL] Fix IDENTIFIER for function invocation
  • [SPARK-45213] [SC-145020][SQL] Assign name to the error _LEGACY_ERROR_TEMP_2151
  • [SPARK-45416] [SC-145126][CONNECT] Sanity check consistency of Arrow results
  • [SPARK-45163] [SC-143386][SQL] Merge UNSUPPORTED_VIEW_OPERATION & UNSUPPORTED_TABLE_OPERATION & fix some issue
  • [SPARK-45421] [SC-144814][SQL] Catch AnalysisException over InlineCTE
  • [SPARK-43664] [SC-145404][CONNECT][PS] Raise exception for ps.sql with Pandas-on-Spark object on Spark Connect.
  • [SPARK-45402] [SC-145371][SQL][PYTHON] Add UDTF API for 'eval' and 'terminate' methods to consume previous 'analyze' result
  • [SPARK-36112] [SC-67885] [SQL] Support correlated EXISTS and IN subqueries using DecorrelateInnerQuery framework
  • [SPARK-45383] [SC-144929][SQL] Fix error message for time travel with non-existing table
  • [SPARK-43254] [SC-143318][SQL] Assign a name to the error _LEGACY_ERROR_TEMP_2018
  • [SPARK-45204] [SC-145167][CONNECT] Add optional ExecuteHolder to SparkConnectPlanner
  • [SPARK-45451] [SC-145144][SQL] Make the default storage level of dataset cache configurable
  • [SPARK-45192] [SC-143194][UI] Fix overdue lineInterpolate parameter for graphviz edge
  • [SPARK-45467] [SC-145146][CORE] Replace Proxy.getProxyClass() with Proxy.newProxyInstance().getClass
  • [SPARK-45397] [SC-145139][ML][CONNECT] Add array assembler feature transformer
  • [SPARK-44855] [SC-145226][CONNECT] Small tweaks to attaching ExecuteGrpcResponseSender to ExecuteResponseObserver
  • [SPARK-45398] [SC-144796][SQL] Append ESCAPE in sql() of the Like expression
  • [SPARK-45494] [SC-145129][CORE][PYTHON] Introduce util functions to read/write a byte array for PythonWorkerUtils
  • [SPARK-45464] [SC-145125][CORE] Fix network-yarn distribution build
  • [SPARK-45461] [SC-144851][CORE][SQL][MLLIB] Introduce a mapper for StorageLevel
  • [SPARK-45450] [SC-145009][PYTHON] Fix imports according to PEP8: pyspark.pandas and pyspark (core)
  • [SPARK-45475] [SC-145018][SQL] Uses DataFrame.foreachPartition instead of RDD.foreachPartition in JdbcUtils
  • [SPARK-45271] [SC-143778][SQL] Merge _LEGACY_ERROR_TEMP_1113 into TABLE_OPERATION & delete some unused method in QueryCompilationErrors
  • [SPARK-45449] [SC-145013][SQL] Cache Invalidation Issue with JDBC Table
  • [SPARK-45473] [SC-145017][SQL] Fix incorrect error message for RoundBase
  • [SPARK-43299] [SC-145022][SS][CONNECT] Convert StreamingQueryException in Scala Client
  • [SPARK-45474] [SC-145014][CORE][WEBUI] Support top-level filtering in MasterPage JSON API
  • [SPARK-45205] [SC-145012][SQL] CommandResultExec to override iterator methods to avoid triggering multiple jobs.
  • [SPARK-45472] [SC-145004][SS] RocksDB State Store Doesn't Need to Recheck checkpoint path existence
  • [SPARK-45470] [SC-145010][SQL] Avoid paste string value of hive orc compression kind
  • [SPARK-45448] [SC-144829][PYTHON] Fix imports according to PEP8: pyspark.testing, pyspark.mllib, pyspark.resource and pyspark.streaming
  • [SPARK-45446] [SC-144828][PYTHON] Fix imports according to PEP8: pyspark.errors and pyspark.ml
  • [SPARK-45239] [SC-144832][CONNECT] Reduce default spark.connect.jvmStacktrace.maxSize
  • [SPARK-45436] [SC-144831][PYTHON][CONNECT] DataFrame methods check same session
  • [SPARK-45413] [SC-144847][CORE] Add warning for prepare drop LevelDB support
  • [SPARK-45462] [SC-144848][CORE][WEBUI] Show Duration in ApplicationPage
  • [SPARK-44527] [SC-144855][SQL] Replace ScalarSubquery with null if its maxRows is 0
  • [SPARK-45401] [SC-144854][PYTHON] Add a new method cleanup in the UDTF interface
  • [SPARK-43704] [SC-144849][CONNECT][PS] Support MultiIndex for to_series()
  • [SPARK-45424] [SC-144888][SQL] Fix TimestampFormatter return optional parse results when only prefix match
  • [SPARK-45441] [SC-144833][PYTHON] Introduce more util functions for PythonWorkerUtils
  • [SPARK-45412] [SC-144803][PYTHON][CONNECT] Validate the plan and session in DataFrame.__init__
  • [SPARK-45408] [SC-144810][CORE] Add RPC SSL settings to TransportConf
  • [SPARK-45432] [SC-144818][CORE] Remove deprecated Hadoop-2 LocatedFileStatus constructor
  • [SPARK-45434] [SC-144819][ML][CONNECT] LogisticRegression checks the training labels
  • [SPARK-45420] [SC-144808][SQL][PYTHON][CONNECT] Add DataType.fromDDL into PySpark
  • [SPARK-45406] [SC-144793][PYTHON][CONNECT] Delete schema from DataFrame constructor
  • [SPARK-45404] [SC-144799][CORE] Support AWS_ENDPOINT_URL env variable
  • [SPARK-43620] [SC-144792][CONNECT][PS] Fix Pandas APIs depends on unsupported features
  • [SPARK-45354] [SC-143991][SQL] Resolve functions bottom-up
  • [SPARK-45394] [SASP-1480][SC-144712][PYTHON][CONNECT] Add retries for artifact API. Improve error handling (follow-up to [SPARK-45093]).
  • [SPARK-45120] [SC-142780][SPARK-45150][UI] Upgrade d3 from v3 to v7(v7.8.5) and apply api changes in UI
  • [SPARK-44838] [SC-143983][SQL] raise_error improvement
  • [SPARK-45312] [SC-143754][SQL][UI] Support toggle display/hide plan svg on execution page
  • [SPARK-45378] [SC-144448][CORE] Add convertToNettyForSsl to ManagedBuffer
  • [SPARK-44762] [SC-144338][CONNECT][CORE] Doc for SparkConnect.addJobTag and Connect SparkSession.addTag
  • [SPARK-45383] Handle unresolved RelationTimeTravel gracefully
  • [SPARK-45347] [SC-144512][SQL][CONNECT] Include SparkThrowable in FetchErrorDetailsResponse
  • [SPARK-45012] [SC-141512][SQL] CheckAnalysis should throw inlined plan in AnalysisException
  • [SPARK-45359] [SC-144336][PYTHON][CONNECT] DataFrame.{columns, colRegex, explain} should raise exceptions when plan is invalid
  • [SPARK-45227] [SC-144233][CORE] Fix a subtle thread-safety issue with CoarseGrainedExecutorBackend
  • [SPARK-45266] [SC-144171][PYTHON] Refactor ResolveFunctions analyzer rule to delay making lateral join when table arguments are used
  • [SPARK-45266] Revert "[SC-144171][PYTHON] Refactor ResolveFunctions analyzer rule to delay making lateral join when table arguments are used"
  • [SPARK-45371] [SC-144389][CONNECT] Fix shading issues in the Spark Connect Scala Client
  • [SPARK-45057] [SC-144214][CORE] Avoid acquire read lock when keepReadLock is false
  • [SPARK-45385] [SC-144357][SQL] Deprecate spark.sql.parser.escapedStringLiterals
  • [SPARK-45340] [SC-143917][SQL] Remove the SQL config spark.sql.hive.verifyPartitionPath
  • [SPARK-45266] [SC-144171][PYTHON] Refactor ResolveFunctions analyzer rule to delay making lateral join when table arguments are used
  • [SPARK-45362] [SC-144198][PYTHON] Project out PARTITION BY expressions before Python UDTF 'eval' method consumes them
  • [SPARK-45346] [SC-143925][SQL] Parquet schema inference should respect case sensitive flag when merging schema
  • [SPARK-45341] [SC-143933][CORE] Correct the title level in the comments of KVStore.java to make sbt doc run successfully with Java 17
  • [SPARK-45334] [SC-143914][SQL] Remove misleading comment in parquetSchemaConverter
  • [SPARK-45337] [SC-143932][CORE] Refactor AbstractCommandBuilder#getScalaVersion to remove the check for Scala 2.12
  • [SPARK-45329] [SC-143918][PYTHON][CONNECT] DataFrame methods skip pandas conversion
  • [SPARK-43662] [SC-143923][PS][CONNECT] Support merge_asof in Spark Connect
  • [SPARK-44126] [SC-143751][CORE] Shuffle migration failure count should not increase when target executor decommissioned
  • [SPARK-44550] [SC-119768][SQL] Enable correctness fixes for null IN (empty list) under ANSI
  • [SPARK-45316] [SC-143783][CORE][SQL] Add new parameters ignoreCorruptFiles/ignoreMissingFiles to HadoopRDD and NewHadoopRDD
  • [SPARK-45093] [SC-143414][CONNECT][PYTHON] Error reporting for addArtifacts query
  • [SPARK-44756] [SC-143787][CORE] Executor hangs when RetryingBlockTransferor fails to initiate retry
  • [SPARK-45333] [SC-143768][CORE] Fix one unit mistake related to spark.eventLog.buffer.kb
  • [SPARK-45317] [SC-143750][SQL][CONNECT] Handle null filename in stack traces of exceptions
  • [SPARK-45191] [SC-143407][SQL] InMemoryTableScanExec simpleStringWithNodeId adds columnar info
  • [SPARK-45235] [SC-143314][CONNECT][PYTHON] Support map and array parameters by sql()
  • [SPARK-45138] [SC-143309][SS] Define a new error class and apply it when checkpointing state to DFS fails
  • [SPARK-45297] [SC-143644][SQL] Remove workaround for dateformatter added in SPARK-31827
  • [SPARK-44345] [SC-143593][CORE] Lower unknown shuffle map output log level to WARN if shuffle migration is enabled
  • [SPARK-44463] [SC-143251][SS][CONNECT] Improve error handling for Connect steaming Python worker
  • [SPARK-45207] [SC-143603][SQL][CONNECT] Implement Error Enrichment for Scala Client
  • [SPARK-45240] [SC-143602][SQL][CONNECT] Implement Error Enrichment for Python Client
  • [SPARK-45216] [SC-143288][SQL] Fix non-deterministic seeded Dataset APIs
  • [SPARK-45251] [SC-143419][CONNECT] Add client_type field for FetchErrorDetails
  • [SPARK-45137] [SC-143001][CONNECT] Support map/array parameters in parameterized sql()
  • [SPARK-45224] [SC-143257][PYTHON] Add examples w/ map and array as parameters of sql()
  • [SPARK-44622] [SC-143205][SQL][CONNECT] Implement FetchErrorDetails RPC
  • [SPARK-45178] [SC-136089] Fallback to execute a single batch for Trigger.AvailableNow with unsupported sources rather than using wrapper
  • [SPARK-44823] [14.x][SC-142979][PYTHON] Update black to 23.9.1 and fixerroneous check
  • [SPARK-45078] [SC-142954][SQL] Fix array_insert ImplicitCastInputTypes not work
  • [SPARK-44579] [SC-138428][SQL] Support Interrupt On Cancel in SQLExecution
  • [SPARK-45252] [SC-143371][CORE] Escape the greater/less than symbols in the comments to make sbt doc execute successfully
  • [SPARK-45189] [SC-142973][SQL] Creating UnresolvedRelation from TableIdentifier should include the catalog field
  • [SPARK-45229] [SC-143254][CORE][UI] Show the number of drivers waiting in SUBMITTED status in MasterPage
  • [SPARK-43453] Revert "[SC-143135][PS] Ignore the names of MultiIndex when axis=1 for concat"

Databricks ODBC/JDBC driver support

Databricks supports ODBC/JDBC drivers released in the past 2 years. Please download the recently released drivers and upgrade (download ODBC, download JDBC).

System environment

  • Operating System: Ubuntu 22.04.3 LTS
  • Java: Zulu 8.72.0.17-CA-linux64
  • Scala: 2.12.15
  • Python: 3.10.12
  • R: 4.3.1
  • Delta Lake: 3.0.0

Installed Python libraries

Library Version Library Version Library Version
anyio 3.5.0 argon2-cffi 21.3.0 argon2-cffi-bindings 21.2.0
asttokens 2.0.5 attrs 22.1.0 backcall 0.2.0
beautifulsoup4 4.11.1 black 22.6.0 bleach 4.1.0
blinker 1.4 boto3 1.24.28 botocore 1.27.96
certifi 2022.12.7 cffi 1.15.1 chardet 4.0.0
charset-normalizer 2.0.4 click 8.0.4 comm 0.1.2
contourpy 1.0.5 cryptography 39.0.1 cycler 0.11.0
Cython 0.29.32 databricks-sdk 0.1.6 dbus-python 1.2.18
debugpy 1.6.7 decorator 5.1.1 defusedxml 0.7.1
distlib 0.3.7 docstring-to-markdown 0.11 entrypoints 0.4
executing 0.8.3 facets-overview 1.1.1 fastjsonschema 2.19.0
filelock 3.12.4 fonttools 4.25.0 googleapis-common-protos 1.61.0
grpcio 1.48.2 grpcio-status 1.48.1 httplib2 0.20.2
idna 3.4 importlib-metadata 4.6.4 ipykernel 6.25.0
ipython 8.14.0 ipython-genutils 0.2.0 ipywidgets 7.7.2
jedi 0.18.1 jeepney 0.7.1 Jinja2 3.1.2
jmespath 0.10.0 joblib 1.2.0 jsonschema 4.17.3
jupyter-client 7.3.4 jupyter-server 1.23.4 jupyter_core 5.2.0
jupyterlab-pygments 0.1.2 jupyterlab-widgets 1.0.0 keyring 23.5.0
kiwisolver 1.4.4 launchpadlib 1.10.16 lazr.restfulclient 0.14.4
lazr.uri 1.0.6 lxml 4.9.1 MarkupSafe 2.1.1
matplotlib 3.7.0 matplotlib-inline 0.1.6 mccabe 0.7.0
mistune 0.8.4 more-itertools 8.10.0 mypy-extensions 0.4.3
nbclassic 0.5.2 nbclient 0.5.13 nbconvert 6.5.4
nbformat 5.7.0 nest-asyncio 1.5.6 nodeenv 1.8.0
notebook 6.5.2 notebook_shim 0.2.2 numpy 1.23.5
oauthlib 3.2.0 packaging 22.0 pandas 1.5.3
pandocfilters 1.5.0 parso 0.8.3 pathspec 0.10.3
patsy 0.5.3 pexpect 4.8.0 pickleshare 0.7.5
Pillow 9.4.0 pip 22.3.1 platformdirs 2.5.2
plotly 5.9.0 pluggy 1.0.0 prometheus-client 0.14.1
prompt-toolkit 3.0.36 protobuf 4.24.0 psutil 5.9.0
psycopg2 2.9.3 ptyprocess 0.7.0 pure-eval 0.2.2
pyarrow 8.0.0 pyarrow-hotfix 0.4 pycparser 2.21
pydantic 1.10.6 pyflakes 3.1.0 Pygments 2.11.2
PyGObject 3.42.1 PyJWT 2.3.0 pyodbc 4.0.32
pyparsing 3.0.9 pyright 1.1.294 pyrsistent 0.18.0
python-dateutil 2.8.2 python-lsp-jsonrpc 1.1.1 python-lsp-server 1.8.0
pytoolconfig 1.2.5 pytz 2022.7 pyzmq 23.2.0
requests 2.28.1 rope 1.7.0 s3transfer 0.6.2
scikit-learn 1.1.1 scipy 1.10.0 seaborn 0.12.2
SecretStorage 3.3.1 Send2Trash 1.8.0 setuptools 65.6.3
six 1.16.0 sniffio 1.2.0 soupsieve 2.3.2.post1
ssh-import-id 5.11 stack-data 0.2.0 statsmodels 0.13.5
tenacity 8.1.0 terminado 0.17.1 threadpoolctl 2.2.0
tinycss2 1.2.1 tokenize-rt 4.2.1 tomli 2.0.1
tornado 6.1 traitlets 5.7.1 typing_extensions 4.4.0
ujson 5.4.0 unattended-upgrades 0.1 urllib3 1.26.14
virtualenv 20.16.7 wadllib 1.3.6 wcwidth 0.2.5
webencodings 0.5.1 websocket-client 0.58.0 whatthepatch 1.0.2
wheel 0.38.4 widgetsnbextension 3.6.1 yapf 0.33.0
zipp 1.0.0

Installed R libraries

R libraries are installed from the Posit Package Manager CRAN snapshot on 2023-02-10.

Library Version Library Version Library Version
arrow 12.0.1 askpass 1.1 assertthat 0.2.1
backports 1.4.1 base 4.3.1 base64enc 0.1-3
bit 4.0.5 bit64 4.0.5 blob 1.2.4
boot 1.3-28 brew 1.0-8 brio 1.1.3
broom 1.0.5 bslib 0.5.0 cachem 1.0.8
callr 3.7.3 caret 6.0-94 cellranger 1.1.0
chron 2.3-61 class 7.3-22 cli 3.6.1
clipr 0.8.0 clock 0.7.0 cluster 2.1.4
codetools 0.2-19 colorspace 2.1-0 commonmark 1.9.0
compiler 4.3.1 config 0.3.1 conflicted 1.2.0
cpp11 0.4.4 crayon 1.5.2 credentials 1.3.2
curl 5.0.1 data.table 1.14.8 datasets 4.3.1
DBI 1.1.3 dbplyr 2.3.3 desc 1.4.2
devtools 2.4.5 diagram 1.6.5 diffobj 0.3.5
digest 0.6.33 downlit 0.4.3 dplyr 1.1.2
dtplyr 1.3.1 e1071 1.7-13 ellipsis 0.3.2
evaluate 0.21 fansi 1.0.4 farver 2.1.1
fastmap 1.1.1 fontawesome 0.5.1 forcats 1.0.0
foreach 1.5.2 foreign 0.8-82 forge 0.2.0
fs 1.6.2 future 1.33.0 future.apply 1.11.0
gargle 1.5.1 generics 0.1.3 gert 1.9.2
ggplot2 3.4.2 gh 1.4.0 gitcreds 0.1.2
glmnet 4.1-7 globals 0.16.2 glue 1.6.2
googledrive 2.1.1 googlesheets4 1.1.1 gower 1.0.1
graphics 4.3.1 grDevices 4.3.1 grid 4.3.1
gridExtra 2.3 gsubfn 0.7 gtable 0.3.3
hardhat 1.3.0 haven 2.5.3 highr 0.10
hms 1.1.3 htmltools 0.5.5 htmlwidgets 1.6.2
httpuv 1.6.11 httr 1.4.6 httr2 0.2.3
ids 1.0.1 ini 0.3.1 ipred 0.9-14
isoband 0.2.7 iterators 1.0.14 jquerylib 0.1.4
jsonlite 1.8.7 KernSmooth 2.23-21 knitr 1.43
labeling 0.4.2 later 1.3.1 lattice 0.21-8
lava 1.7.2.1 lifecycle 1.0.3 listenv 0.9.0
lubridate 1.9.2 magrittr 2.0.3 markdown 1.7
MASS 7.3-60 Matrix 1.5-4.1 memoise 2.0.1
methods 4.3.1 mgcv 1.8-42 mime 0.12
miniUI 0.1.1.1 ModelMetrics 1.2.2.2 modelr 0.1.11
munsell 0.5.0 nlme 3.1-162 nnet 7.3-19
numDeriv 2016.8-1.1 openssl 2.0.6 parallel 4.3.1
parallelly 1.36.0 pillar 1.9.0 pkgbuild 1.4.2
pkgconfig 2.0.3 pkgdown 2.0.7 pkgload 1.3.2.1
plogr 0.2.0 plyr 1.8.8 praise 1.0.0
prettyunits 1.1.1 pROC 1.18.4 processx 3.8.2
prodlim 2023.03.31 profvis 0.3.8 progress 1.2.2
progressr 0.13.0 promises 1.2.0.1 proto 1.0.0
proxy 0.4-27 ps 1.7.5 purrr 1.0.1
r2d3 0.2.6 R6 2.5.1 ragg 1.2.5
randomForest 4.7-1.1 rappdirs 0.3.3 rcmdcheck 1.4.0
RColorBrewer 1.1-3 Rcpp 1.0.11 RcppEigen 0.3.3.9.3
readr 2.1.4 readxl 1.4.3 recipes 1.0.6
rematch 1.0.1 rematch2 2.1.2 remotes 2.4.2
reprex 2.0.2 reshape2 1.4.4 rlang 1.1.1
rmarkdown 2.23 RODBC 1.3-20 roxygen2 7.2.3
rpart 4.1.19 rprojroot 2.0.3 Rserve 1.8-11
RSQLite 2.3.1 rstudioapi 0.15.0 rversions 2.1.2
rvest 1.0.3 sass 0.4.6 scales 1.2.1
selectr 0.4-2 sessioninfo 1.2.2 shape 1.4.6
shiny 1.7.4.1 sourcetools 0.1.7-1 sparklyr 1.8.1
spatial 7.3-15 splines 4.3.1 sqldf 0.4-11
SQUAREM 2021.1 stats 4.3.1 stats4 4.3.1
stringi 1.7.12 stringr 1.5.0 survival 3.5-5
sys 3.4.2 systemfonts 1.0.4 tcltk 4.3.1
testthat 3.1.10 textshaping 0.3.6 tibble 3.2.1
tidyr 1.3.0 tidyselect 1.2.0 tidyverse 2.0.0
timechange 0.2.0 timeDate 4022.108 tinytex 0.45
tools 4.3.1 tzdb 0.4.0 urlchecker 1.0.1
usethis 2.2.2 utf8 1.2.3 utils 4.3.1
uuid 1.1-0 vctrs 0.6.3 viridisLite 0.4.2
vroom 1.6.3 waldo 0.5.1 whisker 0.4.1
withr 2.5.0 xfun 0.39 xml2 1.3.5
xopen 1.0.0 xtable 1.8-4 yaml 2.3.7
zip 2.3.0

Installed Java and Scala libraries (Scala 2.12 cluster version)

Group ID Artifact ID Version
antlr antlr 2.7.7
com.amazonaws amazon-kinesis-client 1.12.0
com.amazonaws aws-java-sdk-autoscaling 1.12.390
com.amazonaws aws-java-sdk-cloudformation 1.12.390
com.amazonaws aws-java-sdk-cloudfront 1.12.390
com.amazonaws aws-java-sdk-cloudhsm 1.12.390
com.amazonaws aws-java-sdk-cloudsearch 1.12.390
com.amazonaws aws-java-sdk-cloudtrail 1.12.390
com.amazonaws aws-java-sdk-cloudwatch 1.12.390
com.amazonaws aws-java-sdk-cloudwatchmetrics 1.12.390
com.amazonaws aws-java-sdk-codedeploy 1.12.390
com.amazonaws aws-java-sdk-cognitoidentity 1.12.390
com.amazonaws aws-java-sdk-cognitosync 1.12.390
com.amazonaws aws-java-sdk-config 1.12.390
com.amazonaws aws-java-sdk-core 1.12.390
com.amazonaws aws-java-sdk-datapipeline 1.12.390
com.amazonaws aws-java-sdk-directconnect 1.12.390
com.amazonaws aws-java-sdk-directory 1.12.390
com.amazonaws aws-java-sdk-dynamodb 1.12.390
com.amazonaws aws-java-sdk-ec2 1.12.390
com.amazonaws aws-java-sdk-ecs 1.12.390
com.amazonaws aws-java-sdk-efs 1.12.390
com.amazonaws aws-java-sdk-elasticache 1.12.390
com.amazonaws aws-java-sdk-elasticbeanstalk 1.12.390
com.amazonaws aws-java-sdk-elasticloadbalancing 1.12.390
com.amazonaws aws-java-sdk-elastictranscoder 1.12.390
com.amazonaws aws-java-sdk-emr 1.12.390
com.amazonaws aws-java-sdk-glacier 1.12.390
com.amazonaws aws-java-sdk-glue 1.12.390
com.amazonaws aws-java-sdk-iam 1.12.390
com.amazonaws aws-java-sdk-importexport 1.12.390
com.amazonaws aws-java-sdk-kinesis 1.12.390
com.amazonaws aws-java-sdk-kms 1.12.390
com.amazonaws aws-java-sdk-lambda 1.12.390
com.amazonaws aws-java-sdk-logs 1.12.390
com.amazonaws aws-java-sdk-machinelearning 1.12.390
com.amazonaws aws-java-sdk-opsworks 1.12.390
com.amazonaws aws-java-sdk-rds 1.12.390
com.amazonaws aws-java-sdk-redshift 1.12.390
com.amazonaws aws-java-sdk-route53 1.12.390
com.amazonaws aws-java-sdk-s3 1.12.390
com.amazonaws aws-java-sdk-ses 1.12.390
com.amazonaws aws-java-sdk-simpledb 1.12.390
com.amazonaws aws-java-sdk-simpleworkflow 1.12.390
com.amazonaws aws-java-sdk-sns 1.12.390
com.amazonaws aws-java-sdk-sqs 1.12.390
com.amazonaws aws-java-sdk-ssm 1.12.390
com.amazonaws aws-java-sdk-storagegateway 1.12.390
com.amazonaws aws-java-sdk-sts 1.12.390
com.amazonaws aws-java-sdk-support 1.12.390
com.amazonaws aws-java-sdk-swf-libraries 1.11.22
com.amazonaws aws-java-sdk-workspaces 1.12.390
com.amazonaws jmespath-java 1.12.390
com.clearspring.analytics stream 2.9.6
com.databricks Rserve 1.8-3
com.databricks databricks-sdk-java 0.7.0
com.databricks jets3t 0.7.1-0
com.databricks.scalapb compilerplugin_2.12 0.4.15-10
com.databricks.scalapb scalapb-runtime_2.12 0.4.15-10
com.esotericsoftware kryo-shaded 4.0.2
com.esotericsoftware minlog 1.3.0
com.fasterxml classmate 1.3.4
com.fasterxml.jackson.core jackson-annotations 2.15.2
com.fasterxml.jackson.core jackson-core 2.15.2
com.fasterxml.jackson.core jackson-databind 2.15.2
com.fasterxml.jackson.dataformat jackson-dataformat-cbor 2.15.2
com.fasterxml.jackson.datatype jackson-datatype-joda 2.15.2
com.fasterxml.jackson.datatype jackson-datatype-jsr310 2.15.1
com.fasterxml.jackson.module jackson-module-paranamer 2.15.2
com.fasterxml.jackson.module jackson-module-scala_2.12 2.15.2
com.github.ben-manes.caffeine caffeine 2.9.3
com.github.fommil jniloader 1.1
com.github.fommil.netlib native_ref-java 1.1
com.github.fommil.netlib native_ref-java 1.1-natives
com.github.fommil.netlib native_system-java 1.1
com.github.fommil.netlib native_system-java 1.1-natives
com.github.fommil.netlib netlib-native_ref-linux-x86_64 1.1-natives
com.github.fommil.netlib netlib-native_system-linux-x86_64 1.1-natives
com.github.luben zstd-jni 1.5.5-4
com.github.wendykierp JTransforms 3.1
com.google.code.findbugs jsr305 3.0.0
com.google.code.gson gson 2.10.1
com.google.crypto.tink tink 1.9.0
com.google.errorprone error_prone_annotations 2.10.0
com.google.flatbuffers flatbuffers-java 1.12.0
com.google.guava guava 15.0
com.google.protobuf protobuf-java 2.6.1
com.helger profiler 1.1.1
com.jcraft jsch 0.1.55
com.jolbox bonecp 0.8.0.RELEASE
com.lihaoyi sourcecode_2.12 0.1.9
com.microsoft.azure azure-data-lake-store-sdk 2.3.9
com.microsoft.sqlserver mssql-jdbc 11.2.2.jre8
com.ning compress-lzf 1.1.2
com.sun.mail javax.mail 1.5.2
com.sun.xml.bind jaxb-core 2.2.11
com.sun.xml.bind jaxb-impl 2.2.11
com.tdunning json 1.8
com.thoughtworks.paranamer paranamer 2.8
com.trueaccord.lenses lenses_2.12 0.4.12
com.twitter chill-java 0.10.0
com.twitter chill_2.12 0.10.0
com.twitter util-app_2.12 7.1.0
com.twitter util-core_2.12 7.1.0
com.twitter util-function_2.12 7.1.0
com.twitter util-jvm_2.12 7.1.0
com.twitter util-lint_2.12 7.1.0
com.twitter util-registry_2.12 7.1.0
com.twitter util-stats_2.12 7.1.0
com.typesafe config 1.2.1
com.typesafe.scala-logging scala-logging_2.12 3.7.2
com.uber h3 3.7.3
com.univocity univocity-parsers 2.9.1
com.zaxxer HikariCP 4.0.3
commons-cli commons-cli 1.5.0
commons-codec commons-codec 1.16.0
commons-collections commons-collections 3.2.2
commons-dbcp commons-dbcp 1.4
commons-fileupload commons-fileupload 1.5
commons-httpclient commons-httpclient 3.1
commons-io commons-io 2.13.0
commons-lang commons-lang 2.6
commons-logging commons-logging 1.1.3
commons-pool commons-pool 1.5.4
dev.ludovic.netlib arpack 3.0.3
dev.ludovic.netlib blas 3.0.3
dev.ludovic.netlib lapack 3.0.3
info.ganglia.gmetric4j gmetric4j 1.0.10
io.airlift aircompressor 0.25
io.delta delta-sharing-client_2.12 1.0.2
io.dropwizard.metrics metrics-annotation 4.2.19
io.dropwizard.metrics metrics-core 4.2.19
io.dropwizard.metrics metrics-graphite 4.2.19
io.dropwizard.metrics metrics-healthchecks 4.2.19
io.dropwizard.metrics metrics-jetty9 4.2.19
io.dropwizard.metrics metrics-jmx 4.2.19
io.dropwizard.metrics metrics-json 4.2.19
io.dropwizard.metrics metrics-jvm 4.2.19
io.dropwizard.metrics metrics-servlets 4.2.19
io.netty netty-all 4.1.96.Final
io.netty netty-buffer 4.1.96.Final
io.netty netty-codec 4.1.96.Final
io.netty netty-codec-http 4.1.96.Final
io.netty netty-codec-http2 4.1.96.Final
io.netty netty-codec-socks 4.1.96.Final
io.netty netty-common 4.1.96.Final
io.netty netty-handler 4.1.96.Final
io.netty netty-handler-proxy 4.1.96.Final
io.netty netty-resolver 4.1.96.Final
io.netty netty-tcnative-boringssl-static 2.0.61.Final-linux-aarch_64
io.netty netty-tcnative-boringssl-static 2.0.61.Final-linux-x86_64
io.netty netty-tcnative-boringssl-static 2.0.61.Final-osx-aarch_64
io.netty netty-tcnative-boringssl-static 2.0.61.Final-osx-x86_64
io.netty netty-tcnative-boringssl-static 2.0.61.Final-windows-x86_64
io.netty netty-tcnative-classes 2.0.61.Final
io.netty netty-transport 4.1.96.Final
io.netty netty-transport-classes-epoll 4.1.96.Final
io.netty netty-transport-classes-kqueue 4.1.96.Final
io.netty netty-transport-native-epoll 4.1.96.Final
io.netty netty-transport-native-epoll 4.1.96.Final-linux-aarch_64
io.netty netty-transport-native-epoll 4.1.96.Final-linux-x86_64
io.netty netty-transport-native-kqueue 4.1.96.Final-osx-aarch_64
io.netty netty-transport-native-kqueue 4.1.96.Final-osx-x86_64
io.netty netty-transport-native-unix-common 4.1.96.Final
io.prometheus simpleclient 0.7.0
io.prometheus simpleclient_common 0.7.0
io.prometheus simpleclient_dropwizard 0.7.0
io.prometheus simpleclient_pushgateway 0.7.0
io.prometheus simpleclient_servlet 0.7.0
io.prometheus.jmx collector 0.12.0
jakarta.annotation jakarta.annotation-api 1.3.5
jakarta.servlet jakarta.servlet-api 4.0.3
jakarta.validation jakarta.validation-api 2.0.2
jakarta.ws.rs jakarta.ws.rs-api 2.1.6
javax.activation activation 1.1.1
javax.el javax.el-api 2.2.4
javax.jdo jdo-api 3.0.1
javax.transaction jta 1.1
javax.transaction transaction-api 1.1
javax.xml.bind jaxb-api 2.2.11
javolution javolution 5.5.1
jline jline 2.14.6
joda-time joda-time 2.12.1
net.java.dev.jna jna 5.8.0
net.razorvine pickle 1.3
net.sf.jpam jpam 1.1
net.sf.opencsv opencsv 2.3
net.sf.supercsv super-csv 2.2.0
net.snowflake snowflake-ingest-sdk 0.9.6
net.sourceforge.f2j arpack_combined_all 0.1
org.acplt.remotetea remotetea-oncrpc 1.1.2
org.antlr ST4 4.0.4
org.antlr antlr-runtime 3.5.2
org.antlr antlr4-runtime 4.9.3
org.antlr stringtemplate 3.2.1
org.apache.ant ant 1.9.16
org.apache.ant ant-jsch 1.9.16
org.apache.ant ant-launcher 1.9.16
org.apache.arrow arrow-format 12.0.1
org.apache.arrow arrow-memory-core 12.0.1
org.apache.arrow arrow-memory-netty 12.0.1
org.apache.arrow arrow-vector 12.0.1
org.apache.avro avro 1.11.2
org.apache.avro avro-ipc 1.11.2
org.apache.avro avro-mapred 1.11.2
org.apache.commons commons-collections4 4.4
org.apache.commons commons-compress 1.23.0
org.apache.commons commons-crypto 1.1.0
org.apache.commons commons-lang3 3.12.0
org.apache.commons commons-math3 3.6.1
org.apache.commons commons-text 1.10.0
org.apache.curator curator-client 2.13.0
org.apache.curator curator-framework 2.13.0
org.apache.curator curator-recipes 2.13.0
org.apache.datasketches datasketches-java 3.1.0
org.apache.datasketches datasketches-memory 2.0.0
org.apache.derby derby 10.14.2.0
org.apache.hadoop hadoop-client-runtime 3.3.6
org.apache.hive hive-beeline 2.3.9
org.apache.hive hive-cli 2.3.9
org.apache.hive hive-jdbc 2.3.9
org.apache.hive hive-llap-client 2.3.9
org.apache.hive hive-llap-common 2.3.9
org.apache.hive hive-serde 2.3.9
org.apache.hive hive-shims 2.3.9
org.apache.hive hive-storage-api 2.8.1
org.apache.hive.shims hive-shims-0.23 2.3.9
org.apache.hive.shims hive-shims-common 2.3.9
org.apache.hive.shims hive-shims-scheduler 2.3.9
org.apache.httpcomponents httpclient 4.5.14
org.apache.httpcomponents httpcore 4.4.16
org.apache.ivy ivy 2.5.1
org.apache.logging.log4j log4j-1.2-api 2.20.0
org.apache.logging.log4j log4j-api 2.20.0
org.apache.logging.log4j log4j-core 2.20.0
org.apache.logging.log4j log4j-slf4j2-impl 2.20.0
org.apache.mesos mesos 1.11.0-shaded-protobuf
org.apache.orc orc-core 1.9.1-shaded-protobuf
org.apache.orc orc-mapreduce 1.9.1-shaded-protobuf
org.apache.orc orc-shims 1.9.1
org.apache.thrift libfb303 0.9.3
org.apache.thrift libthrift 0.12.0
org.apache.ws.xmlschema xmlschema-core 2.3.0
org.apache.xbean xbean-asm9-shaded 4.23
org.apache.yetus audience-annotations 0.13.0
org.apache.zookeeper zookeeper 3.6.3
org.apache.zookeeper zookeeper-jute 3.6.3
org.checkerframework checker-qual 3.31.0
org.codehaus.jackson jackson-core-asl 1.9.13
org.codehaus.jackson jackson-mapper-asl 1.9.13
org.codehaus.janino commons-compiler 3.0.16
org.codehaus.janino janino 3.0.16
org.datanucleus datanucleus-api-jdo 4.2.4
org.datanucleus datanucleus-core 4.1.17
org.datanucleus datanucleus-rdbms 4.1.19
org.datanucleus javax.jdo 3.2.0-m3
org.eclipse.jetty jetty-client 9.4.52.v20230823
org.eclipse.jetty jetty-continuation 9.4.52.v20230823
org.eclipse.jetty jetty-http 9.4.52.v20230823
org.eclipse.jetty jetty-io 9.4.52.v20230823
org.eclipse.jetty jetty-jndi 9.4.52.v20230823
org.eclipse.jetty jetty-plus 9.4.52.v20230823
org.eclipse.jetty jetty-proxy 9.4.52.v20230823
org.eclipse.jetty jetty-security 9.4.52.v20230823
org.eclipse.jetty jetty-server 9.4.52.v20230823
org.eclipse.jetty jetty-servlet 9.4.52.v20230823
org.eclipse.jetty jetty-servlets 9.4.52.v20230823
org.eclipse.jetty jetty-util 9.4.52.v20230823
org.eclipse.jetty jetty-util-ajax 9.4.52.v20230823
org.eclipse.jetty jetty-webapp 9.4.52.v20230823
org.eclipse.jetty jetty-xml 9.4.52.v20230823
org.eclipse.jetty.websocket websocket-api 9.4.52.v20230823
org.eclipse.jetty.websocket websocket-client 9.4.52.v20230823
org.eclipse.jetty.websocket websocket-common 9.4.52.v20230823
org.eclipse.jetty.websocket websocket-server 9.4.52.v20230823
org.eclipse.jetty.websocket websocket-servlet 9.4.52.v20230823
org.fusesource.leveldbjni leveldbjni-all 1.8
org.glassfish.hk2 hk2-api 2.6.1
org.glassfish.hk2 hk2-locator 2.6.1
org.glassfish.hk2 hk2-utils 2.6.1
org.glassfish.hk2 osgi-resource-locator 1.0.3
org.glassfish.hk2.external aopalliance-repackaged 2.6.1
org.glassfish.hk2.external jakarta.inject 2.6.1
org.glassfish.jersey.containers jersey-container-servlet 2.40
org.glassfish.jersey.containers jersey-container-servlet-core 2.40
org.glassfish.jersey.core jersey-client 2.40
org.glassfish.jersey.core jersey-common 2.40
org.glassfish.jersey.core jersey-server 2.40
org.glassfish.jersey.inject jersey-hk2 2.40
org.hibernate.validator hibernate-validator 6.1.7.Final
org.ini4j ini4j 0.5.4
org.javassist javassist 3.29.2-GA
org.jboss.logging jboss-logging 3.3.2.Final
org.jdbi jdbi 2.63.1
org.jetbrains annotations 17.0.0
org.joda joda-convert 1.7
org.jodd jodd-core 3.5.2
org.json4s json4s-ast_2.12 3.7.0-M11
org.json4s json4s-core_2.12 3.7.0-M11
org.json4s json4s-jackson_2.12 3.7.0-M11
org.json4s json4s-scalap_2.12 3.7.0-M11
org.lz4 lz4-java 1.8.0
org.mariadb.jdbc mariadb-java-client 2.7.9
org.mlflow mlflow-spark 2.2.0
org.objenesis objenesis 2.5.1
org.postgresql postgresql 42.6.0
org.roaringbitmap RoaringBitmap 0.9.45
org.roaringbitmap shims 0.9.45
org.rocksdb rocksdbjni 8.3.2
org.rosuda.REngine REngine 2.1.0
org.scala-lang scala-compiler_2.12 2.12.15
org.scala-lang scala-library_2.12 2.12.15
org.scala-lang scala-reflect_2.12 2.12.15
org.scala-lang.modules scala-collection-compat_2.12 2.9.0
org.scala-lang.modules scala-parser-combinators_2.12 1.1.2
org.scala-lang.modules scala-xml_2.12 1.2.0
org.scala-sbt test-interface 1.0
org.scalacheck scalacheck_2.12 1.14.2
org.scalactic scalactic_2.12 3.2.15
org.scalanlp breeze-macros_2.12 2.1.0
org.scalanlp breeze_2.12 2.1.0
org.scalatest scalatest-compatible 3.2.15
org.scalatest scalatest-core_2.12 3.2.15
org.scalatest scalatest-diagrams_2.12 3.2.15
org.scalatest scalatest-featurespec_2.12 3.2.15
org.scalatest scalatest-flatspec_2.12 3.2.15
org.scalatest scalatest-freespec_2.12 3.2.15
org.scalatest scalatest-funspec_2.12 3.2.15
org.scalatest scalatest-funsuite_2.12 3.2.15
org.scalatest scalatest-matchers-core_2.12 3.2.15
org.scalatest scalatest-mustmatchers_2.12 3.2.15
org.scalatest scalatest-propspec_2.12 3.2.15
org.scalatest scalatest-refspec_2.12 3.2.15
org.scalatest scalatest-shouldmatchers_2.12 3.2.15
org.scalatest scalatest-wordspec_2.12 3.2.15
org.scalatest scalatest_2.12 3.2.15
org.slf4j jcl-over-slf4j 2.0.7
org.slf4j jul-to-slf4j 2.0.7
org.slf4j slf4j-api 2.0.7
org.threeten threeten-extra 1.7.1
org.tukaani xz 1.9
org.typelevel algebra_2.12 2.0.1
org.typelevel cats-kernel_2.12 2.1.1
org.typelevel spire-macros_2.12 0.17.0
org.typelevel spire-platform_2.12 0.17.0
org.typelevel spire-util_2.12 0.17.0
org.typelevel spire_2.12 0.17.0
org.wildfly.openssl wildfly-openssl 1.1.3.Final
org.xerial sqlite-jdbc 3.42.0.0
org.xerial.snappy snappy-java 1.1.10.3
org.yaml snakeyaml 2.0
oro oro 2.0.8
pl.edu.icm JLargeArrays 1.5
software.amazon.cryptools AmazonCorrettoCryptoProvider 1.6.1-linux-x86_64
software.amazon.ion ion-java 1.0.2
stax stax-api 1.0.1