Databricks Runtime 13.2
The following release notes provide information about Databricks Runtime 13.2, powered by Apache Spark 3.4.0.
Databricks released these images in July 2023.
New features and improvements
- Changelog checkpointing for RocksDB state store
- Enhanced reliability for VACUUM with shallow clone in Unity Catalog.
- Support for Python UDFs in SQL
- Delta Lake UniForm for Iceberg is in public preview
- Delta Lake liquid clustering is in public preview
- Archival support in Azure Databricks
- IDENTIFIER clause support
- Unity Catalog support for Python and Pandas User-Defined Functions (UDFs)
- Share schemas using Delta Sharing
- Bug fixes
Changelog checkpointing for RocksDB state store
You can enable changelog checkpointing to lower checkpoint duration and end-to-end latency for stateful Structured Streaming workloads. Databricks recommends enabling changelog checkpointing for all Structured Streaming stateful queries. See Enable changelog checkpointing.
Enhanced reliability for VACUUM with shallow clone in Unity Catalog.
When you use Unity Catalog managed tables for the source and target of a shallow clone operation, Unity Catalog manages the underlying data files to improve reliability for the source and target of the clone operation. Running VACUUM on the source of a shallow clone does not break the cloned table. See Vacuum and Unity Catalog shallow clones.
Support for Python UDFs in SQL
You can now declare user-defined functions using Python within SQL CREATE FUNCTION statements in Unity Catalog. See User-defined functions (UDFs) in Unity Catalog.
Delta Lake UniForm for Iceberg is in public preview
Delta Lake Universal Format (UniForm) allows you to read Delta tables with Iceberg clients. See Universal Format (UniForm) for Iceberg compatibility with Delta tables.
Delta Lake liquid clustering is in public preview
Delta Lake liquid clustering replaces table partitioning and ZORDER to simplify data layout decisions and optimize query performance. See Use liquid clustering for Delta tables.
Archival support in Azure Databricks
Archival support in Azure Databricks introduces a collection of capabilities that enable you to use cloud-based lifecycle policies on cloud object storage containing Delta tables to move files to archival storage tiers. See Archival support in Azure Databricks.
IDENTIFIER clause support
Use IDENTIFIER(:mytable) to safely parameterize table, column, function, and schema names in many SQL statements.
Unity Catalog support for Python and Pandas User-Defined Functions (UDFs)
On Databricks Runtime 13.2 and above, user-defined functions (UDFs) written in Python are supported on clusters that use shared access mode in a Unity Catalog-enabled workspace. This includes Python and Pandas UDFs. For details about Python UDFs, see User-defined scalar functions - Python.
Share schemas using Delta Sharing
Databricks Runtime 13.2 adds the ability to use ALTER SHARE <share-name> ADD SCHEMA <schema-name> to share an entire schema using Delta Sharing, giving the recipient access to all of the tables and views in the schema at the moment you share it, along with any tables and views that are added to the schema in the future. See Add schemas to a share and ALTER SHARE.
Bug fixes
- Fixed a bug with the JSON reader, where records were parsed in
PERMISSIVEmode iffailOnUnknownFieldsmodewas enabled. Records are now dropped inDROPMALFORMEDparsermode or are written out to thebadRecordsPath, if any of these options are set.FAILFASTthrows an error if a column type doesn't match.
Library upgrades
- Upgraded Python libraries:
- nodeenv from 1.7.0 to 1.8.0
- Upgraded R libraries:
- Upgraded Java libraries:
- io.delta.delta-sharing-spark_2.12 from 0.6.7 to 0.7.0
Apache Spark
Databricks Runtime 13.2 includes Apache Spark 3.4.0. This release includes all Spark fixes and improvements included in Databricks Runtime 13.1 (unsupported), as well as the following additional bug fixes and improvements made to Spark:
- [SPARK-42750] [SC-133452][DBRRM-273] Support Insert By Name statement
- [SPARK-43133] [SC-133728] Scala Client DataStreamWriter Foreach support
- [SPARK-43663] [SC-134135][CONNECT][PS] Enable
SeriesParityTests.test_compare - [SPARK-43684] [SC-134162][SPARK-43685][SPARK-43686][SPARK-43691][CONNECT][PS] Fix
(NullOps|NumOps).(eq|ne)for Spark Connect. - [SPARK-44021] [SC-134109][SQL] Add spark.sql.files.maxPartitionNum
- [SPARK-43910] [SC-132936][SQL] Strip
__auto_generated_subquery_namefrom ids in errors - [SPARK-43976] [SC-133375][CORE] Handle the case where modifiedConfigs doesn't exist in event logs
- [SPARK-32559] [SC-133992][SQL] Fix the trim logic did't handle ASCII control characters correctly
- [SPARK-43421] [SC-132930][SS] Implement Changelog based Checkpointing for RocksDB State Store Provider
- [SPARK-43594] [SC-133851][SQL] Add LocalDateTime to anyToMicros
- [SPARK-43884] [SC-133867] Param markers in DDL
- [SPARK-43398] [SC-133894][CORE] Executor timeout should be max of idle shuffle and rdd timeout
- [SPARK-43610] [SC-133892][CONNECT][PS] Enable
InternalFrame.attach_distributed_columnin Spark Connect. - [SPARK-43971] [SC-133826][CONNECT][PYTHON] Support Python's createDataFrame in streaming manner
- [SPARK-44006] [SC-133652][CONNECT][PYTHON] Support cache artifacts
- [SPARK-43790] [SC-133468][PYTHON][CONNECT][ML] Add
copyFromLocalToFsAPI - [SPARK-43921] [SC-133461][PROTOBUF] Generate Protobuf descriptor files at build time
- [SPARK-43906] [SC-133638][SC-133404][PYTHON][CONNECT] Implement the file support in SparkSession.addArtifacts
- [SPARK-44010] [SC-133744][PYTHON][SS][MINOR] Python StreamingQueryProgress rowsPerSecond type fix
- [SPARK-43768] [SC-132398][PYTHON][CONNECT] Python dependency management support in Python Spark Connect
- [SPARK-43979] [13.X] Revert "[SC-133456][SQL] CollectedMetrics should be treated as the same one for self-join"
- [SPARK-42290] [SC-133696][SQL] Fix the OOM error can't be reported when AQE on
- [SPARK-43943] [SC-133648][SQL][PYTHON][CONNECT] Add SQL math functions to Scala and Python
- [SPARK-43747] [SC-132047][PYTHON][CONNECT] Implement the pyfile support in SparkSession.addArtifacts
- [SPARK-43700] [SC-133726][SPARK-43701][CONNECT][PS] Enable
TimedeltaOps.(sub|rsub)with Spark Connect - [SPARK-43712] [SC-133641][SPARK-43713][CONNECT][PS] Enable parity test:
test_line_plot,test_pie_plot. - [SPARK-43867] [SC-132895][SQL] Improve suggested candidates for unresolved attribute
- [SPARK-43901] [SC-133454][SQL] Avro to Support custom decimal type backed by Long
- [SPARK-43267] [SC-133024][JDBC] Handle postgres unknown user-defined column as string in array
- [SPARK-43935] [SC-133637][SQL][PYTHON][CONNECT] Add xpath_* functions to Scala and Python
- [SPARK-43916] [SC-133647][SQL][PYTHON][CONNECT] Add percentile to Scala and Python API
- [SPARK-43717] [SC-133455][CONNECT] Scala client reduce agg cannot handle null partitions for scala primitive inputs
- [SPARK-43984] [SC-133469][SQL][PROTOBUF] Change to use
foreachwhenmapdoesn't produce results - [SPARK-43612] [SC-132011][CONNECT][PYTHON] Implement SparkSession.addArtifact(s) in Python client
- [SPARK-44002] [SC-133606][CONNECT] Fix the artifact statuses handler
- [SPARK-43933] [SC-133470][SQL][PYTHON][CONNECT] Add linear regression aggregate functions to Scala and Python
- [SPARK-43920] [SC-133614][SQL][CONNECT] Create sql/api module
- [SPARK-43979] [SC-133456][SQL] CollectedMetrics should be treated as the same one for self-join
- [SPARK-43977] [SC-133373][CONNECT] Fix unexpected check result of
dev/connect-jvm-client-mima-check - [SPARK-43985] [SC-133412][PROTOBUF] spark protobuf: fix enums as ints bug
- [SPARK-43930] [SC-133406][SQL][PYTHON][CONNECT] Add unix_* functions to Scala and Python
- [SPARK-43807] [SC-132545][SQL] Migrate _LEGACY_ERROR_TEMP_1269 to PARTITION_SCHEMA_IS_EMPTY
- [SPARK-43956] [SC-133123][SQL] Fix the bug doesn't display column's sql for Percentile[Cont|Disc]
- [SPARK-42626] [SC-133248][CONNECT] Add Destructive Iterator for SparkResult
- [SPARK-43333] [SC-132898][SQL] Allow Avro to convert union type to SQL with field name stable with type
- [SPARK-43953] [SC-133134][CONNECT] Remove
pass - [SPARK-43970] [SC-133403][PYTHON][CONNECT] Hide unsupported dataframe methods from auto-completion
- [SPARK-43973] [SC-133272][SS][UI] Structured Streaming UI should display failed queries correctly
- [SPARK-42577] [SC-125445][CORE] Add max attempts limitation for stages to avoid potential infinite retry
- [SPARK-43841] [SC-132699][SQL] Handle candidate attributes with no prefix in
StringUtils#orderSuggestedIdentifiersBySimilarity - [SPARK-41497] [SC-124795][CORE][Follow UP]Modify config
spark.rdd.cache.visibilityTracking.enabledsupport version to 3.5.0 - [SPARK-37942] [SC-130983][CORE][SQL] Migrate error classes
- [SPARK-43898] [SC-132926][CORE] Automatically register
immutable.ArraySeq$ofReftoKryoSerializerfor Scala 2.13 - [SPARK-43775] [SC-132891][SQL] DataSource V2: Allow representing updates as deletes and inserts
- [SPARK-43817] [SC-132662][SPARK-43702][PYTHON] Support UserDefinedType in createDataFrame from pandas DataFrame and toPandas
- [SPARK-43418] [SC-132753][SC-130685][CONNECT] Add SparkSession.Builder.getOrCreate
- [SPARK-43549] [SC-132383][SQL] Convert _LEGACY_ERROR_TEMP_0036 to INVALID_SQL_SYNTAX.ANALYZE_TABLE_UNEXPECTED_NOSCAN
- [SPARK-41497] [SC-124644][CORE] Fixing accumulator undercount in the case of the retry task with rdd cache
- [SPARK-43892] [SC-133006][PYTHON] Add autocomplete support for
df[|]inpyspark.sql.dataframe.DataFrame - [SPARK-43881] [SC-133140][SQL][PYTHON][CONNECT] Add optional pattern for Catalog.listDatabases
- [SPARK-43205] [SC-132623] IDENTIFIER clause
- [SPARK-43545] [SC-132378][SQL][PYTHON] Support nested timestamp type
- [SPARK-43949] [SC-133048][PYTHON] Upgrade cloudpickle to 2.2.1
- [SPARK-43760] [SC-132809][SQL] Nullability of scalar subquery results
- [SPARK-43696] [SC-132737][SPARK-43697][SPARK-43698][SPARK-43699][PS] Fix
TimedeltaOpsfor Spark Connect - [SPARK-43895] [SC-132897][CONNECT][GO] Prepare the go package path
- [SPARK-43894] [SC-132892][PYTHON] Fix bug in df.cache()
- [SPARK-43509] [SC-131625][CONNECT] Support Creating multiple Spark Connect sessions
- [SPARK-43882] [SC-132888][SQL] Assign name to _LEGACY_ERROR_TEMP_2122
- [SPARK-43687] [SC-132747][SPARK-43688][SPARK-43689][SPARK-43690][PS] Fix
NumOpsfor Spark Connect - [SPARK-43604] [SC-132165][SQL] Refactor
INVALID_SQL_SYNTAXfor avoiding to embed error's text in source code - [SPARK-43859] [SC-132883][SQL] Override toString in LateralColumnAliasReference
- [SPARK-43792] [SC-132939][SQL][PYTHON][CONNECT] Add optional pattern for Catalog.listCatalogs
- [SPARK-43353] Revert "[SC-132734][PYTHON] Migrate remaining session errors into error class"
- [SPARK-43375] [SC-130309][CONNECT] Improve the error messages for INVALID_CONNECT_URL
- [SPARK-43863] [SC-132721][CONNECT] Remove redundant
toSeqfromSparkConnectPlannerfor Scala 2.13 - [SPARK-43676] [SC-132708][SPARK-43677][SPARK-43678][SPARK-43679][PS] Fix
DatetimeOpsfor Spark Connect - [SPARK-43666] [SC-132689][SPARK-43667][SPARK-43668][SPARK-43669][PS] Fix
BinaryOpsfor Spark Connect - [SPARK-43680] [SC-132709][SPARK-43681][SPARK-43682][SPARK-43683][PS] Fix
NullOpsfor Spark Connect - [SPARK-43782] [SC-132885][CORE] Support log level configuration with static Spark conf
- [SPARK-43132] [SC-131623] [SS] [CONNECT] Python Client DataStreamWriter foreach() API
- [SPARK-43290] [SC-131961][SQL] Adds AES IV and AAD support to ExpressionImplUtils
- [SPARK-43795] [SC-132532][CONNECT] Remove parameters not used for SparkConnectPlanner
- [SPARK-42317] [SC-129194][SQL] Assign name to _LEGACY_ERROR_TEMP_2247: CANNOT_MERGE_SCHEMAS
- [SPARK-43597] [SC-131960][SQL] Assign a name to the error class _LEGACY_ERROR_TEMP_0017
- [SPARK-43834] [SC-132679][SQL] Use error classes in the compilation errors of
ResolveDefaultColumns - [SPARK-43749] [SC-132423][SPARK-43750][SQL] Assign names to the error class _LEGACY_ERROR_TEMP_240[4-5]
- [SPARK-43576] [SC-132533][CORE] Remove unused declarations from Core module
- [SPARK-43314] [SC-129970][CONNECT][PYTHON] Migrate Spark Connect client errors into error class
- [SPARK-43799] [SC-132715][PYTHON] Add descriptor binary option to Pyspark Protobuf API
- [SPARK-43837] [SC-132678][SQL] Assign a name to the error class _LEGACY_ERROR_TEMP_103[1-2]
- [SPARK-43862] [SC-132750][SQL] Assign a name to the error class LEGACY_ERROR_TEMP(1254 & 1315)
- [SPARK-43886] [SC-132808][PYTHON] Accept generics tuple as typing hints of Pandas UDF
- [SPARK-43530] [SC-132653][PROTOBUF] Read descriptor file only once
- [SPARK-43820] [SC-132676][SPARK-43822][SPARK-43823][SPARK-43826][SPARK-43827] Assign names to the error class _LEGACY_ERROR_TEMP_241[1-7]
- [SPARK-43671] [SC-132519][SPARK-43672][SPARK-43673][SPARK-43674][PS] Fix
CategoricalOpsfor Spark Connect - [SPARK-39979] [SC-132685][SQL] Add option to use large variable width vectors for arrow UDF operations
- [SPARK-43692] [SC-132660][SPARK-43693][SPARK-43694][SPARK-43695][PS] Fix
StringOpsfor Spark Connect - [SPARK-43353] [SC-132734][PYTHON] Migrate remaining session errors into error class
- [SPARK-43815] [SC-132781][SQL] Add
to_varcharalias forto_char - [SPARK-43543] [13.x][SC-131839][PYTHON] Fix nested MapType behavior in Pandas UDF
- [SPARK-43361] [SC-131789][PROTOBUF] update documentation for errors related to enum serialization
- [SPARK-43740] [SC-132035][PYTHON][CONNECT] Hide unsupported
sessionmethods from auto-completion - [SPARK-43022] [SC-131681][CONNECT] Support protobuf functions for Scala client
- [SPARK-43304] [13.x][SC-129969][CONNECT][PYTHON] Migrate
NotImplementedErrorintoPySparkNotImplementedError - [SPARK-43860] [SC-132713][SQL] Enable tail-recursion wherever possible
- [SPARK-42421] [SC-132695][CORE] Use the utils to get the switch for dynamic allocation used in local checkpoint
- [SPARK-43590] [SC-132012][CONNECT] Make
connect-jvm-client-mima-checkto support mima check withprotobufmodule - [SPARK-43315] [13.x][CONNECT][PYTHON][SS] Migrate remaining errors from DataFrame(Reader|Writer) into error class
- [SPARK-43361] [SC-130980][PROTOBUF] spark-protobuf: allow serde with enum as ints
- [SPARK-43789] [SC-132397][R] Uses 'spark.sql.execution.arrow.maxRecordsPerBatch' in R createDataFrame with Arrow by default
- [SPARK-43596] [SC-126994][SQL] Handle IsNull predicate in rewriteDomainJoins
- [SPARK-42249] [SC-122565][SQL] Refining html link for documentation in error messages.
- [SPARK-43128] [SC-131628][CONNECT][SS] Make
recentProgressandlastProgressreturnStreamingQueryProgressconsistent with the native Scala Api - [SPARK-43387] [SC-130448][SQL] Provide a human readable error code for _LEGACY_ERROR_TEMP_1168.
- [SPARK-43334] [SC-132358] [UI] Fix error while serializing ExecutorPeakMetricsDistributions into API response
- [SPARK-42958] [SC-131475][CONNECT] Refactor
connect-jvm-client-mima-checkto support mima check with avro module - [SPARK-43591] [SC-131977][SQL] Assign a name to the error class _LEGACY_ERROR_TEMP_0013
- [SPARK-38464] [SC-132211][CORE] Use error classes in org.apache.spark.io
- [SPARK-43759] [SC-132189][SQL][PYTHON] Expose TimestampNTZType in pyspark.sql.types
- [SPARK-43649] [SC-132036][SPARK-43650][SPARK-43651][SQL] Assign names to the error class _LEGACY_ERROR_TEMP_240[1-3]
- [SPARK-43719] [SC-132016][WEBUI] Handle
missing row.excludedInStagesfield - [SPARK-42956] [SC-127257][CONNECT] Support avro functions for Scala client
- [SPARK-43487] [SC-131975][SQL] Fix Nested CTE error message
- [SPARK-43265] [SC-129653] Move Error framework to a common utils module
- [SPARK-40912] [SC-131547][CORE]Overhead of Exceptions in KryoDeserializationStream
- [SPARK-43583] [SC-131976][CORE] get MergedBlockedMetaReqHandler from the delegate instead of the SaslRpcHandler instance
- [SPARK-42996] [SC-131997][CONNECT][PS][ML] Create & assign proper JIRA tickets for all failing tests.
- [SPARK-38469] [SC-131425][CORE] Use error class in org.apache.spark.network
- [SPARK-43718] [SC-132014][SQL] Set nullable correctly for keys in USING joins
- [SPARK-43502] [SC-131419][PYTHON][CONNECT]
DataFrame.dropshould accept empty column - [SPARK-43743] [SC-132038][SQL] Port HIVE-12188(DoAs does not work properly in non-kerberos secured HS2)
- [SPARK-43598] [SC-131929][SQL] Assign a name to the error class _LEGACY_ERROR_TEMP_2400
- [SPARK-43138] [SC-131031][CORE] Fix ClassNotFoundException during migration
- [SPARK-43360] [SC-131526][SS][CONNECT] Scala client StreamingQueryManager
- [SPARK-38467] [SC-131025][CORE] Use error classes in org.apache.spark.memory
- [SPARK-43345] [SC-131790][SPARK-43346][SQL] Rename the error classes LEGACY_ERROR_TEMP[0041|1206]
- [SPARK-42815] [SC-126450][SQL] Subexpression elimination support shortcut expression
- [SPARK-38462] [SC-130449][CORE] Add error class INTERNAL_ERROR_EXECUTOR
- [SPARK-43589] [SC-131798][SQL] Fix
cannotBroadcastTableOverMaxTableBytesErrorto usebytesToString - [SPARK-43539] [SC-131830][SQL] Assign a name to the error class _LEGACY_ERROR_TEMP_0003
- [SPARK-43157] [SC-131688][SQL] Clone InMemoryRelation cached plan to prevent cloned plan from referencing same objects
- [SPARK-43309] [SC-129746][SPARK-38461][CORE] Extend INTERNAL_ERROR with categories and add error class INTERNAL_ERROR_BROADCAST
- [SPARK-43281] [SC-131370][SQL] Fix concurrent writer does not update file metrics
- [SPARK-43383] [SC-130660][SQL] Add
rowCountstatistics to LocalRelation - [SPARK-43574] [SC-131739][PYTHON][SQL] Support to set Python executable for UDF and pandas function APIs in workers during runtime
- [SPARK-43569] [SC-131694][SQL] Remove workaround for HADOOP-14067
- [SPARK-43183] [SC-128938][SS][13.x] Introduce a new callback "onQueryIdle" to StreamingQueryListener
- [SPARK-43528] [SC-131531][SQL][PYTHON] Support duplicated field names in createDataFrame with pandas DataFrame
- [SPARK-43548] [SC-131686][SS] Remove workaround for HADOOP-16255
- [SPARK-43494] [SC-131272][CORE] Directly call
replicate()forHdfsDataOutputStreamBuilderinstead of reflection inSparkHadoopUtil#createFile - [SPARK-43471] [SC-131063][CORE] Handle missing hadoopProperties and metricsProperties
- [SPARK-42899] [SC-126456][SQL] Fix DataFrame.to(schema) to handle the case where there is a non-nullable nested field in a nullable field
- [SPARK-43302] [SC-129747][SQL] Make Python UDAF an AggregateFunction
- [SPARK-43223] [SC-131270][Connect] Typed agg, reduce functions, RelationalGroupedDataset#as
- [SPARK-43482] [SC-131375][SS] Expand QueryTerminatedEvent to contain error class if it exists in exception
- [SPARK-43359] [SC-131463][SQL] Delete from Hive table should throw "UNSUPPORTED_FEATURE.TABLE_OPERATION"
- [SPARK-43284] [SC-130313] Switch back to url-encoded strings
- [SPARK-43492] [SC-131227][SQL] Add 3-args function aliases
DATE_ADDandDATE_DIFF - [SPARK-43473] [SC-131372][PYTHON] Support struct type in createDataFrame from pandas DataFrame
- [SPARK-43386] [SC-131110][SQL] Improve list of suggested column/attributes in
UNRESOLVED_COLUMN.WITH_SUGGESTIONerror class - [SPARK-43443] [SC-131332][SQL] Add benchmark for Timestamp type inference when use invalid value
- [SPARK-43518] [SC-131420][SQL] Convert
_LEGACY_ERROR_TEMP_2029to INTERNAL_ERROR - [SPARK-42604] [SC-130648][CONNECT] Implement functions.typedlit
- [SPARK-42323] [ES-669513] Revert "[SC-123479][SQL] Assign name to
_LEGACY_ERROR_TEMP_2332" - [SPARK-43357] [SC-130649][SQL] Filter date type quote date
- [SPARK-43410] [SC-130526][SQL] Improve vectorized loop for Packed skipValues
- [SPARK-43441] [SC-130995][CORE]
makeDotNodeshould not fail when DeterministicLevel is absent - [SPARK-43324] [SC-130455][SQL] Handle UPDATE commands for delta-based sources
- [SPARK-43272] [SC-131168][CORE] Directly call
createFileinstead of reflection - [SPARK-43296] [13.x][SC-130627][CONNECT][PYTHON] Migrate Spark Connect session errors into error class
- [SPARK-43405] [SC-130987][SQL] Remove useless
getRowFormatSQL,inputRowFormatSQL,outputRowFormatSQLmethod - [SPARK-43447] [SC-131017][R] Support R 4.3.0
- [SPARK-43425] [SC-130981][SQL] Add
TimestampNTZTypetoColumnarBatchRow - [SPARK-41532] [SC-130523][CONNECT][CLIENT] Add check for operations that involve multiple data frames
- [SPARK-43262] [SC-129270][CONNECT][SS][PYTHON] Migrate Spark Connect Structured Streaming errors into error class
- [SPARK-42992] [13.x][SC-129465][PYTHON] Introduce PySparkRuntimeError
- [SPARK-43422] [SC-130982][SQL] Keep tags on LogicalRelation with metadata output
- [SPARK-43390] [SC-130646][SQL] DSv2 allows CTAS/RTAS to reserve schema nullability
- [SPARK-43430] [SC-130989][CONNECT][PROTO] ExecutePlanRequest supports arbitrary request options.
- [SPARK-43313] [SC-130254][SQL] Adding missing column DEFAULT values for MERGE INSERT actions
- [SPARK-42842] [SC-130438][SQL] Merge the error class _LEGACY_ERROR_TEMP_2006 into REGEX_GROUP_INDEX
- [SPARK-43377] [SC-130405][SQL] Enable
spark.sql.thriftServer.interruptOnCancelby default - [SPARK-43292] [SC-130525][CORE][CONNECT] Move
ExecutorClassLoadertocoremodule and simplifyExecutor#addReplClassLoaderIfNeeded - [SPARK-43040] [SC-130310][SQL] Improve TimestampNTZ type support in JDBC data source
- [SPARK-43343] [SC-130629][SS] FileStreamSource should disable an extra file glob check when creating DataSource
- [SPARK-43312] [SC-130294][PROTOBUF] Option to convert Any fields into JSON
- [SPARK-43404] [SC-130586][SS] Skip reusing sst file for same version of RocksDB state store to avoid id mismatch error
- [SPARK-43397] [SC-130614][CORE] Log executor decommission duration in
executorLostmethod - [SPARK-42388] [SC-128568][SQL] Avoid parquet footer reads twice in vectorized reader
- [SPARK-43306] [13.x][SC-130320][PYTHON] Migrate ValueError from Spark SQL types into error class
- [SPARK-43337] [SC-130353][UI][3.4] Asc/desc arrow icons for sorting column does not get displayed in the table column
- [SPARK-43279] [SC-130351][CORE] Cleanup unused members from
SparkHadoopUtil - [SPARK-43250] [SC-130147][SQL] Replace the error class
_LEGACY_ERROR_TEMP_2014with an internal error - [SPARK-43363] [SC-130299][SQL][PYTHON] Make to call
astypeto the category type only when the arrow type is not provided - [SPARK-43261] [SC-130154][PYTHON] Migrate
TypeErrorfrom Spark SQL types into error class - [SPARK-41971] [SC-130230][SQL][PYTHON] Add a config for pandas conversion how to handle struct types
- [SPARK-43381] [SC-130464][CONNECT] Make 'transformStatCov' lazy
- [SPARK-43362] [SC-130481][SQL] Special handling of JSON type for MySQL connector
- [SPARK-42843] [SC-130196][SQL] Update the error class _LEGACY_ERROR_TEMP_2007 to REGEX_GROUP_INDEX
- [SPARK-42585] [SC-129968][CONNECT] Streaming of local relations
- [SPARK-43181] [SC-130476][SQL] Show UI WebURL in
spark-sqlshell - [SPARK-43051] [SC-130194][CONNECT] Add option to emit default values
- [SPARK-43204] [SC-129296][SQL] Align MERGE assignments with table attributes
- [SPARK-43348] [SC-130082][PYTHON] Support
Python 3.8in PyPy3 - [SPARK-42845] [SC-130078][SQL] Update the error class _LEGACY_ERROR_TEMP_2010 to InternalError
- [SPARK-43211] [SC-128978][HIVE] Remove Hadoop2 support in IsolatedClientLoader
System environment
- Operating System: Ubuntu 22.04.2 LTS
- Java: Zulu 8.70.0.23-CA-linux64
- Scala: 2.12.15
- Python: 3.10.12
- R: 4.2.2
- Delta Lake: 2.4.0
Installed Python libraries
| Library | Version | Library | Version | Library | Version |
|---|---|---|---|---|---|
| appdirs | 1.4.4 | argon2-cffi | 21.3.0 | argon2-cffi-bindings | 21.2.0 |
| asttokens | 2.2.1 | attrs | 21.4.0 | backcall | 0.2.0 |
| beautifulsoup4 | 4.11.1 | black | 22.6.0 | bleach | 4.1.0 |
| blinker | 1.4 | boto3 | 1.24.28 | botocore | 1.27.28 |
| certifi | 2022.9.14 | cffi | 1.15.1 | chardet | 4.0.0 |
| charset-normalizer | 2.0.4 | click | 8.0.4 | cryptography | 37.0.1 |
| cycler | 0.11.0 | Cython | 0.29.32 | databricks-sdk | 0.1.6 |
| dbus-python | 1.2.18 | debugpy | 1.5.1 | decorator | 5.1.1 |
| defusedxml | 0.7.1 | distlib | 0.3.6 | docstring-to-markdown | 0.12 |
| entrypoints | 0.4 | executing | 1.2.0 | facets-overview | 1.0.3 |
| fastjsonschema | 2.16.3 | filelock | 3.12.0 | fonttools | 4.25.0 |
| googleapis-common-protos | 1.56.4 | grpcio | 1.48.1 | grpcio-status | 1.48.1 |
| httplib2 | 0.20.2 | idna | 3.3 | importlib-metadata | 4.6.4 |
| ipykernel | 6.17.1 | ipython | 8.10.0 | ipython-genutils | 0.2.0 |
| ipywidgets | 7.7.2 | jedi | 0.18.1 | jeepney | 0.7.1 |
| Jinja2 | 2.11.3 | jmespath | 0.10.0 | joblib | 1.2.0 |
| jsonschema | 4.16.0 | jupyter-client | 7.3.4 | jupyter_core | 4.11.2 |
| jupyterlab-pygments | 0.1.2 | jupyterlab-widgets | 1.0.0 | keyring | 23.5.0 |
| kiwisolver | 1.4.2 | launchpadlib | 1.10.16 | lazr.restfulclient | 0.14.4 |
| lazr.uri | 1.0.6 | MarkupSafe | 2.0.1 | matplotlib | 3.5.2 |
| matplotlib-inline | 0.1.6 | mccabe | 0.7.0 | mistune | 0.8.4 |
| more-itertools | 8.10.0 | mypy-extensions | 0.4.3 | nbclient | 0.5.13 |
| nbconvert | 6.4.4 | nbformat | 5.5.0 | nest-asyncio | 1.5.5 |
| nodeenv | 1.8.0 | notebook | 6.4.12 | numpy | 1.21.5 |
| oauthlib | 3.2.0 | packaging | 21.3 | pandas | 1.4.4 |
| pandocfilters | 1.5.0 | parso | 0.8.3 | pathspec | 0.9.0 |
| patsy | 0.5.2 | pexpect | 4.8.0 | pickleshare | 0.7.5 |
| Pillow | 9.2.0 | pip | 22.2.2 | platformdirs | 2.5.2 |
| plotly | 5.9.0 | pluggy | 1.0.0 | prometheus-client | 0.14.1 |
| prompt-toolkit | 3.0.36 | protobuf | 3.19.4 | psutil | 5.9.0 |
| psycopg2 | 2.9.3 | ptyprocess | 0.7.0 | pure-eval | 0.2.2 |
| pyarrow | 8.0.0 | pycparser | 2.21 | pydantic | 1.10.6 |
| pyflakes | 3.0.1 | Pygments | 2.11.2 | PyGObject | 3.42.1 |
| PyJWT | 2.3.0 | pyodbc | 4.0.32 | pyparsing | 3.0.9 |
| pyright | 1.1.294 | pyrsistent | 0.18.0 | python-dateutil | 2.8.2 |
| python-lsp-jsonrpc | 1.0.0 | python-lsp-server | 1.7.1 | pytoolconfig | 1.2.2 |
| pytz | 2022.1 | pyzmq | 23.2.0 | requests | 2.28.1 |
| rope | 1.7.0 | s3transfer | 0.6.0 | scikit-learn | 1.1.1 |
| scipy | 1.9.1 | seaborn | 0.11.2 | SecretStorage | 3.3.1 |
| Send2Trash | 1.8.0 | setuptools | 63.4.1 | six | 1.16.0 |
| soupsieve | 2.3.1 | ssh-import-id | 5.11 | stack-data | 0.6.2 |
| statsmodels | 0.13.2 | tenacity | 8.1.0 | terminado | 0.13.1 |
| testpath | 0.6.0 | threadpoolctl | 2.2.0 | tokenize-rt | 4.2.1 |
| tomli | 2.0.1 | tornado | 6.1 | traitlets | 5.1.1 |
| typing_extensions | 4.3.0 | ujson | 5.4.0 | unattended-upgrades | 0.1 |
| urllib3 | 1.26.11 | virtualenv | 20.16.3 | wadllib | 1.3.6 |
| wcwidth | 0.2.5 | webencodings | 0.5.1 | whatthepatch | 1.0.2 |
| wheel | 0.37.1 | widgetsnbextension | 3.6.1 | yapf | 0.31.0 |
| zipp | 1.0.0 |
Installed R libraries
R libraries are installed from the Microsoft CRAN snapshot on 2023-02-10 .
| Library | Version | Library | Version | Library | Version |
|---|---|---|---|---|---|
| arrow | 10.0.1 | askpass | 1.1 | assertthat | 0.2.1 |
| backports | 1.4.1 | base | 4.2.2 | base64enc | 0.1-3 |
| bit | 4.0.5 | bit64 | 4.0.5 | blob | 1.2.3 |
| boot | 1.3-28 | brew | 1.0-8 | brio | 1.1.3 |
| broom | 1.0.3 | bslib | 0.4.2 | cachem | 1.0.6 |
| callr | 3.7.3 | caret | 6.0-93 | cellranger | 1.1.0 |
| chron | 2.3-59 | class | 7.3-21 | cli | 3.6.0 |
| clipr | 0.8.0 | clock | 0.6.1 | cluster | 2.1.4 |
| codetools | 0.2-19 | colorspace | 2.1-0 | commonmark | 1.8.1 |
| compiler | 4.2.2 | config | 0.3.1 | cpp11 | 0.4.3 |
| crayon | 1.5.2 | credentials | 1.3.2 | curl | 5.0.0 |
| data.table | 1.14.6 | datasets | 4.2.2 | DBI | 1.1.3 |
| dbplyr | 2.3.0 | desc | 1.4.2 | devtools | 2.4.5 |
| diffobj | 0.3.5 | digest | 0.6.31 | downlit | 0.4.2 |
| dplyr | 1.1.0 | dtplyr | 1.2.2 | e1071 | 1.7-13 |
| ellipsis | 0.3.2 | evaluate | 0.20 | fansi | 1.0.4 |
| farver | 2.1.1 | fastmap | 1.1.0 | fontawesome | 0.5.0 |
| forcats | 1.0.0 | foreach | 1.5.2 | foreign | 0.8-82 |
| forge | 0.2.0 | fs | 1.6.1 | future | 1.31.0 |
| future.apply | 1.10.0 | gargle | 1.3.0 | generics | 0.1.3 |
| gert | 1.9.2 | ggplot2 | 3.4.0 | gh | 1.3.1 |
| gitcreds | 0.1.2 | glmnet | 4.1-6 | globals | 0.16.2 |
| glue | 1.6.2 | googledrive | 2.0.0 | googlesheets4 | 1.0.1 |
| gower | 1.0.1 | graphics | 4.2.2 | grDevices | 4.2.2 |
| grid | 4.2.2 | gridExtra | 2.3 | gsubfn | 0.7 |
| gtable | 0.3.1 | hardhat | 1.2.0 | haven | 2.5.1 |
| highr | 0.10 | hms | 1.1.2 | htmltools | 0.5.4 |
| htmlwidgets | 1.6.1 | httpuv | 1.6.8 | httr | 1.4.4 |
| ids | 1.0.1 | ini | 0.3.1 | ipred | 0.9-13 |
| isoband | 0.2.7 | iterators | 1.0.14 | jquerylib | 0.1.4 |
| jsonlite | 1.8.4 | KernSmooth | 2.23-20 | knitr | 1.42 |
| labeling | 0.4.2 | later | 1.3.0 | lattice | 0.20-45 |
| lava | 1.7.1 | lifecycle | 1.0.3 | listenv | 0.9.0 |
| lubridate | 1.9.1 | magrittr | 2.0.3 | markdown | 1.5 |
| MASS | 7.3-58.2 | Matrix | 1.5-1 | memoise | 2.0.1 |
| methods | 4.2.2 | mgcv | 1.8-41 | mime | 0.12 |
| miniUI | 0.1.1.1 | ModelMetrics | 1.2.2.2 | modelr | 0.1.10 |
| munsell | 0.5.0 | nlme | 3.1-162 | nnet | 7.3-18 |
| numDeriv | 2016.8-1.1 | openssl | 2.0.5 | parallel | 4.2.2 |
| parallelly | 1.34.0 | pillar | 1.8.1 | pkgbuild | 1.4.0 |
| pkgconfig | 2.0.3 | pkgdown | 2.0.7 | pkgload | 1.3.2 |
| plogr | 0.2.0 | plyr | 1.8.8 | praise | 1.0.0 |
| prettyunits | 1.1.1 | pROC | 1.18.0 | processx | 3.8.0 |
| prodlim | 2019.11.13 | profvis | 0.3.7 | progress | 1.2.2 |
| progressr | 0.13.0 | promises | 1.2.0.1 | proto | 1.0.0 |
| proxy | 0.4-27 | ps | 1.7.2 | purrr | 1.0.1 |
| r2d3 | 0.2.6 | R6 | 2.5.1 | ragg | 1.2.5 |
| randomForest | 4.7-1.1 | rappdirs | 0.3.3 | rcmdcheck | 1.4.0 |
| RColorBrewer | 1.1-3 | Rcpp | 1.0.10 | RcppEigen | 0.3.3.9.3 |
| readr | 2.1.3 | readxl | 1.4.2 | recipes | 1.0.4 |
| rematch | 1.0.1 | rematch2 | 2.1.2 | remotes | 2.4.2 |
| reprex | 2.0.2 | reshape2 | 1.4.4 | rlang | 1.0.6 |
| rmarkdown | 2.20 | RODBC | 1.3-20 | roxygen2 | 7.2.3 |
| rpart | 4.1.19 | rprojroot | 2.0.3 | Rserve | 1.8-12 |
| RSQLite | 2.2.20 | rstudioapi | 0.14 | rversions | 2.1.2 |
| rvest | 1.0.3 | sass | 0.4.5 | scales | 1.2.1 |
| selectr | 0.4-2 | sessioninfo | 1.2.2 | shape | 1.4.6 |
| shiny | 1.7.4 | sourcetools | 0.1.7-1 | sparklyr | 1.7.9 |
| SparkR | 3.4.0 | spatial | 7.3-15 | splines | 4.2.2 |
| sqldf | 0.4-11 | SQUAREM | 2021.1 | stats | 4.2.2 |
| stats4 | 4.2.2 | stringi | 1.7.12 | stringr | 1.5.0 |
| survival | 3.5-3 | sys | 3.4.1 | systemfonts | 1.0.4 |
| tcltk | 4.2.2 | testthat | 3.1.6 | textshaping | 0.3.6 |
| tibble | 3.1.8 | tidyr | 1.3.0 | tidyselect | 1.2.0 |
| tidyverse | 1.3.2 | timechange | 0.2.0 | timeDate | 4022.108 |
| tinytex | 0.44 | tools | 4.2.2 | tzdb | 0.3.0 |
| urlchecker | 1.0.1 | usethis | 2.1.6 | utf8 | 1.2.3 |
| utils | 4.2.2 | uuid | 1.1-0 | vctrs | 0.5.2 |
| viridisLite | 0.4.1 | vroom | 1.6.1 | waldo | 0.4.0 |
| whisker | 0.4.1 | withr | 2.5.0 | xfun | 0.37 |
| xml2 | 1.3.3 | xopen | 1.0.0 | xtable | 1.8-4 |
| yaml | 2.3.7 | zip | 2.2.2 |
Installed Java and Scala libraries (Scala 2.12 cluster version)
| Group ID | Artifact ID | Version |
|---|---|---|
| antlr | antlr | 2.7.7 |
| com.amazonaws | amazon-kinesis-client | 1.12.0 |
| com.amazonaws | aws-java-sdk-autoscaling | 1.12.390 |
| com.amazonaws | aws-java-sdk-cloudformation | 1.12.390 |
| com.amazonaws | aws-java-sdk-cloudfront | 1.12.390 |
| com.amazonaws | aws-java-sdk-cloudhsm | 1.12.390 |
| com.amazonaws | aws-java-sdk-cloudsearch | 1.12.390 |
| com.amazonaws | aws-java-sdk-cloudtrail | 1.12.390 |
| com.amazonaws | aws-java-sdk-cloudwatch | 1.12.390 |
| com.amazonaws | aws-java-sdk-cloudwatchmetrics | 1.12.390 |
| com.amazonaws | aws-java-sdk-codedeploy | 1.12.390 |
| com.amazonaws | aws-java-sdk-cognitoidentity | 1.12.390 |
| com.amazonaws | aws-java-sdk-cognitosync | 1.12.390 |
| com.amazonaws | aws-java-sdk-config | 1.12.390 |
| com.amazonaws | aws-java-sdk-core | 1.12.390 |
| com.amazonaws | aws-java-sdk-datapipeline | 1.12.390 |
| com.amazonaws | aws-java-sdk-directconnect | 1.12.390 |
| com.amazonaws | aws-java-sdk-directory | 1.12.390 |
| com.amazonaws | aws-java-sdk-dynamodb | 1.12.390 |
| com.amazonaws | aws-java-sdk-ec2 | 1.12.390 |
| com.amazonaws | aws-java-sdk-ecs | 1.12.390 |
| com.amazonaws | aws-java-sdk-efs | 1.12.390 |
| com.amazonaws | aws-java-sdk-elasticache | 1.12.390 |
| com.amazonaws | aws-java-sdk-elasticbeanstalk | 1.12.390 |
| com.amazonaws | aws-java-sdk-elasticloadbalancing | 1.12.390 |
| com.amazonaws | aws-java-sdk-elastictranscoder | 1.12.390 |
| com.amazonaws | aws-java-sdk-emr | 1.12.390 |
| com.amazonaws | aws-java-sdk-glacier | 1.12.390 |
| com.amazonaws | aws-java-sdk-glue | 1.12.390 |
| com.amazonaws | aws-java-sdk-iam | 1.12.390 |
| com.amazonaws | aws-java-sdk-importexport | 1.12.390 |
| com.amazonaws | aws-java-sdk-kinesis | 1.12.390 |
| com.amazonaws | aws-java-sdk-kms | 1.12.390 |
| com.amazonaws | aws-java-sdk-lambda | 1.12.390 |
| com.amazonaws | aws-java-sdk-logs | 1.12.390 |
| com.amazonaws | aws-java-sdk-machinelearning | 1.12.390 |
| com.amazonaws | aws-java-sdk-opsworks | 1.12.390 |
| com.amazonaws | aws-java-sdk-rds | 1.12.390 |
| com.amazonaws | aws-java-sdk-redshift | 1.12.390 |
| com.amazonaws | aws-java-sdk-route53 | 1.12.390 |
| com.amazonaws | aws-java-sdk-s3 | 1.12.390 |
| com.amazonaws | aws-java-sdk-ses | 1.12.390 |
| com.amazonaws | aws-java-sdk-simpledb | 1.12.390 |
| com.amazonaws | aws-java-sdk-simpleworkflow | 1.12.390 |
| com.amazonaws | aws-java-sdk-sns | 1.12.390 |
| com.amazonaws | aws-java-sdk-sqs | 1.12.390 |
| com.amazonaws | aws-java-sdk-ssm | 1.12.390 |
| com.amazonaws | aws-java-sdk-storagegateway | 1.12.390 |
| com.amazonaws | aws-java-sdk-sts | 1.12.390 |
| com.amazonaws | aws-java-sdk-support | 1.12.390 |
| com.amazonaws | aws-java-sdk-swf-libraries | 1.11.22 |
| com.amazonaws | aws-java-sdk-workspaces | 1.12.390 |
| com.amazonaws | jmespath-java | 1.12.390 |
| com.clearspring.analytics | stream | 2.9.6 |
| com.databricks | Rserve | 1.8-3 |
| com.databricks | jets3t | 0.7.1-0 |
| com.databricks.scalapb | compilerplugin_2.12 | 0.4.15-10 |
| com.databricks.scalapb | scalapb-runtime_2.12 | 0.4.15-10 |
| com.esotericsoftware | kryo-shaded | 4.0.2 |
| com.esotericsoftware | minlog | 1.3.0 |
| com.fasterxml | classmate | 1.3.4 |
| com.fasterxml.jackson.core | jackson-annotations | 2.14.2 |
| com.fasterxml.jackson.core | jackson-core | 2.14.2 |
| com.fasterxml.jackson.core | jackson-databind | 2.14.2 |
| com.fasterxml.jackson.dataformat | jackson-dataformat-cbor | 2.14.2 |
| com.fasterxml.jackson.datatype | jackson-datatype-joda | 2.14.2 |
| com.fasterxml.jackson.datatype | jackson-datatype-jsr310 | 2.13.4 |
| com.fasterxml.jackson.module | jackson-module-paranamer | 2.14.2 |
| com.fasterxml.jackson.module | jackson-module-scala_2.12 | 2.14.2 |
| com.github.ben-manes.caffeine | caffeine | 2.9.3 |
| com.github.fommil | jniloader | 1.1 |
| com.github.fommil.netlib | native_ref-java | 1.1 |
| com.github.fommil.netlib | native_ref-java | 1.1-natives |
| com.github.fommil.netlib | native_system-java | 1.1 |
| com.github.fommil.netlib | native_system-java | 1.1-natives |
| com.github.fommil.netlib | netlib-native_ref-linux-x86_64 | 1.1-natives |
| com.github.fommil.netlib | netlib-native_system-linux-x86_64 | 1.1-natives |
| com.github.luben | zstd-jni | 1.5.2-5 |
| com.github.wendykierp | JTransforms | 3.1 |
| com.google.code.findbugs | jsr305 | 3.0.0 |
| com.google.code.gson | gson | 2.8.9 |
| com.google.crypto.tink | tink | 1.7.0 |
| com.google.errorprone | error_prone_annotations | 2.10.0 |
| com.google.flatbuffers | flatbuffers-java | 1.12.0 |
| com.google.guava | guava | 15.0 |
| com.google.protobuf | protobuf-java | 2.6.1 |
| com.h2database | h2 | 2.1.214 |
| com.helger | profiler | 1.1.1 |
| com.jcraft | jsch | 0.1.55 |
| com.jolbox | bonecp | 0.8.0.RELEASE |
| com.lihaoyi | sourcecode_2.12 | 0.1.9 |
| com.microsoft.azure | azure-data-lake-store-sdk | 2.3.9 |
| com.microsoft.sqlserver | mssql-jdbc | 11.2.2.jre8 |
| com.ning | compress-lzf | 1.1.2 |
| com.sun.mail | javax.mail | 1.5.2 |
| com.sun.xml.bind | jaxb-core | 2.2.11 |
| com.sun.xml.bind | jaxb-impl | 2.2.11 |
| com.tdunning | json | 1.8 |
| com.thoughtworks.paranamer | paranamer | 2.8 |
| com.trueaccord.lenses | lenses_2.12 | 0.4.12 |
| com.twitter | chill-java | 0.10.0 |
| com.twitter | chill_2.12 | 0.10.0 |
| com.twitter | util-app_2.12 | 7.1.0 |
| com.twitter | util-core_2.12 | 7.1.0 |
| com.twitter | util-function_2.12 | 7.1.0 |
| com.twitter | util-jvm_2.12 | 7.1.0 |
| com.twitter | util-lint_2.12 | 7.1.0 |
| com.twitter | util-registry_2.12 | 7.1.0 |
| com.twitter | util-stats_2.12 | 7.1.0 |
| com.typesafe | config | 1.2.1 |
| com.typesafe.scala-logging | scala-logging_2.12 | 3.7.2 |
| com.uber | h3 | 3.7.0 |
| com.univocity | univocity-parsers | 2.9.1 |
| com.zaxxer | HikariCP | 4.0.3 |
| commons-cli | commons-cli | 1.5.0 |
| commons-codec | commons-codec | 1.15 |
| commons-collections | commons-collections | 3.2.2 |
| commons-dbcp | commons-dbcp | 1.4 |
| commons-fileupload | commons-fileupload | 1.5 |
| commons-httpclient | commons-httpclient | 3.1 |
| commons-io | commons-io | 2.11.0 |
| commons-lang | commons-lang | 2.6 |
| commons-logging | commons-logging | 1.1.3 |
| commons-pool | commons-pool | 1.5.4 |
| dev.ludovic.netlib | arpack | 3.0.3 |
| dev.ludovic.netlib | blas | 3.0.3 |
| dev.ludovic.netlib | lapack | 3.0.3 |
| info.ganglia.gmetric4j | gmetric4j | 1.0.10 |
| io.airlift | aircompressor | 0.21 |
| io.delta | delta-sharing-spark_2.12 | 0.7.0 |
| io.dropwizard.metrics | metrics-core | 4.2.10 |
| io.dropwizard.metrics | metrics-graphite | 4.2.10 |
| io.dropwizard.metrics | metrics-healthchecks | 4.2.10 |
| io.dropwizard.metrics | metrics-jetty9 | 4.2.10 |
| io.dropwizard.metrics | metrics-jmx | 4.2.10 |
| io.dropwizard.metrics | metrics-json | 4.2.10 |
| io.dropwizard.metrics | metrics-jvm | 4.2.10 |
| io.dropwizard.metrics | metrics-servlets | 4.2.10 |
| io.netty | netty-all | 4.1.87.Final |
| io.netty | netty-buffer | 4.1.87.Final |
| io.netty | netty-codec | 4.1.87.Final |
| io.netty | netty-codec-http | 4.1.87.Final |
| io.netty | netty-codec-http2 | 4.1.87.Final |
| io.netty | netty-codec-socks | 4.1.87.Final |
| io.netty | netty-common | 4.1.87.Final |
| io.netty | netty-handler | 4.1.87.Final |
| io.netty | netty-handler-proxy | 4.1.87.Final |
| io.netty | netty-resolver | 4.1.87.Final |
| io.netty | netty-transport | 4.1.87.Final |
| io.netty | netty-transport-classes-epoll | 4.1.87.Final |
| io.netty | netty-transport-classes-kqueue | 4.1.87.Final |
| io.netty | netty-transport-native-epoll | 4.1.87.Final |
| io.netty | netty-transport-native-epoll | 4.1.87.Final-linux-aarch_64 |
| io.netty | netty-transport-native-epoll | 4.1.87.Final-linux-x86_64 |
| io.netty | netty-transport-native-kqueue | 4.1.87.Final-osx-aarch_64 |
| io.netty | netty-transport-native-kqueue | 4.1.87.Final-osx-x86_64 |
| io.netty | netty-transport-native-unix-common | 4.1.87.Final |
| io.prometheus | simpleclient | 0.7.0 |
| io.prometheus | simpleclient_common | 0.7.0 |
| io.prometheus | simpleclient_dropwizard | 0.7.0 |
| io.prometheus | simpleclient_pushgateway | 0.7.0 |
| io.prometheus | simpleclient_servlet | 0.7.0 |
| io.prometheus.jmx | collector | 0.12.0 |
| jakarta.annotation | jakarta.annotation-api | 1.3.5 |
| jakarta.servlet | jakarta.servlet-api | 4.0.3 |
| jakarta.validation | jakarta.validation-api | 2.0.2 |
| jakarta.ws.rs | jakarta.ws.rs-api | 2.1.6 |
| javax.activation | activation | 1.1.1 |
| javax.el | javax.el-api | 2.2.4 |
| javax.jdo | jdo-api | 3.0.1 |
| javax.transaction | jta | 1.1 |
| javax.transaction | transaction-api | 1.1 |
| javax.xml.bind | jaxb-api | 2.2.11 |
| javolution | javolution | 5.5.1 |
| jline | jline | 2.14.6 |
| joda-time | joda-time | 2.12.1 |
| ml.combust.mleap | mleap-databricks-runtime_2.12 | v0.20.0-db2 |
| net.java.dev.jna | jna | 5.8.0 |
| net.razorvine | pickle | 1.3 |
| net.sf.jpam | jpam | 1.1 |
| net.sf.opencsv | opencsv | 2.3 |
| net.sf.supercsv | super-csv | 2.2.0 |
| net.snowflake | snowflake-ingest-sdk | 0.9.6 |
| net.snowflake | snowflake-jdbc | 3.13.29 |
| net.sourceforge.f2j | arpack_combined_all | 0.1 |
| org.acplt.remotetea | remotetea-oncrpc | 1.1.2 |
| org.antlr | ST4 | 4.0.4 |
| org.antlr | antlr-runtime | 3.5.2 |
| org.antlr | antlr4-runtime | 4.9.3 |
| org.antlr | stringtemplate | 3.2.1 |
| org.apache.ant | ant | 1.9.16 |
| org.apache.ant | ant-jsch | 1.9.16 |
| org.apache.ant | ant-launcher | 1.9.16 |
| org.apache.arrow | arrow-format | 11.0.0 |
| org.apache.arrow | arrow-memory-core | 11.0.0 |
| org.apache.arrow | arrow-memory-netty | 11.0.0 |
| org.apache.arrow | arrow-vector | 11.0.0 |
| org.apache.avro | avro | 1.11.1 |
| org.apache.avro | avro-ipc | 1.11.1 |
| org.apache.avro | avro-mapred | 1.11.1 |
| org.apache.commons | commons-collections4 | 4.4 |
| org.apache.commons | commons-compress | 1.21 |
| org.apache.commons | commons-crypto | 1.1.0 |
| org.apache.commons | commons-lang3 | 3.12.0 |
| org.apache.commons | commons-math3 | 3.6.1 |
| org.apache.commons | commons-text | 1.10.0 |
| org.apache.curator | curator-client | 2.13.0 |
| org.apache.curator | curator-framework | 2.13.0 |
| org.apache.curator | curator-recipes | 2.13.0 |
| org.apache.datasketches | datasketches-java | 3.1.0 |
| org.apache.datasketches | datasketches-memory | 2.0.0 |
| org.apache.derby | derby | 10.14.2.0 |
| org.apache.hadoop | hadoop-client-runtime | 3.3.4 |
| org.apache.hive | hive-beeline | 2.3.9 |
| org.apache.hive | hive-cli | 2.3.9 |
| org.apache.hive | hive-jdbc | 2.3.9 |
| org.apache.hive | hive-llap-client | 2.3.9 |
| org.apache.hive | hive-llap-common | 2.3.9 |
| org.apache.hive | hive-serde | 2.3.9 |
| org.apache.hive | hive-shims | 2.3.9 |
| org.apache.hive | hive-storage-api | 2.8.1 |
| org.apache.hive.shims | hive-shims-0.23 | 2.3.9 |
| org.apache.hive.shims | hive-shims-common | 2.3.9 |
| org.apache.hive.shims | hive-shims-scheduler | 2.3.9 |
| org.apache.httpcomponents | httpclient | 4.5.14 |
| org.apache.httpcomponents | httpcore | 4.4.16 |
| org.apache.ivy | ivy | 2.5.1 |
| org.apache.logging.log4j | log4j-1.2-api | 2.19.0 |
| org.apache.logging.log4j | log4j-api | 2.19.0 |
| org.apache.logging.log4j | log4j-core | 2.19.0 |
| org.apache.logging.log4j | log4j-slf4j2-impl | 2.19.0 |
| org.apache.mesos | mesos | 1.11.0-shaded-protobuf |
| org.apache.orc | orc-core | 1.8.3-shaded-protobuf |
| org.apache.orc | orc-mapreduce | 1.8.3-shaded-protobuf |
| org.apache.orc | orc-shims | 1.8.3 |
| org.apache.thrift | libfb303 | 0.9.3 |
| org.apache.thrift | libthrift | 0.12.0 |
| org.apache.xbean | xbean-asm9-shaded | 4.22 |
| org.apache.yetus | audience-annotations | 0.13.0 |
| org.apache.zookeeper | zookeeper | 3.6.3 |
| org.apache.zookeeper | zookeeper-jute | 3.6.3 |
| org.checkerframework | checker-qual | 3.19.0 |
| org.codehaus.jackson | jackson-core-asl | 1.9.13 |
| org.codehaus.jackson | jackson-mapper-asl | 1.9.13 |
| org.codehaus.janino | commons-compiler | 3.0.16 |
| org.codehaus.janino | janino | 3.0.16 |
| org.datanucleus | datanucleus-api-jdo | 4.2.4 |
| org.datanucleus | datanucleus-core | 4.1.17 |
| org.datanucleus | datanucleus-rdbms | 4.1.19 |
| org.datanucleus | javax.jdo | 3.2.0-m3 |
| org.eclipse.jetty | jetty-client | 9.4.50.v20221201 |
| org.eclipse.jetty | jetty-continuation | 9.4.50.v20221201 |
| org.eclipse.jetty | jetty-http | 9.4.50.v20221201 |
| org.eclipse.jetty | jetty-io | 9.4.50.v20221201 |
| org.eclipse.jetty | jetty-jndi | 9.4.50.v20221201 |
| org.eclipse.jetty | jetty-plus | 9.4.50.v20221201 |
| org.eclipse.jetty | jetty-proxy | 9.4.50.v20221201 |
| org.eclipse.jetty | jetty-security | 9.4.50.v20221201 |
| org.eclipse.jetty | jetty-server | 9.4.50.v20221201 |
| org.eclipse.jetty | jetty-servlet | 9.4.50.v20221201 |
| org.eclipse.jetty | jetty-servlets | 9.4.50.v20221201 |
| org.eclipse.jetty | jetty-util | 9.4.50.v20221201 |
| org.eclipse.jetty | jetty-util-ajax | 9.4.50.v20221201 |
| org.eclipse.jetty | jetty-webapp | 9.4.50.v20221201 |
| org.eclipse.jetty | jetty-xml | 9.4.50.v20221201 |
| org.eclipse.jetty.websocket | websocket-api | 9.4.50.v20221201 |
| org.eclipse.jetty.websocket | websocket-client | 9.4.50.v20221201 |
| org.eclipse.jetty.websocket | websocket-common | 9.4.50.v20221201 |
| org.eclipse.jetty.websocket | websocket-server | 9.4.50.v20221201 |
| org.eclipse.jetty.websocket | websocket-servlet | 9.4.50.v20221201 |
| org.fusesource.leveldbjni | leveldbjni-all | 1.8 |
| org.glassfish.hk2 | hk2-api | 2.6.1 |
| org.glassfish.hk2 | hk2-locator | 2.6.1 |
| org.glassfish.hk2 | hk2-utils | 2.6.1 |
| org.glassfish.hk2 | osgi-resource-locator | 1.0.3 |
| org.glassfish.hk2.external | aopalliance-repackaged | 2.6.1 |
| org.glassfish.hk2.external | jakarta.inject | 2.6.1 |
| org.glassfish.jersey.containers | jersey-container-servlet | 2.36 |
| org.glassfish.jersey.containers | jersey-container-servlet-core | 2.36 |
| org.glassfish.jersey.core | jersey-client | 2.36 |
| org.glassfish.jersey.core | jersey-common | 2.36 |
| org.glassfish.jersey.core | jersey-server | 2.36 |
| org.glassfish.jersey.inject | jersey-hk2 | 2.36 |
| org.hibernate.validator | hibernate-validator | 6.1.7.Final |
| org.javassist | javassist | 3.25.0-GA |
| org.jboss.logging | jboss-logging | 3.3.2.Final |
| org.jdbi | jdbi | 2.63.1 |
| org.jetbrains | annotations | 17.0.0 |
| org.joda | joda-convert | 1.7 |
| org.jodd | jodd-core | 3.5.2 |
| org.json4s | json4s-ast_2.12 | 3.7.0-M11 |
| org.json4s | json4s-core_2.12 | 3.7.0-M11 |
| org.json4s | json4s-jackson_2.12 | 3.7.0-M11 |
| org.json4s | json4s-scalap_2.12 | 3.7.0-M11 |
| org.lz4 | lz4-java | 1.8.0 |
| org.mariadb.jdbc | mariadb-java-client | 2.7.4 |
| org.mlflow | mlflow-spark | 2.2.0 |
| org.objenesis | objenesis | 2.5.1 |
| org.postgresql | postgresql | 42.3.8 |
| org.roaringbitmap | RoaringBitmap | 0.9.39 |
| org.roaringbitmap | shims | 0.9.39 |
| org.rocksdb | rocksdbjni | 7.8.3 |
| org.rosuda.REngine | REngine | 2.1.0 |
| org.scala-lang | scala-compiler_2.12 | 2.12.15 |
| org.scala-lang | scala-library_2.12 | 2.12.15 |
| org.scala-lang | scala-reflect_2.12 | 2.12.15 |
| org.scala-lang.modules | scala-collection-compat_2.12 | 2.4.3 |
| org.scala-lang.modules | scala-parser-combinators_2.12 | 1.1.2 |
| org.scala-lang.modules | scala-xml_2.12 | 1.2.0 |
| org.scala-sbt | test-interface | 1.0 |
| org.scalacheck | scalacheck_2.12 | 1.14.2 |
| org.scalactic | scalactic_2.12 | 3.2.15 |
| org.scalanlp | breeze-macros_2.12 | 2.1.0 |
| org.scalanlp | breeze_2.12 | 2.1.0 |
| org.scalatest | scalatest-compatible | 3.2.15 |
| org.scalatest | scalatest-core_2.12 | 3.2.15 |
| org.scalatest | scalatest-diagrams_2.12 | 3.2.15 |
| org.scalatest | scalatest-featurespec_2.12 | 3.2.15 |
| org.scalatest | scalatest-flatspec_2.12 | 3.2.15 |
| org.scalatest | scalatest-freespec_2.12 | 3.2.15 |
| org.scalatest | scalatest-funspec_2.12 | 3.2.15 |
| org.scalatest | scalatest-funsuite_2.12 | 3.2.15 |
| org.scalatest | scalatest-matchers-core_2.12 | 3.2.15 |
| org.scalatest | scalatest-mustmatchers_2.12 | 3.2.15 |
| org.scalatest | scalatest-propspec_2.12 | 3.2.15 |
| org.scalatest | scalatest-refspec_2.12 | 3.2.15 |
| org.scalatest | scalatest-shouldmatchers_2.12 | 3.2.15 |
| org.scalatest | scalatest-wordspec_2.12 | 3.2.15 |
| org.scalatest | scalatest_2.12 | 3.2.15 |
| org.slf4j | jcl-over-slf4j | 2.0.6 |
| org.slf4j | jul-to-slf4j | 2.0.6 |
| org.slf4j | slf4j-api | 2.0.6 |
| org.threeten | threeten-extra | 1.7.1 |
| org.tukaani | xz | 1.9 |
| org.typelevel | algebra_2.12 | 2.0.1 |
| org.typelevel | cats-kernel_2.12 | 2.1.1 |
| org.typelevel | spire-macros_2.12 | 0.17.0 |
| org.typelevel | spire-platform_2.12 | 0.17.0 |
| org.typelevel | spire-util_2.12 | 0.17.0 |
| org.typelevel | spire_2.12 | 0.17.0 |
| org.wildfly.openssl | wildfly-openssl | 1.1.3.Final |
| org.xerial | sqlite-jdbc | 3.42.0.0 |
| org.xerial.snappy | snappy-java | 1.1.8.4 |
| org.yaml | snakeyaml | 1.33 |
| oro | oro | 2.0.8 |
| pl.edu.icm | JLargeArrays | 1.5 |
| software.amazon.cryptools | AmazonCorrettoCryptoProvider | 1.6.1-linux-x86_64 |
| software.amazon.ion | ion-java | 1.0.2 |
| stax | stax-api | 1.0.1 |