Databricks Runtime 17.3 LTS (Beta)

The following release notes provide information about Databricks Runtime 17.3 LTS (Beta), powered by Apache Spark 4.0.0.

Azure Databricks released this LTS version in October 2025.

New features and improvements

• EXECUTE IMMEDIATE using constant expressions
• Use LIMIT ALL to suspend the total size restriction on recursive CTE
• Allow spark.sql.files.maxPartitionBytes in serverless compute
• Appending to files in Unity Catalog volumes returns correct error
• st_dump function is now supported
• st_interiorringn function is now supported
• st_numinteriorrings function is now supported

EXECUTE IMMEDIATE using constant expressions

You can pass constant expressions as teh SQL String as well as the arguments to parameter markers. This generalization eliminates the need to compose the query and arguments using temporary variables. For more information see: EXECUTE IMMEDIATE

Use LIMIT ALL to suspend the total size restriction on recursive CTE

In Databricks Runtime 17.3 and above, you can use LIMIT ALL to suspend any limit on the total size of a recursive common table expression.

Allow spark.sql.files.maxPartitionBytes in serverless compute

You can now configure spark.sql.files.maxPartitionBytes in serverless compute. This setting was already available in serverless SQL warehouses, and is now allowlisted for serverless compute as well. See Performance tuning.

Appending to files in Unity Catalog volumes returns correct error

Any code that attempts to append content to a Unity Catalog volume file now fails with an "Illegal seek" error message. Previously, it failed with an incorrect "operation not supported" error.

st_dump function is now supported

You can now use the st_dump function to get an array containing the single geometries of the input geometry. See st_dump function.

st_interiorringn function is now supported

You can now use the st_interiorringn function to extract the n-th inner boundary of a polygon and return it as a linestring. See st_interiorringn function.

st_numinteriorrings function is now supported

You can now use the st_numinteriorrings function to get the number of inner boundaries (rings) of a polygon. See st_numinteriorrings function.

Behavioral changes

• Support MV/ST refresh information in DESCRIBE EXTENDED AS JSON
• Fix temporal value conversion in struct literals when using Connect mode
• Support for complex types in observations in the Spark Connect Scala client
• Preserve null values in array, map, and struct literals
• Default mode change for FSCK REPAIR TABLE command
• Preserve nullability for typed literals
• Fix case class conversion in array and map literals
• Improved handling of null structs in Parquet
• Upgrade aws-msk-iam-auth library for Kafka

Support MV/ST refresh information in DESCRIBE EXTENDED AS JSON

Azure Databricks now generates a section for materialized view and streaming table refresh information in the DESCRIBE EXTENDED AS JSON output, including last refreshed time, refresh type, status, and schedule.

Fix temporal value conversion in struct literals when using Connect mode

In Spark Connect mode, struct literals in TypedLit now correctly handle temporal values instead of throwing exceptions.

Support for complex types in observations in the Spark Connect Scala client

The Spark Connect Scala client now supports array, map, and struct types in observations, bringing behavior in line with dedicated clusters.

Preserve null values in array, map, and struct literals

In Spark Connect mode, null values inside array, map, and struct literals are now preserved instead of being replaced with protobuf default values.

Default mode change for FSCK REPAIR TABLE command

The FSCK REPAIR TABLE command now includes an initial metadata repair step that validates checkpoints and partition values before removing references to missing data files.

Preserve nullability for typed literals

The Spark Connect Scala client now correctly preserves the nullability of array and map types for typed literals. Previously, array elements and map values were always treated as nullable.

Fix case class conversion in array and map literals

In Spark Connect mode, array and map literals in TypedLit now correctly handle case class values instead of throwing exceptions.

Improved handling of null structs in Parquet

Azure Databricks now correctly detects null structs when all requested struct fields are missing from a Parquet file, which makes behavior consistent between Photon and non-Photon readers.

Upgrade aws-msk-iam-auth library for Kafka

The aws-msk-iam-auth library has been upgraded to version 2.0.1+ to support setting a regional Security Token Service (STS) endpoint for Managed Streaming for Apache Kafka (MSK) IAM authentication.

Library upgrades

Upgraded Python libraries:

No Python libraries were upgraded in this version.

Upgraded R libraries:

No R libraries were upgraded in this version.

Upgraded Java libraries:

No Java libraries were upgraded in this version.

Apache Spark

Databricks Runtime 17.3 (Beta) includes Apache Spark 4.0.0. This release includes all Spark fixes and improvements included in version 17.2, as well as the following additional bug fixes and improvements made to Spark:

• SPARK-53507 Add breaking change info to errors
• SPARK-53574 Fix AnalysisContext being wiped during nested plan resolution
• SPARK-53444 Rework execute immediate
• SPARK-53560 Crash looping when retrying uncommitted batch in Kafka source and AvailableNow trigger
• SPARK-53127 Enable LIMIT ALL to override recursion row limit
• SPARK-53574 Revert "[SC-206548] Fix AnalysisContext being wipe…
• SPARK-53674 Handle single-pass analyzer LCAs when assigning aliases
• SPARK-53492 Reject second ExecutePlan with an operation id that has already completed
• SPARK-53677 Improve debuggability for JDBC data source when query contains syntax error
• SPARK-53490 Fix Protobuf conversion in observed metrics
• SPARK-53671 Exclude 0-args from @udf eval type inference
• SPARK-53592 Rework "Make @udf support vectorized UDF"
• SPARK-53654 Rework "Support seed in function uuid"
• SPARK-53429 Support Direct Passthrough Partitioning in the PySpark Dataframe API
• SPARK-53372 Shared Testing Runtime Package for LDP
• SPARK-53574 Fix AnalysisContext being wiped during nested plan resolution
• SPARK-53654 Revert "[SC-207022][sql][PYTHON] Support seed in function uuid"
• SPARK-53574 Revert "[SC-206548] Fix AnalysisContext being wiped during nested plan resolution"
• SPARK-53559 Fix HLL sketch updates to use raw collation key bytes
• SPARK-53654 Support seed in function uuid
• SPARK-52449 Make datatypes for Expression.Literal.Map/Array optional
• SPARK-53574 Fix AnalysisContext being wiped during nested plan resolution
• SPARK-53625 Propagate metadata columns through projections to address ApplyCharTypePadding incompatibility
• SPARK-53558 Revert "[SC-206544][sql] Show fully qualified table name including the catalog name in the exception message when the table is not found"
• SPARK-53558 Show fully qualified table name including the catalog name in the exception message when the table is not found
• SPARK-53357 Update pandas to 2.3.2
• SPARK-52659Misleading modulo error message in ansi mode
• SPARK-53592 Revert "[SC-206971][python] Make @udf support vectorized UDF"
• SPARK-53592 Make @udf support vectorized UDF
• SPARK-52601 Support primitive types in TransformingEncoder
• SPARK-53355 fix numpy 1.x repr in type tests
• SPARK-53387 Add support for Arrow UDTFs with PARTITION BY
• SPARK-52991 Implement MERGE INTO with SCHEMA EVOLUTION for V2 Data Source
• SPARK-53568 Fix several small bugs in Spark Connect Python client error handling logic
• SPARK-43579 optim: Cache the converter between Arrow and pandas for reuse
• SPARK-53524 Fix temporal value conversion in LiteralValueProtoConverter
• SPARK-53600 Revise SessionHolder last access time log message
• SPARK-53529 Fix pyspark connect client to support IPv6
• SPARK-53537 Adding Support for Parsing CONTINUE HANDLER
• SPARK-53623 improve reading large table properties performance
• SPARK-53523 Named parameters respect spark.sql.caseSensitive
• SPARK-53449 Simply options for builtin Datasource Scan related classes
• SPARK-53620 SparkSubmit should print stacktrace when exitFn is called
• SPARK-53518 No truncation for catalogString of User Defined Type
• SPARK-53568 Revert "[SC-206538][connect][PYTHON] Fix several small bugs in Spark Connect Python client error handling logic"
• SPARK-53602 Profile dump improvement and profiler doc fix
• SPARK-53402 Support Direct Passthrough Partitioning Dataset API in Spark Connect in Scala
• SPARK-53491 Fix exponential formatting of inputRowsPerSecond and processedRowsPerSecond in progress metrics JSON
• SPARK-53413 Shuffle cleanup for commands
• SPARK-53518 Revert "[SC-205989][sql] No truncation for catalogString of User Defined Type"
• SPARK-53568 Fix several small bugs in Spark Connect Python client error handling logic
• SPARK-53552 Optimize substr SQL function
• SPARK-53527 Improve fallback of analyzeExistenceDefaultValue
• SPARK-53584 Improve process_column_param validation and column parameter docstring
• SPARK-53498 Correctly Reference pyspark/pipelines/cli.py from spark-pipelines Binary
• SPARK-53518 No truncation for catalogString of User Defined Type
• SPARK-53233 Make the code related to streaming uses the correct package name
• SPARK-53598 Check the existence of numParts before reading large table property
• SPARK-53372 SDP End to End Testing Suite
• SPARK-53563 Optimize: sql_processor by avoiding inefficient string concatenation
• SPARK-53323 Enable Spark Connect tests for df.asTable() in Arrow UDTF
• SPARK-53157 Decouple driver and executor polling intervals
• SPARK-53555 Fix: SparkML-connect can't load SparkML (legacy mode) saved model
• SPARK-53361 Optimizing JVM-Python Communication in TWS by Grouping Multiple Keys into One Arrow Batch
• SPARK-53233 Revert "[SC-203716][sql][SS][mllib][CONNECT] Make the code related to streaming uses the correct package name"
• SPARK-53233 Make the code related to streaming uses the correct package name
• SPARK-53525 Spark Connect ArrowBatch Result Chunking
• SPARK-53444 Revert "[SC-206536][sql] Rework execute immediate"
• SPARK-53594 Make arrow UDF respect user-specified eval type
• SPARK-53444 Rework execute immediate
• SPARK-53582 Extend isExtractable so it can be applied on UnresolvedExtractValue
• SPARK-53572 Avoid throwing from ExtractValue.isExtractable
• SPARK-52346 Fix counter initialization and increment logic for flow retries
• SPARK-53561 Catch Interruption Exception in TransformWithStateInPySparkStateServer during outputStream.flush to avoid the worker crash
• SPARK-53512 Better unification of DSv2 PushDownUtils
• SPARK-53459 Use ReadOnlySparkConf in more places
• SPARK-53549 Always close the arrow allocator when list state request process is completed
• SPARK-53332 Enable StateDataSource with state checkpoint v2 (only snapshotStartBatchId option)
• SPARK-53550 Union output partitioning should compare canonicalized attributes
• SPARK-53506 Disallow % between Decimal and float under ANSI
• SPARK-52238 Rename Pipeline Spec Field "definitions" to 'libraries'
• SPARK-53538 ExpandExec should initialize the unsafe projections
• SPARK-53521 Refactor Star expression
• SPARK-53358 Improve arrow Python UDTF output type mismatch error message
• SPARK-53531 Better error message for HadoopRDD.getInputFormat
• SPARK-53434 ColumnarRow's get should also check isNullAt
• SPARK-53008 Add OSS SQL UDF data access routine validation
• SPARK-44856 Improve Python UDTF arrow serializer performance
• SPARK-53488 Update CollationTypeCoercion to be compatible with single-pass Analyzer logic
• SPARK-53029 Support return type coercion for Arrow Python UDTFs
• SPARK-53479 Align == behavior with pandas when comparing against scalar under ANSI
• SPARK-53497 Disallow /|//|* between Decimal and float under ANSI
• SPARK-53441 Bools |/&/^ None should fail under ANSI
• SPARK-53474 Add conf DATA_SOURCE_V2_EXPR_FOLDING
• SPARK-53333 Enable StateDataSource with state checkpoint v2 (only readChangeFeed)
• SPARK-53502 Improve the naming of methods in LiteralValueProtoConverter
• SPARK-52694 Add o.a.s.sql.Encoders#udtAPI
• SPARK-53499 from_arrow_type should respect containsNull in ArrayType
• SPARK-53515 Remove unused private lazy val from SchemaOfCsv/Xml
• SPARK-53480 Refactor arrow python runner code
• SPARK-53481 Make Hybrid queue class
• SPARK-53401 Enable Direct Passthrough Partitioning in the DataFrame API
• SPARK-52930 Use DataType.Array/Map for Array/Map Literals
• SPARK-52346 Propagate partition columns from destination for BatchTableWrite
• SPARK-53342 Fix Arrow converter to handle multiple record batches in single IPC stream
• SPARK-53421 Propagate Logical Plan ID in SDP Analysis
• SPARK-53408 Remove unused functions from QueryCompilationErrors
• SPARK-53044 Change Declarative Pipelines import alias convention from "sdp" to "dp"
• SPARK-53450 Nulls are filled unexpectedly after converting hive table scan to logical relation
• SPARK-53290 Fix Metadata backward-compatibility breaking
• SPARK-52982 Disallow lateral join with Arrow Python UDTFs
• SPARK-52851 Remove public APIs for append once flows
• SPARK-52511 Support dry-run mode in spark-pipelines command
• SPARK-53290 Revert "[SC-204277][sql][CONNECT] Fix Metadata backward-compatibility breaking"
• SPARK-53290 Fix Metadata backward-compatibility breaking
• SPARK-52577 Add tests for Declarative Pipelines DatasetManager with Hive catalog
• SPARK-53012 Support Arrow Python UDTF in Spark Connect
• SPARK-53251 Enable DataFrame API testing with asTable() for Arrow Python UDTFs
• SPARK-52432 Scope DataflowGraphRegistry to Session
• SPARK-52853 Prevent imperative PySpark methods in declarative pipelines
• SPARK-52745 Ensure one of the schema and columns in the Table interface is implemented and columns is preferable
• SPARK-52991 Revert "[SC-204944][sql] Implement MERGE INTO with SCHEMA EVOLUTION for V2 Data Source"
• SPARK-52981 Add table argument support for Arrow Python UDTFs
• SPARK-53487 Improve comments in NormalizePlan.scala
• SPARK-52991 Implement MERGE INTO with SCHEMA EVOLUTION for V2 Data Source
• SPARK-52281 Change ALTER TABLE ALTER COLUMN TYPE STRING not to apply default collation if original data type was instance of StringType
• SPARK-52759 Throw exception if pipeline has no tables or persisted views
• SPARK-51575 Combine Python Data Source pushdown & plan read workers
• SPARK-53359 Fix Arrow UDTF to handle the results as iterator
• SPARK-52757 Rename "plan" field in DefineFlow to "relation"
• SPARK-52431 Finishing touches on Declarative Pipelines runner
• SPARK-52591 Validate streaming-ness of DFs returned by SDP table and standalone flow definitions
• SPARK-53030 Support Arrow writer for streaming Python data sources
• SPARK-52980 Support Arrow Python UDTFs
• SPARK-42841Assign a name to the error class _LEGACY_ERROR_TEMP_2005
• SPARK-52423 Add missing golang package name to pipelines.proto
• SPARK-52409 Only use PipelineRunEventBuffer in tests
• SPARK-52533 Support enabling only driver profiler
• SPARK-52716 Remove comment from Flow trait and references
• SPARK-52348 Add support for Spark Connect handlers for pipeline commands
• SPARK-52281 Revert "[SC-198058][sql] Change ALTER TABLE ALTER COLUMN TYPE STRING not to apply default collation if original data type was instance of StringType"
• SPARK-49386 Add memory based thresholds for shuffle spill
• SPARK-52281 Change ALTER TABLE ALTER COLUMN TYPE STRING not to apply default collation if original data type was instance of StringType
• SPARK-53329 Improve exception handling when adding artifacts
• SPARK-52772 Revert "[SC-202707][engrm-327][SQL] Inconsistent table attribute handling during updates"
• SPARK-52772 Inconsistent table attribute handling during updates
• SPARK-51739 Validate Arrow schema from mapInArrow & mapInPandas & DataSource
• SPARK-53355 test python udf type behavior
• SPARK-53443 Update python to SPDX-license
• SPARK-49968 The split function produces incorrect results with an empty regex and a limit
• SPARK-52582 Improve the memory usage of XML parser
• SPARK-53452 from_arrow_type should respect valueContainsNull
• SPARK-53103 Revert "[SC-204946][ss] Throw an error if state directory is not empty when query starts"
• SPARK-53103 Throw an error if state directory is not empty when query starts
• SPARK-51575 Revert "[SC-192406][python] Combine Python Data Source pushdown & plan read workers"
• SPARK-51575 Combine Python Data Source pushdown & plan read workers
• SPARK-53095 Support HMS v4.1
• SPARK-53126 Refactor SparkErrorUtils#stackTraceToString to handle null input and use it to replace ExceptionUtils#getStackTrace
• SPARK-49489 HMS client respects hive.thrift.client.maxmessage.size
• SPARK-53108 Implement the time_diff function in Scala
• SPARK-45265 Support Hive 4.0 metastore
• SPARK-53255 Ban org.apache.parquet.Preconditions
• SPARK-52142 Display table constraints in SHOW CREATE TABLE COMMAND
• SPARK-52875 Simplify V2 expression translation if the input is context-independent-foldable
• SPARK-53240 Ban com.google.common.collect.(ArrayList)?Multimap

• SPARK-52875 Revert "[SC-201880][sql] Simplify V2 expression translation if the input is context-independent-foldable"
• SPARK-52141 Display constraints in DESC commands
• SPARK-53386 Support query parameter ending with semicolon in JDBC connectors
• SPARK-52875 Simplify V2 expression translation if the input is context-independent-foldable
• SPARK-53437 InterpretedUnsafeProjection shall setNull4Bytes for YearMonthIntervalType field
• SPARK-53341 Expand golden test coverage on multivariable DECLARE
• SPARK-53156 Track Driver Memory Metrics when the Application ends
• SPARK-52689 Send DML Metrics to V2Write
• SPARK-52641 Primary key columns should not be nullable
• SPARK-48547 Add opt-in flag to have SparkSubmit automatically call System.exit after user code main method exits
• SPARK-52575 Introduce contextIndependentFoldable attribute for Expressions
• SPARK-52860 Support V2 write schema evolution in InMemoryTable
• SPARK-53435 Fix race condition in CachedRDDBuilder
• SPARK-51987 DSv2 expressions in column defaults on write
• SPARK-52235 Add implicit cast to DefaultValue V2 Expressions passed to DSV2
• SPARK-52236 Standardize analyze exceptions for default value
• SPARK-53294 Enable StateDataSource with state checkpoint v2 (only batchId option)
• SPARK-52095 Alter table alter column to pass V2Expression to DSV2
• SPARK-53415 Simply options for builtin FileFormats
• SPARK-53318 Support the time type by make_timestamp_ltz()
• SPARK-53143 Fix self join in DataFrame API - Join is not the only expected output from analyzer
• SPARK-53423 Move all the single-pass resolver related tags to ResolverTag
• SPARK-52116 Improve exception for non-deterministic default values
• SPARK-51906 Dsv2 expressions in alter table add columns
• SPARK-53403 Improve add/sub tests under ANSI
• SPARK-53418 Support TimeType in ColumnAccessor
• SPARK-52777 Enable shuffle cleanup mode configuration in Spark SQL
• SPARK-52485 Declarative Pipelines code cleanup
• SPARK-53236 Use Java ArrayList constructors instead of Lists.newArrayList in Java code
• SPARK-53392 Move SpecializedArray handling to connect-common
• SPARK-53176 Spark launcher should respect --load-spark-defaults
• SPARK-53237 Use Java Base64 instead of org.apache.commons.codec.binary.Base64 instance
• SPARK-53390 Raise error when bools with None astype to ints under ANSI
• SPARK-53109 Support TIME in the make_timestamp_ntz and try_make_timestamp_ntz functions in Scala
• SPARK-53393 Disable memory profiler for Arrow Scalar Iterator UDFs
• SPARK-53367 add int to decimal coercion for Arrow UDFs
• SPARK-53365 Unify code for persisting of configs in views and UDFs
• SPARK-53228 Use Java Map constructors instead of Maps.new*HashMap()
• SPARK-53197 Use java.util.Objects#requireNonNull instead of com.google.common.base.Preconditions#checkNotNull

• SPARK-53216 Move is*(Blank|Empty) from object SparkStringUtils to trait SparkStringUtils
• SPARK-53385 Refactor Identifier evaluation out
• SPARK-53384 Refactor variable resolution out
• SPARK-53195 Use Java InputStream.readNBytes instead of ByteStreams.read
• SPARK-53205 Support createParentDirs in SparkFileUtils
• SPARK-53196 Use Java OutputStream.nullOutputStream instead of ByteStreams.nullOutputStream
• SPARK-53381 Avoid creating temporary collections in toCatalystStruct
• SPARK-53275 Handle stateful expressions when ordering in interpreted mode
• SPARK-51585 Oracle dialect supports pushdown datetime functions
• SPARK-53200 Use Java Files.newInputStream instead of Files.asByteSource().openStream()
• SPARK-53190 Use Java InputStream.transferTo instead of ByteStreams.copy
• SPARK-53188 Support readFully in SparkStreamUtils and JavaUtils
• SPARK-53354 Simplify LiteralValueProtoConverter.toCatalystStruct
• SPARK-52873 Further restrict when SHJ semi/anti join can ignore duplicate keys on the build side
• SPARK-53180 Use Java InputStream.skipNBytes instead of ByteStreams.skipFully
• SPARK-53369 Fix error message for UDFs with CHAR/VARCHAR return type
• SPARK-53330 Fix Arrow UDF with DayTimeIntervalType (bounds != start/end)
• SPARK-53366 Apply formatting rules to sql/connect/shims
• SPARK-53348 Always persist ANSI value when creating a view or assume it when querying if not stored
• SPARK-53119 Support touch in SparkFileUtils
• SPARK-52592 Support creating a ps.Series from a ps.Series
• SPARK-53360 Once strategy with ConstantFolding's idempotence should not be broken
• SPARK-53135 Support copyURLToFile in SparkFileUtils and JavaUtils
• SPARK-53150 Improve list(File|Path)s to handle non-existent, non-directory, symlink input
• SPARK-53135 Revert "[SC-203164][core][SQL] Support copyURLToFile in SparkFileUtils and JavaUtils"
• SPARK-53137 Support forceDeleteOnExit in SparkFileUtils and JavaUtils
• SPARK-53352 Refine the error message for unsupported return type
• SPARK-53353 Fail Scalar Iterator Arrow UDF with 0-arg
• SPARK-53135 Support copyURLToFile in SparkFileUtils and JavaUtils
• SPARK-53101 Support (left|right)Pad in SparkStringUtils
• SPARK-53117 Support moveDirectory in SparkFileUtils and JavaUtils
• SPARK-53121 Use deleteRecursively instead of FileUtils.forceDelete
• SPARK-53100 Use Java String.substring instead of StringUtils.substring
• SPARK-53092 Ban org.apache.commons.lang3.SystemUtils
• SPARK-53328 Improve debuggability for SparkML-connect
• SPARK-52065 Produce another plan tree with output columns (name, data type, nullability) in plan change logging
• SPARK-51182 DataFrameWriter should throw dataPathNotSpecifiedError when path is not specified
• SPARK-52410 Deprecate PipelineConf and use SqlConf directly
• SPARK-52852 Remove unused spark_conf in create_streaming_table
• SPARK-52714 Remove unused comment arg in append_flow decorator
• SPARK-52663 Introduce name field to pipeline spec
• SPARK-53091 Ban org.apache.commons.lang3.Strings
• SPARK-52393 Pipeline SQL Graph Element Registration
• SPARK-52912 Improve SparkStringUtils to support is(Not)?(Blank|Empty)
• SPARK-53307 Remove RetriesExceeded error from Spark Connect Python and Scala clients
• SPARK-52346 Declarative Pipeline DataflowGraph execution and event logging
• SPARK-52877 Improve Python UDF Arrow Serializer Performance
• SPARK-51920 Fix composite/nested type in value state for python TWS
• SPARK-52741 RemoveFiles ShuffleCleanup mode doesnt work with non-adaptive execution
• SPARK-52238 Python client for Declarative Pipelines
• SPARK-51926 Add INVALID_CONF_VALUE.subXXX error class to conf errors
• SPARK-53155 Global lower agggregation should not be replaced with a project
• SPARK-52911 Remove StringUtils.(split|chop) usage
• SPARK-51926 Revert "[SC-195096][core][SQL] Add INVALID_CONF_VALUE.subXXX error class to conf errors"
• SPARK-53346 Avoid creating temporary collections in toCatalystArray/toCatalystMap
• SPARK-51926 Add INVALID_CONF_VALUE.subXXX error class to conf errors
• SPARK-53136 tryWithResource & tryInitializeResource shall close resource quietly
• SPARK-53132 Support list(File|Path)s in SparkFileUtils and JavaUtils
• SPARK-51896 Add Java Enum Support for TypedConfigBuilder
• SPARK-53311 Make PullOutNonDeterministic use canonicalized expressions
• SPARK-51527 Make codegen log level configurable via SQLConf
• SPARK-52223 Add SDP Spark Connect Protos
• SPARK-52283 Declarative Pipelines DataflowGraph creation and resolution
• SPARK-52880 Improve toString by JEP-280 instead of ToStringBuilder
• SPARK-53284 Adjust imports of Spark config in tests
• SPARK-53050 Enable MultiIndex.to_series() to return struct for each entry
• SPARK-52988 Fix race conditions at CREATE TABLE and FUNCTION when IF NOT EXISTS is used
• SPARK-52874 Support o.a.s.util.Pair Java Record
• SPARK-52710 DESCRIBE SCHEMA should print collation
• SPARK-49984 Fix supplementJava(Module|IPv6)Options to update extraJavaOptions only

Azure Databricks ODBC/JDBC driver support

Azure Databricks supports ODBC/JDBC drivers released in the past 2 years. Please download the recently released drivers and upgrade (download ODBC, download JDBC).

Maintenance updates

See Databricks Runtime maintenance updates.

System environment

• Operating System: Ubuntu 24.04.3 LTS
• Java: Zulu17.58+21-CA
• Scala: 2.13.16
• Python: 3.12.3
• R: 4.4.2
• Delta Lake: 4.0.0

Installed Python libraries

Installed R libraries

R libraries are installed from the Posit Package Manager CRAN snapshot on 2025-03-20.

Installed Java and Scala libraries (Scala 2.13 cluster version)