Databricks Runtime 17.2 (Beta)

The following release notes provide information about Databricks Runtime 17.2 (Beta), powered by Apache Spark 4.0.0.

Azure Databricks released this version in August 2025.

New features and improvements

• ST_ExteriorRing function is now supported
• Show pushdown details in DESCRIBE and Spark UI for remote scans
• Support TEMPORARY keyword for metric view creation
• Support additional commands with fine-grained access control
• Use native I/O for LokiFileSystem.getFileStatus on S3
• ANSI mode enabled by default for Pandas API on Spark

ST_ExteriorRing function is now supported

You can now use the ST_ExteriorRing function to extract the outer boundary of a polygon and return it as a linestring. See st_exteriorring function.

Show pushdown details in DESCRIBE and Spark UI for remote scans

The DESCRIBE command and Spark UI for RemoteSparkConnectScan on dedicated compute now display predicates, aggregations, group by clauses, limits, and samples that are pushed down into the remote scan.

Support TEMPORARY keyword for metric view creation

You can now use the TEMPORARY keyword when creating a metric view. Temporary metric views are visible only in the session that created them and are dropped when the session ends. See CREATE VIEW.

Support additional commands with fine-grained access control

Fine-grained access control on dedicated compute now supports the FSCK REPAIR TABLE and DESCRIBE DETAIL commands.

Use native I/O for LokiFileSystem.getFileStatus on S3

LokiFileSystem.getFileStatus now uses the native I/O stack for Amazon S3 traffic and returns org.apache.hadoop.fs.FileStatus objects instead of shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileStatus.

ANSI mode enabled by default for Pandas API on Spark

ANSI_MODE is now enabled by default for Pandas API on Spark with compute.ansi_mode_support=True, which ensures ANSI compliance parity with native pandas. The compute.fail_on_ansi_mode option still applies if compute.ansi_mode_support is set to False.

Behavioral changes

• DESCRIBE CONNECTION shows environment settings for JDBC connections
• Option to truncate uniform history during managed tables migration
• SQL syntax for Delta read options in streaming queries
• Correct results for split with empty regex and positive limit
• Fix url_decode and try_url_decode error handling in Photon
• Shared execution environment for Unity Catalog Python UDTFs

DESCRIBE CONNECTION shows environment settings for JDBC connections

Azure Databricks now includes user-defined environment settings in the DESCRIBE CONNECTION output for JDBC connections that support custom drivers and run in isolation. Other connection types remain unchanged.

Option to truncate uniform history during managed tables migration

You can now truncate uniform history when migrating tables with Uniform/Iceberg enabled using ALTER TABLE...SET MANAGED. This simplifies migrations and reduces downtime compared to disabling and re-enabling Uniform manually.

SQL syntax for Delta read options in streaming queries

You can now specify Delta read options for SQL-based streaming queries using the WITH clause. For example:

SELECT * FROM STREAM tbl WITH (SKIPCHANGECOMMITS=true, STARTINGVERSION=X);

Correct results for split with empty regex and positive limit

Azure Databricks now returns correct results when using split function with an empty regex and a positive limit. Previously, the function incorrectly truncated the remaining string instead of including it in the last element.

Fix url_decode and try_url_decode error handling in Photon

In Photon, try_url_decode() and url_decode() with failOnError = false now return NULL for invalid URL-encoded strings instead of failing the query.

Shared execution environment for Unity Catalog Python UDTFs

Azure Databricks now shares the execution environment for Python user-defined table functions (UDTFs) from the same owner and Spark session. An optional STRICT ISOLATION clause is available to disable sharing for UDTFs with side effects, such as modifying environment variables or executing arbitrary code.

Library upgrades

Upgraded Python libraries:

No Python libraries were upgraded in this version.

Upgraded R libraries:

No R libraries were upgraded in this version.

Upgraded Java libraries:

• io.delta.delta-sharing-client_2.13 from 1.3.3 to 1.3.5

Apache Spark

Databricks Runtime 17.2 (Beta) includes Apache Spark 4.0.0. This release includes all Spark fixes and improvements included in version 17.1, as well as the following additional bug fixes and improvements made to Spark:

• SPARK-53183 Use Java Files.readString instead of o.a.s.sql.catalyst.util.fileToString
• SPARK-51817 Re-introduce ansiConfig fields in messageParameters of CAST_INVALID_INPUT and CAST_OVERFLOW
• SPARK-53124 Prune unnecessary fields from JsonTuple
• SPARK-53106 Add schema evolution tests for TWS Scala spark connect suites
• SPARK-53201 Use SparkFileUtils.contentEquals instead of Files.equal
• SPARK-53308 Don't remove aliases in RemoveRedundantAliases that would cause duplicates
• SPARK-53241 Support createArray in SparkCollectionUtils
• SPARK-53239 Improve MapSort and SortArray performance via parallelSort
• SPARK-53184 Fix melt() when value columns mix strings and numbers
• SPARK-53144 Make CreateViewCommand in SparkConnectPlanner side effect free
• SPARK-53303 Use the empty state encoder when the initial state is not provided in TWS
• SPARK-52917 Read support to enable round-trip for binary in xml format
• SPARK-53110 Implement the time_trunc function in PySpark
• SPARK-53107 Implement the time_trunc function in Scala
• SPARK-52146 Detect cyclic function references in SQL UDFs
• SPARK-52469 Use JEP 223 API to process Java version
• SPARK-53094 Fix CUBE with aggregate containing HAVING clauses
• SPARK-51874 Add TypedConfigBuilder for Scala Enumeration
• SPARK-53287 Add ANSI Migration Guide
• SPARK-43100 Push Based Shuffle metrics must be deserialized correctly
• SPARK-53045 DESCRIBE EXTENDED should be resilient to corrupt metadata
• SPARK-53114 Support join in JavaUtils
• SPARK-53297 Fix StreamingTable Declarative Pipelines API docstring
• SPARK-53181 Enable doc tests under ANSI
• SPARK-52482 Improve exception handling for reading certain corrupt zstd files
• SPARK-52990 Support StringSubstitutor
• SPARK-51874 Add TypedConfigBuilder for Scala Enumeration
• SPARK-53169 Remove comments related to "Set the logger level of File Appender to" from log4j2.properties
• SPARK-53080 Support cleanDirectory in SparkFileUtils and JavaUtils
• SPARK-53045 DESCRIBE EXTENDED should be resilient to corrupt metadata
• SPARK-53000 Rename StringUtils.scala to StringConcat.scala in sql/api module
• SPARK-52952 Add PySpark UDF Type Coercion Dev Script
• SPARK-52998 Multiple variables inside declare
• SPARK-51817 Re-introduce ansiConfig fields in messageParameters of CAST_INVALID_INPUT and CAST_OVERFLOW
• SPARK-52820 Capture full plans in golden files
• SPARK-53274 Support left and right join pushdown in JDBCScanBuilder
• SPARK-52110 Implement SQL syntax support for pipelines
• SPARK-52950 Enable ANSI mode in DataSourceV2StrategySuite
• SPARK-52947 Fix image path in declarative pipelines programming guide
• SPARK-52592 Support creating a ps.Series from a ps.Series
• SPARK-53301 Differentiate type hints of Pandas UDF and Arrow UDF
• SPARK-53146 Make MergeIntoTable in SparkConnectPlanner side effect free
• SPARK-53166 Use SparkExitCode.EXIT_FAILURE in SparkPipelines object
• SPARK-53288 Fix assertion error with streaming global limit
• SPARK-52394 Fix autocorr divide-by-zero error under ANSI mode
• SPARK-51555 Add the time_diff() function
• SPARK-52948 Enable test_np_spark_compat_frame under ANSI
• SPARK-53134 Clean up unused ANSI imports in tests
• SPARK-52593 Avoid CAST_INVALID_INPUT of MultiIndex.to_series, Series.dot and DataFrame.dot in ANSI mode
• SPARK-53291 Fix nullability for value column
• SPARK-53097 Make WriteOperationV2 in SparkConnectPlanner side effect free
• SPARK-53305 Support TimeType in createDataFrame
• SPARK-52914 Support On-Demand Log Loading for rolling logs in History Server
• SPARK-33538 Directly push IN/NOT predicates to the Hive Metastore
• SPARK-52849 Add stringifyException to o.a.s.util.Utils
• SPARK-52771 Fix float32 type widening in truediv/floordiv
• SPARK-52502 Thread count overview
• SPARK-52788 Fix error of converting binary value in BinaryType to XML
• SPARK-53123 Support getRootCause in SparkErrorUtils
• SPARK-53129 Improve SparkShell to import java.net._ by default
• SPARK-53061 Support copyFileToDirectory in SparkFileUtils
• SPARK-52683 Support ExternalCatalog alterTableSchema
• SPARK-52871 Merge o.a.s.sql.catalyst.util.SparkStringUtils to o.a.s.util.SparkStringUtils
• SPARK-52817 Fix Like Expression performance
• SPARK-52545 Standardize double-quote escaping to follow SQL specification
• SPARK-52711 Fix float32 type widening in mul/rmul under ANSI
• SPARK-52615 Replace File.mkdirs with Utils.createDirectory
• SPARK-52381 JsonProtocol: Only accept subclasses of SparkListenerEvent
• SPARK-52613 Restore printing full stacktrace when HBase/Hive DelegationTokenProvider hit exception
• SPARK-52651 Handle User Defined Type in Nested ColumnVector
• SPARK-52611 Fix SQLConf version for excludeSubqueryRefsFromRemoveRedundantAliases…
• SPARK-52552 Skip CHECK constraint enforcement for deletion vector deletes
• SPARK-52587 spark-shell 2.13 support -i -I parameter
• SPARK-52492 Make InMemoryRelation.convertToColumnarIfPossible customizable
• SPARK-52451 Make WriteOperation in SparkConnectPlanner side effect free
• SPARK-53272 Refactor SPJ pushdown logic out of BatchScanExec
• SPARK-53071 Support copyFile in SparkFileUtils
• SPARK-51415 Support the time type by make_timestamp()
• SPARK-51554 Add the time_trunc() function
• SPARK-52426 Support redirecting stdout/stderr to logging system
• SPARK-51834 Support end-to-end table constraint management
• SPARK-53063 Implement and call new APIs in FileCommitProtocol instead of the deprecated
• SPARK-52546 check sparkContext if has stopped when running catch code block in execute(), otherwise, it will return wrong state.
• SPARK-53023 Remove commons-io dependency from sql/api module
• SPARK-52396 Artifact Root Directory should use tmpdir
• SPARK-53131 Improve SparkShell to import java.nio.file._ by default
• SPARK-42322 Assign name to_LEGACY_ERROR_TEMP_2235
• SPARK-51834 Support end-to-end table constraint management
• SPARK-52484 Skip child.supportsColumnar assertion from driver side in ColumnarToRowExec
• SPARK-52384 Fix bug Connect should insensitive for JDBC options
• SPARK-52034 Add common methods in SparkOperation trait for thriftserver operations
• SPARK-53031 Support getFile in SparkFileUtils
• SPARK-52976 Fix Python UDF not accepting collated string as input param/return type (17.x)
• SPARK-52943 Enable arrow_cast for all pandas UDF eval types
• SPARK-53263 Support TimeType in df.toArrow
• SPARK-53141 Add APIs to get overhead memory size and offheap memory size from resource profile
• SPARK-53259 Correct the message for INVALID_UDF_EVAL_TYPE
• SPARK-53167 Spark launcher isRemote also respects properties file
• SPARK-53165 Add SparkExitCode.CLASS_NOT_FOUND
• SPARK-53171 Improvement UTF8String repeat
• SPARK-53170 Improve SparkUserAppException to have cause parameter
• SPARK-52989 Add explicit close() API to State Store iterators
• SPARK-53074 Avoid partial clustering in SPJ to meet a child's required distribution
• SPARK-52252 ScalaUDF encoders in subquery should be resolved
• SPARK-53244 Don't store dual-run enabled and tentative mode enabled confs during view creation
• SPARK-53192 Always cache a DataSource in the Spark Connect Plan Cache
• SPARK-51813 Add a nonnullable DefaultCachedBatchKryoSerializer to avoid null propagating in DefaultCachedBatch serde
• SPARK-52904 Reapply "[SC-202233][python] Enable convertToArrowArrayS…
• SPARK-53253 Fix register UDF of type SQL_SCALAR_ARROW_ITER_UDF
• SPARK-53243 List the supported eval types in arrow nodes
• SPARK-53130 Fix toJson behavior of collated string types
• SPARK-53003 Support strip in SparkStringUtils
• SPARK-51902 Sync OSS: Enforce check constraint on table insertion
• SPARK-52047 Raise PySparkValueError for unsupported plot kinds
• SPARK-53004 Support abbreviate in SparkStringUtils
• SPARK-53066 Improve EXPLAIN output for DSv2 Join pushdown
• SPARK-30848 Remove productHash from TreeNode
• SPARK-53046 Use Java readAllBytes instead of IOUtils.toByteArray
• SPARK-53206 Use SparkFileUtils.move instead of com.google.common.io.Files.move
• SPARK-53066 Improve EXPLAIN output for DSv2 Join pushdown
• SPARK-53069 Fix incorrect state store metrics with virtual column families
• SPARK-52904 Enable convertToArrowArraySafely by default [17.x]
• SPARK-52821 add int->DecimalType pyspark udf return type coercion
• SPARK-51562 Add time function
• SPARK-52971 Limit idle Python worker queue size
• SPARK-53057 Support sizeOf in SparkFileUtils and JavaUtils
• SPARK-53040 Ban self references inside topmost CTEs in Recursive CTEs
• SPARK-53104 Introduce ansi_mode_context to avoid multiple config checks per API call
• SPARK-47547 Add BloomFilter V2 and use it as default
• SPARK-53098 DeduplicateRelations shouldn't remap expressions if old ExprId still exists in output
• SPARK-53049 Support toString in SparkStreamUtils
• SPARK-53062 Support deleteQuietly in SparkFileUtils and JavaUtils
• SPARK-53070 Support is(Not)?Empty in SparkCollectionUtils
• SPARK-53020 JPMS args should also apply to non-SparkSubmit process
• SPARK-52968 Emit additional state store metrics
• SPARK-52975 Simplify field names in pushdown join sql
• SPARK-52926 Added SQLMetric for remote schema fetching time duration.
• SPARK-53059 Arrow UDF no need to depend on pandas
• SPARK-52646 Avoid CAST_INVALID_INPUT of __eq__ in ANSI mode
• SPARK-52622 Avoid CAST_INVALID_INPUT of DataFrame.melt in ANSI mode
• SPARK-52985 Raise TypeError for pandas numpy operand in comparison operators
• SPARK-52580 Avoid CAST_INVALID_INPUT of replace in ANSI mode
• SPARK-52549 Disable Recursive CTE self-references from window functions and inside sorts
• SPARK-52895 Don't add duplicate elements in resolveExprsWithAggregate
• SPARK-50748 Fix a race condition issue which happens when operations are interrupted
• SPARK-52737 Pushdown predicate and number of apps to FsHistoryProvider when listing applications
• SPARK-53018 ArrowStreamArrowUDFSerializer should respect argument arrow_cast
• SPARK-53013 Fix Arrow-optimized Python UDTF returning no rows on lateral join
• SPARK-51834 Sync OSS: support end-to-end table constraint alter add/drop constraint
• SPARK-52921 Specify outputPartitioning for UnionExec for same output partitoning as children operators
• SPARK-52908 Prevent for iterator variable name clashing with names of labels in the path to the root of AST
• SPARK-52954 Arrow UDF support return type coercion
• SPARK-52925 Return correct error message for anchor self references in rCTEs
• SPARK-52889 Implement the current_time function in PySpark
• SPARK-52675 Interrupt hanging ML handlers in tests
• SPARK-52959 Support UDT in Arrow-optimized Python UDTF
• SPARK-52962 BroadcastExchangeExec should not reset metrics
• SPARK-52956 Preserve alias metadata when collapsing projects
• SPARK-52890 Implement the to_time and try_to_time functions in PySpark
• SPARK-52888 Implement the make_time function in PySpark
• SPARK-52892 Support TIME in the hour, minute, and second functions in PySpark
• SPARK-52837 Support TimeType in pyspark
• SPARK-52023 Fix data corruption/segfault returning Option[Product] from udaf
• SPARK-52955 Change return types of WindowResolution.resolveOrder and WindowResolution.resolveFrame to WindowExpression
• SPARK-52166 Add support for PipelineEvents
• SPARK-51834 Fix: remove valid from toDDL
• SPARK-52735 Fix missing error conditions for SQL UDFs
• SPARK-51834 Sync OSS Support end-to-end create/replace table with constraint
• SPARK-51834 Sync OSS update ResolvedIdentifier unapply calls
• SPARK-52929 Support MySQL and SQLServer connector for DSv2 Join pushdown
• SPARK-52448 Add simplified Struct Expression.Literal
• SPARK-52882 Implement the current_time function in Scala
• SPARK-52905 Arrow UDF for window
• SPARK-52876 Fix a typo buffer to body in ChunkFetchSuccess.toString
• SPARK-52883 Implement the to_time and try_to_time functions in Scala
• SPARK-52751 Don't eagerly validate column name in dataframe['col_name']
• SPARK-52791 Fix error when inferring a UDT with a null first element
• SPARK-52686 Union should be resolved only if there are no duplicates
• SPARK-52881 Implement the make_time function in Scala
• SPARK-52919 Fix DSv2 Join pushdown to use previously aliased column
• SPARK-52866 Add support for try_to_date
• SPARK-52846 Add a metric in JDBCRDD for how long it takes to fetch the resultset
• SPARK-52859 Add SparkSystemUtils trait
• SPARK-52823 Support DSv2 Join pushdown for Oracle connector
• SPARK-52165 Setup build scaffolding for the pipelines project
• SPARK-52869 Add FrameLessOffsetWindowFunction validation to validateResolvedWindowExpression for reuse in single-pass analyzer
• SPARK-52885 Implement the hour, minute, and second functions in Scala for the TIME type
• SPARK-52903 Trim non-top-level aliases before LCA resolution
• SPARK-52832 Fix JDBC dialect identifier quoting
• SPARK-52870 Properly quote variable names in FOR statement
• SPARK-52859 Add SparkSystemUtils trait
• SPARK-52900 Use SparkStringUtils.stringToSeq in FsHistoryProvider
• SPARK-52809 Don't hold reader and iterator references for all partitions in task completion listeners for metric update
• SPARK-52815 Improve SparkClassUtils to support getAllInterfaces
• SPARK-52795 Include runId in state store logs
• SPARK-52779 Support TimeType literal in Connect
• SPARK-52899 Fix QueryExecutionErrorsSuite test to register H2Dialect back
• SPARK-52862 Fix the nullability check of Decimal('NaN')
• SPARK-52214 Arrow UDF for aggregation
• SPARK-52787 Reorganize streaming execution dir around runtime and checkpoint areas
• SPARK-51564 TIME parsing in the 12hr clock format
• SPARK-52859 Add SparkSystemUtils trait
• SPARK-51562 Add time function
• SPARK-52850 Skip calling conversions if identity function
• SPARK-52863 Clean up code paths for old pandas versions
• SPARK-52516 Don't hold previous iterator reference after advancing to next file in ParquetPartitionReaderFactory
• SPARK-52848 Avoid cast to Double in casting TIME/TIMESTAMP to DECIMAL
• SPARK-52770 Support TIME type in connect proto
• SPARK-52842 New functionality and bugfixes for single-pass analyzer
• SPARK-52620 Support casting of TIME to DECIMAL
• SPARK-52783 Refactor windowFunction validation logic from checkAnalysis for reuse in single-pass analyzer
• SPARK-51695 Sync OSS create/replace/alter table for unique constraint via DSv2
• SPARK-51695 Sync OSS drop constraint via DSv2
• SPARK-52824 CheckpointFileManager error classification
• SPARK-52829 Fix LocalDataToArrowConversion.convert to handle empty rows properly
• SPARK-52811 Optimize ArrowTableToRowsConversion.convert to improve its performance
• SPARK-52796 Optimize LocalDataToArrowConversion.convert to improve its performance
• SPARK-51695 Sync OSS Alter Table Add Constraint parser changes
• SPARK-52619 Cast TimeType to IntegralType
• SPARK-52187 Introduce Join pushdown for DSv2
• SPARK-52782 Return NULL from +/- on datetime with NULL
• SPARK-51695 Introduce Parser Changes for Table Constraints (CHECK, PK, FK)
• SPARK-52808 spark.history.retainedApplications should be positive
• SPARK-52776 Do not split the comm field in ProcfsMetricsGetter
• SPARK-52793 Support isUnix at o.a.s.util.Utils
• SPARK-52792 Remove commons-lang3 dependency from network-common
• SPARK-52797 Use Utils.isWindows instead of re-evaluating in PythonWorkerFactory
• SPARK-52784 Add TimeZoneAware casting in WindowFrameTypeCoercion.createBoundaryCast
• SPARK-52725 Delay resource profile manager initialization until plugin is loaded
• SPARK-52565 Enforce ordinal resolution before other sort order expressions
• SPARK-52740 Fix NPE in HDFSBackedStateStoreProvider accessing StateStoreConf.sqlConf when checkpoint format version is >=2
• SPARK-52673 Add grpc RetryInfo handling to Spark Connect retry policies
• SPARK-52724 Enhance broadcast join OOM error handling with SHUFFLE_MERGE hint support
• SPARK-52781 Fix dtyeps typo in internal.py
• SPARK-52730 Store underlying driver and database version in JDBCRDD
• SPARK-52727 Refactor Window resolution in order to reuse it in single-pass analyzer
• SPARK-52695 User Defined Type write support for xml file format
• SPARK-52722 Deprecate JdbcRDD class
• SPARK-51695 Introduce Parser Changes for Table Constraints (CHECK, PK, FK)
• SPARK-52763 Support TIME subtract
• SPARK-52768 Fix typo "moveing" in pandas/series.py
• SPARK-52730 Store underlying driver and database version in JDBCRDD
• SPARK-52694 Add o.a.s.sql.Encoders#udtAPI
• SPARK-52720 Fix float32 type widening in add/radd under ANSI
• SPARK-52723 Server side column name validation
• SPARK-52760 Fix float32 type widening in sub/rsub under ANSI
• SPARK-52738 Support aggregating the TIME type with a UDAF when the underlying buffer is an UnsafeRow
• SPARK-52704 Simplify interoperations between SQLConf and file-format options in TextBasedFileFormats
• SPARK-52706 Fix inconsistencies and refactor primitive types in parser
• SPARK-52718 Fix float32 type widening in rmod/mod under ANSI
• SPARK-52736 Fix typos 'moveing' in pyspark/pandas/generic.py

Azure Databricks ODBC/JDBC driver support

Azure Databricks supports ODBC/JDBC drivers released in the past 2 years. Please download the recently released drivers and upgrade (download ODBC, download JDBC).

System environment

• Operating System: Ubuntu 24.04.3 LTS
• Java: Zulu17.58+21-CA
• Scala: 2.13.16
• Python: 3.12.3
• R: 4.4.2
• Delta Lake: 4.0.0

Installed Python libraries

Installed R libraries

R libraries are installed from the Posit Package Manager CRAN snapshot on 2025-03-20.

Installed Java and Scala libraries (Scala 2.13 cluster version)