Databricks Runtime 6.1 (EoS)

Article
09/30/2024

Note

Support for this Databricks Runtime version has ended. For the end-of-support date, see End-of-support history. For all supported Databricks Runtime versions, see Databricks Runtime release notes versions and compatibility.

Databricks released this version in October 2019.

The following release notes provide information about Databricks Runtime 6.1, powered by Apache Spark.

New features

Delta Lake enhancements

Python APIs for Delta tables (Public Preview)

You can now delete from and upsert data into Delta tables, clean tables, and get table history using Python APIs.

For details, see What is Delta Lake?.

Dynamic File Pruning (DFP) enabled by default

DFP uses Delta table metadata (for example, min/max column statistics in a file) significantly improves the performance of many queries by skipping files with column values out of range. Unlike Dynamic Partition Pruning (DPP), DFP works on non-partitioned columns as well. DFP can be beneficial for workloads that join on non-partitioned keys.

Configure clusters with your own container image using Databricks Container Services

Databricks Container Services lets you configure a cluster with your own container image. You can pre-package complex environments within a container, publish it to a popular container registry such as ACR, ECR, or Docker Hub, and then have Databricks pull the image to build a cluster. Some example use cases include:

Library customization - you have full control over the system libraries you want installed
Golden container environment - your Docker image is a locked down environment that will never change
Docker CI/CD integration - you can integrate Azure Databricks with your Docker CI/CD pipelines

There are many other use cases, ranging from specifying configuration to installing machine learning packages.

For details, see Customize containers with Databricks Container Service.

Secrets referenced by Spark configuration properties and environment variables (Public Preview)

As of the November 7 maintenance update of Databricks Runtime 6.1, the ability to reference a secret in a Spark configuration property or environment variable is in Public Preview. For details, see Use a secret in a Spark configuration property or environment variable.

Ganglia metrics now available on clusters with table ACLs and credential passthrough

Previously Ganglia metrics were not supported on clusters with table access control or credential passthrough enabled. They are now available.

Improvements

Upgraded netty-all from 4.1.17.Final to 4.1.39.Final
Upgraded Snowflake connector from 2.4.10 to 2.5.3

Credential passthrough

Microsoft Entra ID credentials are now propagated in notebooks run using dbutils.notebook.run().
All PySpark ML classes are now supported.
The Ganglia UI is now available.

For details, see Access Azure Data Lake Storage using Microsoft Entra ID credential passthrough (legacy).

Apache Spark

Databricks Runtime 6.1 includes Apache Spark 2.4.4. This release includes all Spark fixes and improvements included in Databricks Runtime 6.0 (EoS), as well as the following additional bug fixes and improvements made to Spark:

[SPARK-28782][SQL] Generator support in aggregate expressions
[SPARK-19147][CORE] Gracefully handle error in task after executor is stopped
[SPARK-26003][SQL] Improve SQLAppStatusListener.aggregateMetrics performance
[SPARK-29101][SQL] Fix count API for csv file when DROPMALFORMED mode is selected
[SPARK-29042][CORE] Sampling-based RDD with unordered input should be INDETERMINATE
[SPARK-29124][CORE] Use MurmurHash3 bytesHash(data, seed) instead of bytesHash(data)
[SPARK-26713][CORE] Interrupt pipe IO threads in PipedRDD when task is finished
[SPARK-29046][SQL] Fix NPE in SQLConf.get when active SparkContext is stopping
[SPARK-28483] Fix barrier mode task hang on barrier() when task killed
[SPARK-29087][CORE][STREAMING] Use DelegatingServletContextHandler to avoid CCE
[SPARK-27122][CORE] Jetty classes must not be return via getters in org.apache.spark.ui.WebUI
[SPARK-11150][SQL] Dynamic Partition Pruning
[SPARK-27592][SQL] Set the bucketed data source table SerDe correctly
[SPARK-23519][SQL] Create view should work from query with duplicate output columns
[SPARK-29011][BUILD] Update netty-all from 4.1.17-Final to 4.1.39-Final
[SPARK-28912] Fixed MatchError in getCheckpointFiles()
[SPARK-26046][SS] Add StreamingQueryManager.listListeners()
[SPARK-3137][CORE] Replace the global TorrentBroadcast lock with fine grained KeyLock (6.x)
[SPARK-28709][DSTREAMS] Fix StreamingContext leak through Streaming
[SPARK-22955][DSTREAMS] - graceful shutdown shouldn't lead to job gen…
[SPARK-28730][SPARK-28495] Introduce ANSI store assignment policy for table insertion
[SPARK-27992][SPARK-28881][PYTHON] Allow Python to join with connection thread to propagate errors
[SPARK-28025][SS] Fix FileContextBasedCheckpointFileManager leaking crc files
[SPARK-28699][CORE] Fix a corner case for aborting indeterminate stage
[SPARK-28780][ML] deprecate LinearSVCModel.setWeightCol
[SPARK-28844][SQL] Fix typo in SQLConf FILE_COMRESSION_FACTOR
[SPARK-26895][CORE] prepareSubmitEnvironment should be called within doAs for proxy users

Maintenance updates

See Databricks Runtime 6.1 maintenance updates.

System environment

Operating System: Ubuntu 16.04.6 LTS
Java: 1.8.0_232
Scala: 2.11.12
Python: 3.7.3
R: R version 3.6.1 (2019-07-05)
Delta Lake: 0.4.0