Databricks Connect release notes

This page lists releases and maintenance updates issued for Databricks Connect.

The Databricks Runtime version of the cluster must be greater than or equal to the Databricks Connect version. Databricks recommends using the latest version to receive any bug fixes and security updates. For system environment information for Databricks Runtime releases, see Databricks Runtime release notes versions and compatibility.

December 3, 2024

The Databricks Connect client is updated to support Databricks Runtime 16.0.

November 29, 2024

This release of Databricks Connect for Python supports Databricks Runtime 16.0.

September 3, 2024

This release of Databricks Connect for Scala supports Databricks Runtime 15.4 LTS.

November 29, 2024

Bug fixes:

• Fixed an issue where df.write.parquet did not correctly recognize the gzip compression option.

October 31, 2024

Minor fixes and internal improvements.

September 13, 2024

Bug fixes:

• Fixed a bug where error messages were not complete. Error messages now include the stack trace from Spark if it is relevant.

September 2, 2024

Features and improvements:

• validateSession() and the databricks-connect test command now fail when the Databricks Connect version is incompatible with the Databricks serverless compute version. For information about Databricks Connect serverless support, see Configure a connection to serverless compute.

August 27, 2024

This release of Databricks Connect for Python supports Databricks Runtime 15.4 LTS.

September 3, 2024

Minor internal fixes and improvements.

July 22, 2024

Features and improvements:

• Supported Dataset.observe for getting stats of Columns of a DataFrame. See SPARK-47545.
• Added a new function timestamp_add for adding the specified number of units to the given timestamp. See SPARK-48369.
• Added a new function timestamp_diff for getting the difference between the timestamps in the specified units by truncating the fraction. See SPARK-48272.
• Added a new function try_mod for computing the remainder of the division of two numbers, which results in null on division by zero. See SPARK-48954.
• Added a new function try_parse_json, which attempts to parse a JSON string and construct a Variant value. If the attempt fails, the result is null. See SPARK-47922.
• Changed the default value for spark.sql.legacy.timeParserPolicy from EXCEPTION to CORRECTED. Instead of raising an INCONSISTENT_BEHAVIOR_CROSS_VERSION error, CANNOT_PARSE_TIMESTAMP will be raised if ANSI mode is enabled. NULL will be returned if ANSI mode is disabled. See SPARK-47719.

October 31, 2024

Minor fixes and internal improvements.

September 13, 2024

Bug fixes:

• Fixed a bug where error messages were not complete. Error messages now include the stack trace from Spark if it is relevant.

September 2, 2024

Features and improvements:

• validateSession() and the databricks-connect test command now fail when the Databricks Connect version is incompatible with the Databricks serverless compute version. For information about Databricks Connect serverless support, see Configure a connection to serverless compute.

August 27, 2024

Features and improvements:

• A warning now occurs when the Databricks Connect version is incompatible with the serverless version .
• Improved error message when the local Python version does not match the serverless version .

July 18, 2024

Bug fixes:

• Serverless Spark Connect sessions expire after 10 minutes of inactivity. Databricks Connect now automatically closes expired sessions on the client side. Subsequent calls to DatabricksSession.builder.getOrCreate() will return a new active connect session.

July 15, 2024

Features and improvements:

• Added a new function timestamp_add for adding the specified number of units to the given timestamp. See SPARK-48369.
• Added a new function timestamp_diff for getting the difference between the timestamps in the specified units by truncating the fraction. See SPARK-48272.
• Added a new function try_parse_json, which attempts to parse a JSON string and construct a Variant value. If the attempt fails, the result is null. See SPARK-47922.
• Changed the default value for spark.sql.legacy.timeParserPolicy from EXCEPTION to CORRECTED. Instead of raising an INCONSISTENT_BEHAVIOR_CROSS_VERSION error, CANNOT_PARSE_TIMESTAMP will be raised if ANSI mode is enabled. NULL will be returned if ANSI mode is disabled. See SPARK-47719.

September 3, 2024

Minor internal fixes and improvements.

July 22, 2024

Features and improvements:

• The split function now supports column type. See SPARK-47845.
• Added new functions is_variant_null, variant_get, try_variant_get, schema_of_variant, and schema_of_variant_agg to improve the usability of the Variant type. See SPARK-47890.
• Added new functions collate and collation to access collation capabilities. See SPARK-47395.
• Extensions are now allowed to log extended information in the explain plan. See SPARK-47289.

September 26, 2024

Minor fixes and internal improvements.

September 13, 2024

Bug fixes:

• Fixed a bug where error messages were not complete. Error messages now include the stack trace from Spark if it is relevant.

September 2, 2024

Features and improvements:

• validateSession() and the databricks-connect test command now fail when the Databricks Connect version is incompatible with the Databricks serverless compute version. For information about Databricks Connect serverless support, see Configure a connection to serverless compute.

August 27, 2024

Features and improvements:

• A warning now occurs when the Databricks Connect version is incompatible with the serverless version .

• Improved error message when the local Python version does not match the serverless version .

July 18, 2024

Bug fixes:

• Serverless spark connect sessions expire after 10 minutes of inactivity. Databricks Connect now automatically closes expired sessions on the client side. Subsequent calls to DatabricksSession.builder.getOrCreate() will return a new active connect session.

July 15, 2024

Features and improvements:

• The split function now supports column type. See SPARK-47845.
• Added new functions is_variant_null, variant_get, try_variant_get, schema_of_variant, and schema_of_variant_agg to improve the usability of the Variant type. See SPARK-47890.
• Added new functions collate and collation to access collation capabilities. See SPARK-47395.
• Extensions are now allowed to log extended information in the explain plan. See SPARK-47289.

September 3, 2024

Minor internal fixes and improvements.

July 22, 2024

Features and improvements:

• Added a new function try_cast, which attempts to cast to a different data type. If the attempt fails, the result is null. See SPARK-47669.
• Added a new function monthname that returns the three-letter abbreviated month name from a given date or timestamp. See SPARK-46515.

Bug fixes:

• Session stop() API is idempotent now and will not fail if the session is already closed by the server. See SPARK-48810.
• The bloomFilter function now throws an AnalysisException for invalid input. See SPARK-46787.

September 13, 2024

Bug fixes:

• Fixed a bug where error messages were not complete. Error messages now include the stack trace from Spark if it is relevant.

September 2, 2024

Features and improvements:

• Improved error message when the local Python version does not match the Python version of the serverless release .

July 18, 2024

Bug fixes:

• Serverless spark connect sessions expire after 10 minutes of inactivity. Databricks Connect now automatically closes expired sessions on the client side. Subsequent calls to DatabricksSession.builder.getOrCreate() will return a new active connect session.

• Supported Python versions are now limited to 3.10 and 3.11, because the distutils dependency has been removed in Python 3.12.

July 15, 2024

Features and improvements:

• Databricks Connect for Python now supports connecting to serverless compute. This feature is available in Public Preview. See Configure a connection to serverless compute.
• Added a new function try_cast, which attempts to cast to a different data type. If the attempt fails, the result is null. See SPARK-47669.
• Added a new function monthname that returns the three-letter abbreviated month name from a given date or timestamp. See SPARK-46515.

Bug fixes:

• Session stop() API is idempotent now and will not fail if the session is already closed by the server. See SPARK-48810.
• Supported numpy versions are now limited to 1.15<=numpy<2.0 because PySpark 3.5.0 is incompatible with numpy 2.0.

November 1, 2024

Minor fixes and internal improvements.

Bug fixes:

• This release includes a fix for a bug that causes cluster connect requests to fail immediately if the first connection attempt fails. With this fix, the request retries as expected instead of failing immediately.

June 24, 2024

Features and improvements:

• A number of internal dependencies that were vendored as part of the Databricks Connect Maven package were not shaded. In this release, all internal dependencies that are not part of the Databricks Connect public API surface are shaded.

Bug fixes:

• Fixed the issue where a serial version UID mismatch error occurred when executing queries with UDFs.
• json4s and log4j have been declared as dependencies instead of being vendored. This was required as part of the fix for serial version UID.

November 29, 2024

No changes since the last version.

October 29, 2024

Minor fixes and internal improvements.

September 11, 2024

Bug fixes:

• Calling spark.stop() on an already stopped or expired session will no longer fail.
• Fixed a bug where error messages were not complete. Error messages now include the stack trace from Spark if it is relevant.

May 8, 2024

Bug fixes:

• Automatically re-try plan execution in some cases of SESSION_NOT_FOUND error from the service.
• Subsequent calls to SparkSession.builder.getOrCreate() after DatabricksSession.builder.getOrCreate() will return the active default connect session.

February 22, 2024

Minor fixes and internal improvements.

February 1, 2024

Features and improvements:

• Databricks Connect now ships with the databricks-connect binary. Users can now validate their Python environment and the connection to the Azure Databricks cluster by running: databricks-connect test.

Bug fixes:

• Limit the number of retries when Spark Connect requests fails to the service.
• spark.table() now accepts Delta table time travel syntax.
• Aggregate functions run on top of df.melt() now work. Seehttps://issues.apache.org/jira/browse/SPARK-46484.
• DataFrame joins using queries that contain common table expressions (CTE) now work. Seehttps://issues.apache.org/jira/browse/SPARK-46337.
• df.withColumnsRenamed() now retains original dictionary ordering. Seehttps://issues.apache.org/jira/browse/SPARK-46261.
• df.withColumn() now accepts an argument of Column type. Seehttps://issues.apache.org/jira/browse/SPARK-46028.
• The behavior of df.column() in Spark Connect is now more aligned with the behavior in classic Apache Spark. Seehttps://issues.apache.org/jira/browse/SPARK-45509.
• Referencing the join column in df.drop() resolves correctly, instead of generating an "ambiguous column" error code. Seehttps://issues.apache.org/jira/browse/SPARK-45770.
• An object of Struct type can now be converted to XML using the to_xml() method. Seehttps://issues.apache.org/jira/browse/SPARK-44790.
• Improve the usability of exceptions raised by Spark Connect (for example, SparkConnectGrpcException). Seehttps://issues.apache.org/jira/browse/SPARK-45808.

February 1, 2024

Beginning with Databricks Runtime 14.3 LTS, Databricks Connect is now also generally available for Scala.

Features and improvements:

• The dependencies on Apache Spark packages has been removed and replaced by bundling these packages and their transitive dependencies into the Databricks Connect JAR, improving the alignment of Databricks Connect releases with the releases of the Databricks Runtime.

January 12, 2024

Bug fixes:

• Previously missed pyspark module (pyspark.sql.connect.protobuf) is now included. The to_protobuf() method is now available.

December 7, 2023

Features and improvements:

• The SparkSession stop() method now closes the session on the server, interrupting release running operations and releasing resources.
• Improved error handling for addArtifact API and for cluster start up.

December 6, 2023

Features and improvements:

• The SparkSession stop() method now closes the session on the server, interrupting release running operations and releasing resources.
• Improved error handling: Previous generic GRPC and Spark Connect server exceptions now come with more user-friendly messages.
• Improved retry logic when waiting for cluster start up.

January 12, 2024

Bug fixes:

• Previously missed pyspark module (pyspark.sql.connect.protobuf) is now included. The to_protobuf() method is now available.

October 23, 2023

Minor fixes and improvements.

November 7, 2023

Features and improvements:

• User-defined functions (UDFs) can now be called by using the new addCompiledArtifacts() API. See User-defined functions in Databricks Connect for Scala.
• Logging improvements.

January 12, 2024

Bug fixes:

• Previously missed pyspark module (pyspark.sql.connect.protobuf) is now included. The to_protobuf() method is now available.

September 29, 2023

Bug fixes:

• Skip starting PyTorch distribution server.

Improvements:

• The AddArtifact API upload speed has been improved.

September 22, 2023

Improvements:

• The Databricks license file is now included in the JAR.

September 20, 2023

Features and improvements:

• Queries can now run for longer than 1 hour.
• Databricks Connect can now reconnect and resume from network interruptions while waiting for query results.
• Ability to interrupt running queries.
• Support for all Structured Streaming APIs, including foreachBatch.

October 12, 2023

Bug fixes:

• Improved retry logic for network problems.

September 21, 2023

Bug fixes:

• We fixed a bug where createDataFrame() failed when a large dataset was used.

September 18, 2023

Features and improvements:

• This release improves support for mypy type hints. mypy provides optional static typing improving the IDE experience and error checking when working on Python code.

August 28, 2023

Features and improvements:

• The Spark Connect Service can now be disabled by setting the Spark conf on the cluster spark.databricks.service.server.enabled false.
• The DatabricksSession class is now available and usable in a Databricks notebook. Calling DatabricksSession.builder.getOrCreate() is equivalent to using the default spark object.
• It is now possible to upload local data larger than 128 MB via createDataFrame().

Bug fixes:

• The DatabricksSession.getOrCreate() API did not work correctly when configuring multiple sessions.

August 22, 2023

Bug fixes:

• Fix Databricks Connect client not waiting for clusters to start up.

July 6, 2023

Features and improvements:

• You can now pass in custom headers as part of the Spark Connect request.
• An Azure Databricks SDK configuration profile can now be directly specified in DatabricksSession by using DatabricksSession.profile().
• The built-in PySpark REPL now recognizes connection parameters (such as the host and cluster ID) from environment variables and the .databrickscfg file. The behavior is similar to the default behaviorwhen a session is created from DatabricksSession.

Bug fixes:

• DatabricksSession was able to keep only one connection at any time. Now you can use DatabricksSession to set up multiple connections to different workspaces or the same workspace but with different credentials.
• A bug that was affecting auto-completion in Visual Studio Code has been fixed.

June 2, 2023

Databricks Connect "v2" is now generally available for Python.

• Databricks SDK integration which enables authentication with OAuth and Azure Active Director (in Public Preview in the Python SDK).
• Support for all pandas API on Spark.
• Streaming support except for a few items in structured streaming, such as foreachBatch and streaming event listener.
• Full Delta support.

Beginning with Databricks Runtime 13.0, Databricks Connect builds upon Spark Connect, enabling developers to use the power of Databricks and their IDE of choice to interactively develop and debug their code in any Databricks workspace.

From this version, Databricks Connect can connect to Unity Catalog's single user and shared cluster access modes.

October 14, 2024

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

August 20, 2024

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

May 30, 2024

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

April 25, 2024

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

April 8, 2024

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

March 14, 2024

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

February 27, 2024

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

February 14, 2024

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

February 2, 2024

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

January 24, 2024

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

December 13, 2023

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

December 4, 2023

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

November 7, 2023

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

October 30, 2023

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

October 16, 2023

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

September 28, 2023

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

September 11, 2023

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

August 28, 2023

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

August 18, 2023

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

August 7, 2023

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

July 19, 2023

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

July 5, 2023

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

June 15, 2023

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

June 12, 2023

• Databricks Connect client update to support Databricks Runtime 12.2 LTS maintenance release.

May 16, 2023

• Initial Databricks Connect client release to support Databricks Runtime 12.2 LTS.
• The Python and Java integrations are functioning correctly.

October 14, 2024

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

August 20, 2024

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

May 30, 2024

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

April 25, 2024

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

April 8, 2024

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

March 14, 2024

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

February 27, 2024

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

February 14, 2024

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

February 2, 2024

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

January 24, 2024

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

December 13, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

December 4, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

November 7, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

October 30, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

October 16, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

September 28, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

September 11, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

August 28, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

August 18, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

August 7, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

July 19, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

July 5, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

June 15, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

June 12, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

May 16, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

May 1, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

April 11, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release.

March 8, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release of March 8, 2023.
• [ES-589567] Fixes java.lang.NoSuchMethodError error when enabling spark.databricks.io.cache.enabled

February 20, 2023

• Databricks Connect client update to support Databricks Runtime 11.3 LTS maintenance release of February 20, 2023.
• [ES-573062] Fixes java.lang.ClassCastException when assigning fields to org.apache.spark.sql.catalyst.json.JSONOptions

December 8, 2022

• Initial Databricks Connect client release to support Databricks Runtime 11.3 LTS.
• There is a known issue in the R integration (some Spark plans will fail to execute).

October 14, 2024

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

August 20, 2024

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

May 30, 2024

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

April 25, 2024

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

April 8, 2024

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

March 14, 2024

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

February 27, 2024

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

February 14, 2024

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

February 2, 2024

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

January 24, 2024

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

December 13, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

December 4, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

November 7, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

October 30, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

October 16, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

September 28, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

September 11, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

August 28, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

August 18, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

August 7, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

July 19, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

July 5, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

June 15, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

June 12, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

May 16, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.
• [ES-573304] Fix issues relating to resolving tables from file paths (i.e. parquet."/some/path")

May 1, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

April 11, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release.

March 8, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release of March 8, 2023.

February 20, 2023

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release of February 20, 2023.
• [ES-573062] Fixes java.lang.ClassCastException when assigning fields to org.apache.spark.sql.catalyst.json.JSONOptions

December 1, 2022

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release of December 1, 2022.

September 12, 2022

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release of September 6, 2022.

August 25, 2022

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release of August 25, 2022.

August 12, 2022

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release of August 12, 2022.

July 29, 2022

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release of July 29, 2022.

July 21, 2022

• Databricks Connect client update to support Databricks Runtime 10.4 LTS maintenance release of July 19, 2022.

June 16, 2022

Databricks Connect 10.4 is now GA.

March 29, 2022

• Initial Databricks Connect client release to support Databricks Runtime 10.4 LTS.
• The Python and Java integrations are functioning correctly
• There is a known issue in the R integration (some Spark plans will fail to execute)

October 14, 2024

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

August 20, 2024

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

May 30, 2024

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

April 25, 2024

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

April 8, 2024

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

March 14, 2024

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

February 27, 2024

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

February 14, 2024

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

February 2, 2024

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

January 24, 2024

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

December 13, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

December 4, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

November 7, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

October 30, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

October 16, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

September 28, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

September 11, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

August 28, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

August 18, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

August 7, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

July 19, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

July 5, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

June 15, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

June 12, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

May 16, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

May 1, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

April 11, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release.

March 8, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of March 8, 2023.

February 20, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of February 20, 2023.
• [ES-573062] Fixes java.lang.ClassCastException when assigning fields to org.apache.spark.sql.catalyst.json.JSONOptions

December 1, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of December 1, 2022.

September 12, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of September 6, 2022.

August 25, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of August 25, 2022.

August 12, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of August 12, 2022.

July 29, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of July 29, 2022.

July 15, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of July 14, 2022.

June 16, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of June 12, 2022.

June 7, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of June 2, 2022.

May 20, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of May 20, 2022.

May 5, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of May 5, 2022.

April 21, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of April 21, 2022.

April 7, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of April 7, 2022.

March 29, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of March 29, 2022.

March 10, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of March 10, 2022.
• Fix ClassNotFoundException error message

February 23, 2022

• Security enhancements

February 8, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of February 8, 2022.

January 26, 2022

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of January 26, 2022.

January 19, 2022

• [ES-216554] Fixed an issue where cluster-attached libraries were not loaded correctly server-side in some cases

December 7, 2021

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of December 7, 2021.

November 16, 2021

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of November 16, 2021.

November 4, 2021

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of November 4, 2021.

October 20, 2021

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of October 20, 2021.

October 5, 2021

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of October 5, 2021.

September 23, 2021

Databricks Connect 9.1 is now GA.

September 23, 2021

• Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of September 23, 2021.

September 15, 2021

• Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of September 15, 2021.

September 8, 2021

• Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of September 8, 2021.

August 25, 2021

• Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of August 25, 2021.

August 11, 2021

• Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of August 11, 2021.

July 30, 2021

• Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of July 30, 2021.

July 14, 2021

• Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of July 14, 2021.

June 29, 2021

• Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of June 29, 2021.

June 15, 2021

• Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of June 15, 2021.

June 7, 2021

• [ES-102916] Support no_proxy environment variable in DBConnnect client. Set it to a comma-separated list of fully qualified domain names for which the proxy set in https_proxy or http_proxy environment variable should not be used.

May 26, 2021

• Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of May 26, 2021.

April 30, 2021

• Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of April 30, 2021.

April 27, 2021

• Databricks Connect client update to support Databricks Runtime 8.1 maintenance release of April 27, 2021.

April 7, 2021

• Improved robustness of Databricks Connect startup process.
• The previous mechanism for accessing secrets over Databricks Connect (using a dkea token) has been deprecated. Contact support to enable the new access mechanism for your workspace.
• Fixed a bug that occasionally caused Spark configurations to fail to be picked up correctly.
• [ES-77737] Fixed a bug preventing proper installation of cluster-wide libraries on Spark executors.

March 22, 2021

Databricks Connect 8.1 is now GA.

• Improved support of Catalog API V2 commands
• Improved support of SQL expressions in ANSI mode

September 11, 2023

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release.

August 28, 2023

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release.

August 18, 2023

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release.

August 7, 2023

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release.

July 19, 2023

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release.

July 5, 2023

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release.

June 15, 2023

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release.

June 12, 2023

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release.

May 16, 2023

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release.

May 1, 2023

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release.

April 11, 2023

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release.

March 8, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of March 8, 2023.

February 20, 2023

• Databricks Connect client update to support Databricks Runtime 9.1 LTS maintenance release of February 20, 2023.
• [ES-573062] Fixes java.lang.ClassCastException when assigning fields to org.apache.spark.sql.catalyst.json.JSONOptions

December 1, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of December 1, 2022.

September 12, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of September 6, 2022.

August 25, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of August 25, 2022.

August 12, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of August 12, 2022.

July 29, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of July 29, 2022.

July 15, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of July 14, 2022.

June 16, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of June 12, 2022.

June 7, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of June 2, 2022.

May 20, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of May 20, 2022.

May 5, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of May 5, 2022.

April 21, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of April 21, 2022.

April 7, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of April 7, 2022.

March 29, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of March 29, 2022.

March 10, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of March 10, 2022.

February 23, 2022

• Security enhancements

February 8, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of February 8, 2022.

January 26, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of January 26, 2022.

January 19, 2022

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of January 19, 2022.

December 7, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of December 7, 2021.

November 16, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of November 16, 2021.

November 4, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of November 4, 2021.

October 20, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of October 20, 2021.

October 5, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of October 5, 2021.

September 23, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of September 23, 2021.

September 15, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of September 15, 2021.

September 8, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of September 8, 2021.

August 25, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of August 25, 2021.

August 11, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of August 11, 2021.

July 30, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of July 30, 2021.

July 14, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of July 14, 2021.

June 29, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of June 29, 2021.

June 15, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of June 15, 2021.

June 7, 2021

• [ES-102916] Support no_proxy environment variable in DBConnnect client. Set it to a comma-separated list of fully qualified domain names for which the proxy set in https_proxy or http_proxy environment variable should not be used.

May 26, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of May 26, 2021.

April 30, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of April 30, 2021.

April 27, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of April 27, 2021.

April 7, 2021

• Improved robustness of Databricks Connect startup process.
• The previous mechanism for accessing secrets over Databricks Connect (using a dkea token) has been deprecated. Contact support to enable the new access mechanism for your workspace.
• Fixed a bug that occasionally caused Spark configurations to fail to be picked up correctly.
• [ES-77737] Fixed a bug preventing proper installation of cluster-wide libraries on Spark executors.

March 24, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of March 24, 2021

March 9, 2021

• [ES-69946] Fix path separator on Windows for databricks-connect get-jar-dir

• [ES-68776] Update Azure Blob File System client

• Add support for FlatMapCoGroupsInPandas

• Make initial connection process during cluster startup more robust

  Note

  The Databricks Runtime cluster must be running the latest maintenance release, and the Databricks Connect client must be updated to version 7.3.10 to continue to use the Azure Blob File System and mapPartitions and mapElements RDD operations with Databricks Connect. A restart of the Databricks Runtime cluster is required to update it to the latest maintenance release.

February 23, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of February 23, 2021

February 4, 2021

• [ES-58153] Fix Analysis Exception when creating tables or views based on a query with aggregations.

January 20, 2021

• Databricks Connect client update to support Databricks Runtime 7.3 LTS maintenance release of January 20, 2021

January 12, 2021

• [ES-59054] Fix Delta Optimize Table command through Databricks Connect
• [ES-53497] Fix Databricks Connect on Databricks Container Services (DCS)

December 9, 2020

Databricks Connect 7.3 is now GA.

• [ES-49865] Prevent 500 Internal Server Errors due to automatic synchronization of dependent libraries when using Databricks Connect with SBT.

November 24, 2020

• [ES-44382] Set spark.databricks.service.allowTerminatedClusterStart=false to prevent Databricks Connect from starting a terminated Databricks Runtime cluster.
• Fix an error where client could not synchronize state with server correctly if INFO logging level was disabled.

November 3, 2020

Initial Databricks Connect release for Databricks Runtime 7.3. This release includes:

• Support for Microsoft Entra ID credential passthrough. You can now use credential passthrough with Standard clusters using Databricks Connect. This enables you to authenticate automatically to Azure Data Lake Storage Gen2 from Databricks Connect using the same Microsoft Entra ID identity that you use to authenticate to Azure Databricks.

• Support for Delta Lake time travel.
• Easy transition between Databricks Connect clients and Databricks Runtime jobs or notebooks when using DBUtils.

Databricks Runtime version validation is disabled in this beta release. Make sure to connect this client to a 7.3 Databricks Runtime cluster.

February 4, 2021

• Databricks Connect client update to support Databricks Runtime 7.1 maintenance release of February 4, 2021

January 20, 2021

• Databricks Connect client update to support Databricks Runtime 7.1 maintenance release of January 20, 2021

January 12, 2021

• [ES-59054] Fix Delta Optimize Table command through Databricks Connect
• [ES-53497] Fix Databricks Connect on Databricks Container Services (DCS)

December 9, 2020

• [ES-49865] Prevent 500 Internal Server Errors due to automatic synchronization of dependent libraries when using Databricks Connect with SBT.

November 24, 2020

• Prevent status polls from restarting the cluster.
• [ES-43656] Fix serialization error related to Azure Data Lake Storage Gen2 support.
• [ES-30594] Allow larger HTTP headers and enlarge buffers in Databricks Connect.
• [ES-34186] Avoid out of memory exception related to logging.
• [ES-39493] Prevent recursive load error in databricks-connect test.
• [ES-44382] Set spark.databricks.service.allowTerminatedClusterStart=false to prevent Databricks Connect from starting a Databricks Runtime cluster that is terminated.
• Fix an error in which client could not synchronize state with server correctly if INFO logging level was disabled.
• Minor fixes.

September 23, 2020

• [ES-32536] Fix an issue with serialization of nested timezone-aware expressions.
• [ES-33705] Support Azure Data Lake Storage Gen2 filesystem in DButils.

August 13, 2020

Initial Databricks Connect release for Databricks Runtime 7.1.

• Support new Spark 3.0 DataSource V2 APIs.
• Support using multiple SparkSessions in a single Databricks Connect application.
• Optimize Databricks Connect state synchronization between client and server. Avoid excessive polling for status updates when the server is idle.

February 23, 2022

• Security enhancements

April 7, 2021

• The previous mechanism for accessing secrets over Databricks Connect (using a dkea token) has been deprecated. Contact support to enable the new access mechanism for your workspace.

March 24, 2021

• Databricks Connect client update to support Databricks Runtime 6.4 maintenance release of March 24, 2021

March 15, 2021

• New version has been released, but there are no changes compared to Databricks Connect 6.4.25

March 9, 2021

• [ES-69946] Fix path separator on Windows for databricks-connect get-jar-dir
• Add support for FlatMapCoGroupsInPandas

February 23, 2021

• Databricks Connect client update to support Databricks Runtime 6.4 maintenance release of February 23, 2021

February 4, 2021

• Databricks Connect client update to support Databricks Runtime 6.4 maintenance release of February 4, 2021

January 12, 2021

• [ES-59054] Fix Delta Optimize Table command through Databricks Connect
• [ES-53497] Fix Databricks Connect on Databricks Container Services (DCS)

December 9, 2020

• [ES-25890] Fix an error where in some cases DataFrame cache() would have no effect and the query would be executed again.
• [ES-26196] Support using NATURAL and USING syntax in joins.
• [ES-33475][ES-34186] Avoid out of memory exception related to logging.
• [ES-39493] Prevent recursive load error in databricks-connect test.
• [ES-43656] Fix serialization error related to Azure Data Lake Storage Gen2 support.
• [ES-30594] Allow larger HTTP headers and enlarge buffers in Databricks Connect.
• Minor fixes

November 5, 2020

• [ES-47216] Fix version check compatibility issue with Databricks Runtime 6.4 November 3, 2020 maintenance update.

April 8, 2020

Minor fixes and updates.

March 7, 2020

Initial Databricks Connect release for Databricks Runtime 6.4.