August 2023

  • Article

These features and Azure Databricks platform improvements were released in August 2023.

Note

The release date and content listed below only corresponds to actual deployment of the Azure Public Cloud in most case.

It provide the evolution history of Azure Databricks service on Azure Public Cloud for your reference that may not be suitable for Azure operated by 21Vianet.

Note

Releases are staged. Your Azure Databricks workspace might not be updated until a week or more after the initial release date.

August 31, 2023

Tables registered in Unity Catalog now appear in navigational search in the top bar. See Search for workspace objects.

Databricks CLI updated to version 0.203.3 (Public Preview)

August 30, 2023

The Databricks command-line interface (Databricks CLI) has been updated to version 0.203.3. For details, see the changelog for version 0.203.3.

Databricks JDBC driver 2.6.34

August 30, 2023

We have released version 2.6.34 of the Databricks JDBC driver (download). This release addresses a number of vulnerability issues by upgrading third-party libraries. In addition, this release fixes the following issues:

  • Thread leakage when a query fails to connect to the server.
  • Missing comments in the REMARKS column of the table metadata.
  • When a resultset closure operation returns an error, the connector does not clean up the operation handle entries from the heartbeat thread.

The release also improves error message handling by honoring SQLState and error codes from the server, and checks the server protocol version correctly.

Databricks SDK for Go updated to version 0.18.0 (Beta)

August 29, 2023

Databricks SDK for Go version 0.18.0 introduces 17 breaking API changes, adds 30 APIs, and adds about a dozen fields. For details, see the changelog for version 0.18.0.

Databricks SDK for Python updated to version 0.7.0 (Beta)

August 29, 2023

Databricks SDK for Python version 0.7.0 introduces a breaking change for the files API as well as 17 additional breaking changes. This release also adds 14 fields and adds 27 dataclasses. For details, see the changelog for version 0.7.0.

Databricks Terraform provider updated to version 1.24.1

August 29, 2023

Databricks Terraform provider version 1.24.1 includes a fix for verification of workspace reachability. For details, see the changelog for version 1.24.1.

Init scripts on DBFS end of life extended to Dec 1, 2023

August 29, 2023

Databricks previously announced that init scripts on DBFS would reach end of life on Sept 1, 2023. The end of life date has been extended to Dec 1, 2023.

Databricks Runtime 14.0 (Beta)

August 29, 2023

Databricks Runtime 14.0 and Databricks Runtime 14.0 ML are now available as Beta releases.

See Databricks Runtime 14.0 (EoS) and Databricks Runtime 14.0 for Machine Learning (EoS).

Unified navigation experience is GA

August 28, 2023

The unified Azure Databricks navigation experience is now generally available. Users can no longer revert to the classic navigation experience. For more information, see Navigate the workspace.

Databricks Terraform provider updated to version 1.24.0

August 25, 2023

Databricks Terraform provider version 1.24.0 adds account-level API support for Unity Catalog objects, adds acl_principal_id to several data sources, and more. For details, see the changelog for version 1.24.0.

Databricks Runtime for Genomics setting removed from the workspace admin settings page

August 25, 2023

Workspace admins can no longer enable or disable Databricks Runtime for Genomics from the workspace admin settings page or the Workspace configuration API. Databricks Genomics runtime is deprecated and support for Databricks Runtime for Genomics was removed on September 24, 2022.

The Databricks Genomics runtime has been deprecated. For open source equivalents, see repos for genomics-pipelines and Glow.

For more information about the Databricks Runtime deprecation policy and schedule, see Supported Databricks Runtime LTS releases.

Container Services setting removed from the workspace admin settings page

August 25, 2023

Workspace admins can no longer enable or disable Container Services from the workspace admin settings page. Container Services enables you to use customer containers on your clusters. It is enabled by default on all new workspaces. Container Services can still be enabled or disabled using the Workspace configuration API. For more information, see Enable Container Services.

Databricks CLI updated to version 0.203.2 (Public Preview)

August 24, 2023

The Databricks command-line interface (Databricks CLI) has been updated to version 0.203.2. For details, see the changelog for version 0.203.2.

Go to definition for functions and variables in Python notebooks

August 24, 2023

In Databricks Runtime 12.2 LTS and above, when a Python notebook is attached to a cluster, you can automatically navigate to the definition of a variable or function that is defined in that notebook. See Go to definition for details.

Databricks Runtime 13.3 LTS is GA

August 22, 2023

Databricks Runtime 13.3 LTS and Databricks Runtime 13.3 LTS ML are now generally available.

See Databricks Runtime 13.3 LTS and Databricks Runtime 13.3 LTS for Machine Learning.

Introducing tags with Unity Catalog

August 22, 2023

You can now apply tags to your securable objects in Unity Catalog. Tags help you organize securable objects and simplify search and discovery of your data assets. See Apply tags to Unity Catalog securable objects.

Allowlist for init scripts, JARs, and Maven coordinates on Unity Catalog shared clusters is in Public Preview

August 23, 2023

You can now use the Unity Catalog allowlist to control installation of init scripts, JARs, and Maven coordinates on compute with shared access mode running Databricks Runtime 13.3 and above. See Allowlist libraries and init scripts on shared compute.

Volumes support for init scripts and JARs is in Public Preview

August 23, 2023

You can now use Unity Catalog volumes to store init scripts and JARs on compute with assigned or shared access modes running Databricks Runtime 13.3 and above. See Cluster-scoped libraries and Install libraries from a volume.

Easier Databricks Repos .ipynb file output commits

August 21, 2023

Databricks Repos has added features to make .ipynb file output commits easier to control and see:

  • Repos can create an commit_output config file for you.
  • Notifications in the UI let you know the status of output commits.
  • From the File menu, you can control whether noteook outputs are committed.

See Control .ipynb notebook output commits.

IPYNB notebook support in Databricks Repos is GA

August 21, 2023

IPYNB notebook file support in Databricks Repos is GA. For more information about Repos, see:

Databricks SDK for Go updated to version 0.17.0 (Beta)

August 18, 2023

Databricks SDK for Go version 0.17.0 adds over 30 APIs and renames about 10 APIs. For details, see the changelog for version 0.17.0.

Databricks SDK for Python updated to version 0.6.0 (Beta)

August 18, 2023

Databricks SDK for Python version 0.6.0 adds about 20 dataclasses, adds about 10 APIs, and includes about 20 renames. For details, see the changelog for version 0.6.0.

Databricks CLI updated to version 0.203.1 (Public Preview)

August 16, 2023

The Databricks command-line interface (Databricks CLI) has been updated to version 0.203.1. For details, see the changelog for version 0.203.1.

Unified schema browser is now GA

August 16, 2023

A unified way to navigate your data is now available for notebooks, SQL Editor, and Catalog Explorer, including the following new capabilities:

  • Condensed version of the tree.
  • Display of Unity Catalog object count at the schema level when filtering.
  • Right-click menu to view actions.

Serverless subnet changes may affect you if you encode IDs in a script or template

August 15, 2023

If you use an Azure Storage firewall, until the end of August, some subnet IDs are changing. This does not affect most customers because Azure automatically migrates subnet IDs in existing storage firewall rules. This migration does not affect existing workloads.

If you encode the list of serverless subnets in a template or script that updates your Azure storage firewall rules, you must get the latest list of subnet IDs before you attempt to update the rules. For the latest IDs for each region, see Azure Databricks regions.

Databricks SDK for Go updated to version 0.16.0 (Beta)

August 14, 2023

Databricks SDK for Go version 0.16.0 adds about a dozen fields, removes about a half-dozen fields, and changes the signatures of about a half-dozen APIs. For details, see the changelog for version 0.16.0.

Partner Connect supports Census

August 14, 2023

You can now use Partner Connect to connect your Azure Databricks workspace to Census. For more information, see Connect to Census.

Azure Key Vault Managed HSM keys now supported for customer-managed keys

August 14, 2023

Some Azure Databricks services and data support adding a customer-managed key to help protect and control access to encrypted data. Azure Databricks has three customer-managed key features that involve different types of data and locations. All three of these features now support customer-managed keys from Azure Key Vault Managed HSM (Hardware Security Modules) alongside Azure Key Vault vaults. See Customer-managed keys for encryption.

Databricks SDK for Python updated to version 0.5.0 (Beta)

August 11, 2023

Databricks SDK for Python version 0.5.0 changes the signatures of a half-dozen APIs, adds a dozen fields, removes a half-dozen fields, and adds a few dataclasses. For details, see the changelog for version 0.5.0.

Programmatic write support for workspace files

August 11, 2023

Clusters using Databricks Runtime 11.2 and above support writing to workspace files. You can programmatically create, update, and delete files and directories within your workspace.

Databricks CLI updated to version 0.203.0 (Public Preview)

August 10, 2023

The Databricks command-line interface (Databricks CLI) version 0.203.0 improves the auth login command, and more. For details, see the changelog for version 0.203.0.

Databricks Terraform provider updated to version 1.23.0

August 9, 2023

Databricks Terraform provider version 1.23.0 extends RunJobTask with additional supported fields in databricks_job, adds exporter support for databricks_model_serving, and more. For details, see the changelog for version 1.23.0.

Groups can now be renamed

August 8, 2023

You can now rename existing account groups and workspace-local groups. See Change the name of a group.

Databricks SDK for Go updated to version 0.15.0 (Beta)

August 7, 2023

Databricks SDK for Go version 0.15.0 refactors permissions-related APIs to make them more understandable and easier to use. For details, see the changelog for version 0.15.0.

Databricks SDK for Python updated to version 0.4.0 (Beta)

August 7, 2023

Databricks SDK for Python version 0.4.0 refactors permissions-related APIs to make them more understandable and easier to use. For details, see the changelog for version 0.4.0.

Third-party iFraming prevention configuration setting was removed

August 7, 2023

Azure Databricks reduces the risk of third-party domains from iFraming Azure Databricks by adding appropriate HTTP headers. Third-party iFraming prevention is still enabled by default but Azure Databricks removed the workspace admin setting to disable it. If you previously disabled third-party iFraming for a workspace, the feature continues to be disabled for that workspace. To re-enable third-party iFraming, contact your Azure Databricks account team.

Databricks extension for Visual Studio Code updated to version 1.1.1

August 4, 2023

The Databricks extension for Visual Studio Code version 1.1.1 provides several minor fixes, for instance correctly handling the HOME environment variable on Windows, and more. For details, see the changelog for version 1.1.1.

LangChain available in 13.1 and above

August 4, 2023

LangChain is available as an experimental MLflow flavor which allows LangChain customers to leverage the robust tools and experiment tracking capabilities of MLflow directly from the Azure Databricks environment.

LangChain is a software framework designed to help create applications that utilize large language models (LLMs) and combine them with external data to bring more training context for LLMs.

Databricks Runtime for Machine Learning includes langchain in Databricks Runtime 13.1 ML and above.

Feature Engineering in Unity Catalog is Public Preview

August 3, 2023

With Feature Engineering in Unity Catalog, Unity Catalog becomes your feature store. You can use any Delta table with a primary key as a feature table for model training or inference. Unity Catalog provides feature discovery and governance.

Improved error handling for repeated continuous job failures

August 3, 2023

Continuous jobs exceeding a threshold for consecutive failures now use an exponential backoff scheme to efficiently retry job runs and allow jobs to recover from transient issues. Previously, continuous jobs exceeding the failure threshold were paused and not retried.

Share schemas using Delta Sharing (Public Preview)

August 2, 2023

You can now use Delta Sharing to share an entire schema, giving the recipient access to all of the tables and views in the schema at the moment you share it, along with any tables and views that are added to the schema in the future. Adding schemas to a share using SQL commands requires a SQL warehouse or a cluster running Databricks Runtime 13.2 or above. Doing the same using Catalog Explorer has no compute requirements. See Add schemas to a share.

Run tasks conditionally in your Azure Databricks jobs

August 2, 2023

You can use the new Run if condition to run a job task based on the status of the task's dependencies, providing more flexibility when you implement a workflow with an Azure Databricks job. For example, you can use the Run if condition to run a task when some or all of the task's dependencies have failed. See Configure task dependencies.

Databricks Terraform provider updated to version 1.22.0

August 1, 2023

Databricks Terraform provider version 1.22.0 adds a databricks_access_control_rule_set resource, adds databricks_metastore and databricks_metastore data sources, and more. For details, see the changelog for version 1.22.0.

Enhanced security and compliance features

August 1, 2023

You can enable the compliance security profile , which provides additional monitoring, a hardened compute image, and other features, on your workspaces. The compliance security profile is required to use Azure Databricks to process data that is regulated by PCI-DSS , but you can choose to enable the profile if you are not processing such data. You are solely responsible for ensuring your own compliance with all applicable laws and regulations.

Alternatively, you can separately enable enhanced security monitoring , which provides additional monitoring and a hardened compute image.

Both features are available as part of the new add-on Enhanced Security and Compliance. If you enable either of these features on a workspace, you will be charged for the new add-on as described on the pricing page. Both features require the Premium pricing tier.

Lakehouse Monitoring is Public Preview

August 1, 2023

Databricks Lakehouse Monitoring integrates data and model quality monitoring into the Databricks platform. It provides out of the box quality metrics such as summary statistics, changes in distribution over time, and model performance including fairness and bias. Lakehouse Monitoring automatically generates a summary dashboard to help you visualize data quality over time. You can also set up alerts based on the data.

Databricks Runtime 13.3 LTS (Beta)

August 1, 2023

Databricks Runtime 13.3 LTS and Databricks Runtime 13.3 LTS ML are now available as Beta releases.

See Databricks Runtime 13.3 LTS and Databricks Runtime 13.3 LTS for Machine Learning.

New Git operations are generally available: Merge branches, rebase and pull with conflict resolution

August 1, 2023

The following Git operations are now generally available in Databricks Repos:

  • Merge and rebase: You can now Git merge or rebase two branches directly in Azure Databricks. This simplifies the process of combining branches and provides the ability to resolve conflicts that arise during the operation.
  • Pull with conflict resolution: You can now use Databricks Repos to manage and resolve conflicts that happen when you pull from a remote Git repo using Azure Databricks.

With these powerful new conflict resolution tools, collaborating on code using Databricks Repos becomes even more streamlined and efficient.