September 2023

  • Article

These features and Azure Databricks platform improvements were released in September 2023.

Note

The release date and content listed below only corresponds to actual deployment of the Azure Public Cloud in most case.

It provide the evolution history of Azure Databricks service on Azure Public Cloud for your reference that may not be suitable for Azure operated by 21Vianet.

Note

Releases are staged. Your Azure Databricks workspace might not be updated until a week or more after the initial release date.

Lakehouse Federation available on single user clusters

September 29, 2023

Lakehouse Federation is now available on single user clusters using Databricks Runtime 13.1 and above. Only the connection owner can run queries on federated catalogs. See What is Lakehouse Federation?.

Delegate the ability to create a storage credential in Unity Catalog

September 29, 2023

You can now grant users permissions to create a storage credential in a Unity Catalog metastore using the new CREATE STORAGE CREDENTIAL privilege. See CREATE STORAGE CREDENTIAL.

Databricks Runtime 14.1 (Beta)

September 28, 2023

Databricks Runtime 14.1 and Databricks Runtime 14.1 ML are now available as Beta releases.

See Databricks Runtime 14.1 and Databricks Runtime 14.1 for Machine Learning.

Prevent runs of a job from being skipped when you reach concurrency limits

September 27, 2023 - October 2, 2023

To prevent runs of an Azure Databricks job from being skipped because of concurrency limits, you can now enable queueing on the job. When queueing is enabled, if a concurrency limit is reached, the job is placed in a queue until capacity is available to run the job. See Enable queueing of job runs.

Databricks Terraform provider updated to version 1.27.0

September 27, 2023

Databricks Terraform provider version 1.27.0 adds owner and force_destroy to databricks_metastore_data_access, includes several additional improvements and bug fixes, and more. For details, see the changelog for version 1.27.0.

Databricks CLI updated to version 0.206.0 (Public Preview)

September 27, 2023

The Databricks command-line interface (Databricks CLI) has been updated to version 0.206.0. This release enables target overrides for pipeline clusters in Databricks Asset Bundles, improves the bundle template initialization experience, and more. For details, see the changelog for version 0.206.0.

Databricks SDK for Go updated to version 0.21.0 (Beta)

September 27, 2023

Databricks SDK for Go version 0.21.0 introduces over a dozen breaking API changes, adds over a dozen APIs, and adds a dozen fields. For details, see the changelog for version 0.21.0.

Databricks ODBC driver 2.7.5

September 23, 2023

We have released version 2.7.5 of the Databricks ODBC driver (download). This release adds the following new features and enhancements:

  • OAuth 2.0 client authentication enhancement and OIDC endpoint discovery support.
  • Server-side encryption with customer provided keys.
  • Timedate function.
  • Improved error messaging for Cloud Fetch.
  • Security updates.

This release also resolves the following regression in 2.7.3:

  • The SQLForeignKeys function returns an empty result if the caller passes a NULL value for the catalog of the primary table or the foreign table.

The release 2.7.5 now returns all the primary keys or foreign keys associated with the related table if the catalog of the other table is set NULL.

Databricks extension for Visual Studio Code updated to version 1.1.3

September 21, 2023

The Databricks extension for Visual Studio Code version 1.1.3 adds a few minor fixes. For details, see the changelog for version 1.1.3.

Running jobs as a service principal is GA

September 21, 2023

Roles on service principals and the job's setting to run jobs using the service principal's identity is now GA.

See Roles for managing service principals and Configure identity for job runs.

Databricks CLI updated to version 0.205.2 (Public Preview)

September 21, 2023

The Databricks command-line interface (Databricks CLI) has been updated to version 0.205.2. This release contains a few minor updates. For details, see the changelog for version 0.205.2.

Databricks CLI updated to version 0.205.1 (Public Preview)

September 20, 2023

The Databricks command-line interface (Databricks CLI) has been updated to version 0.205.1. This release enables target overrides for job tasks in Databricks Asset Bundles, and more. For details, see the changelog for version 0.205.1.

Databricks SDK for Go updated to version 0.20.0 (Beta)

September 20, 2023

Databricks SDK for Go version 0.20.0 adds over a dozen APIs, adds a half-dozen fields, changes a few fields and methods, and removes a few APIs. For details, see the changelog for version 0.20.0.

Databricks SDK for Python updated to version 0.9.0 (Beta)

September 20, 2023

Databricks SDK for Python version 0.9.0 changes a few methods and fields, adds about 10 fields, adds about 20 dataclasses, and introduces additional changes and removals. For details, see the changelog for version 0.9.0.

Databricks SDK for Python updated to version 0.7.1 (Beta)

September 20, 2023

Databricks SDK for Python version 0.7.1 includes a minor fix to improve file download performance. For details, see the changelog for version 0.7.1.

Databricks Terraform provider updated to version 1.26.0

September 20, 2023

Databricks Terraform provider version 1.26.0 adds support for init scripts from Unity Catalog volumes, includes several additional improvements and bug fixes, and more. For details, see the changelog for version 1.26.0.

Databricks Connect for Databricks Runtime 14.0

September 20, 2023

Databricks Connect for Databricks Runtime 14.0 supports full Structured Streaming (including foreachBatch) and long running queries that reattach to the query after interruption.

Databricks Connect V2 is Public Preview for Scala

September 19, 2023

Databricks Connect for Databricks Runtime 13.3 LTS now supports Scala in Public Preview, including DataFrames, Delta, Structured Streaming, and more.

For Python, query cancellation is now supported, and the DatabricksSession class can now be used in Azure Databricks notebooks and jobs.

On-demand feature computation now available in Unity Catalog (Public Preview)

September 14, 2023

Machine learning features can now be computed on-demand at inference time. Feature computation logic, models, and data are all governed by Unity Catalog. This enables models to compute features using inputs that are only available at inference time, such as a user's current location. For more information, see Compute features on demand using Python user-defined functions.

Databricks Runtime 14.0 is GA

September 14, 2023

Databricks Runtime 14.0 and Databricks Runtime 14.0 ML are now generally available.

See Databricks Runtime 14.0 (EoS) and Databricks Runtime 14.0 for Machine Learning (EoS).

Account email notifications upgrade

September 14, 2023

Databricks no longer sends email notifications when users are added to an Azure Databricks account. Users will continue to receive email notifications when they are granted access to a workspace.

Databricks Terraform provider updated to version 1.25.1

September 13, 2023

Databricks Terraform provider version 1.25.1 has been released with minor improvements and bug fixes. For details, see the changelog for version 1.25.1.

Databricks SDK for Go updated to version 0.19.2 (Beta)

September 13, 2023

Databricks SDK for Go version 0.19.2 contains one minor fix. For details, see the changelog for version 0.19.2.

Databricks CLI updated to version 0.205.0 (Public Preview)

September 13, 2023

The Databricks command-line interface (Databricks CLI) has been updated to version 0.205.0. This is the first release of the Databricks CLI to support Databricks Asset Bundles in Public Preview. For details, see the changelog for version 0.205.0.

Reserved location for /Workspace

September 13, 2023

The location /Workspace is now reserved. You can no longer define a directory named /Workspace relative to the root path in Azure Databricks.

Databricks Asset Bundles (Public Preview)

September 12, 2023

Databricks Asset Bundles, now in Public Preview, enable end-to-end data, analytics, and ML projects to be expressed as a collection of source files. This makes it simpler to apply data engineering best practices such as source control, code review, testing, and CI/CD.

Databricks Terraform provider updated to version 1.25.0

September 12, 2023

Databricks Terraform provider version 1.25.0 includes the addition of IS_OWNER permission for Databricks SQL warehouses, minor fixes, documentation updates, and more. For details, see the changelog for version 1.25.0.

Databricks extension for Visual Studio Code updated to version 1.1.2

September 12, 2023

The Databricks extension for Visual Studio Code version 1.1.2 adds experimental support for %run and # MAGIC in Azure Databricks notebooks, fixes path issues with multiple local Databricks CLI installations, and more. For details, see the changelog for version 1.1.2.

Databricks CLI updated to version 0.204.1 (Public Preview)

September 11, 2023

The Databricks command-line interface (Databricks CLI) has been updated to version 0.204.1. For details, see the changelog for version 0.204.1.

Filter sensitive data with row filters and column masks (Public Preview)

September 11, 2023

You can now apply row filters and column masks to tables to prevent access to sensitive data by certain users. These filters and masks are implemented as SQL user-defined functions (UDFs). See Filter sensitive table data using row filters and column masks.

Pricing system table is now available (Public Preview)

September 8, 2023

System tables now include a pricing table (system.billing.list_prices) where you can monitor the cost of a SKU over time. These logs can help you perform cost analysis and monitor pricing changes. See Pricing system table reference .

Databricks SDK for Go updated to version 0.19.1 (Beta)

September 7, 2023

Databricks SDK for Go version 0.19.1 continues a few minor updates and fixes. For details, see the changelog for version 0.19.1.

Databricks ODBC driver 2.7.3

September 7, 2023

We have released version 2.7.3 of the Databricks ODBC driver (download). This release includes the following features and fixes:

  • Vulnerability fixes by upgrading third-party dependencies.
  • SQL_ATTR_MAX_ROWS support that allows a client to limit the number of rows to fetch in a query to expedite fast loading (such as preview).
  • Updated MacOS support. The driver now supports Apple Silicon/M1.
  • OAuth2 browser-based authentication support to facilitate single sign-on (SSO).

Important

If you use primary key and foreign key APIs, the SQLForeignKeys function returns an empty result if the caller passes a NULL value for the catalog of the primary table or the foreign table. This issue is known to developers.

Databricks CLI updated to version 0.204.0 (Public Preview)

September 6, 2023

The Databricks command-line interface (Databricks CLI) has been updated to version 0.204.0. This release includes the grouping of permission related commands, adds and changes to several commands, and more. For details, see the changelog for version 0.204.0.

Data Explorer is now Catalog Explorer

September 6, 2023

Data Explorer is renamed to Catalog Explorer to recognize the fact that you can use it to work with all securable objects in Unity Catalog, not just data objects. See What is Catalog Explorer?.

Databricks SDK for Go updated to version 0.19.0 (Beta)

September 4, 2023

Databricks SDK for Go version 0.19.0 renames permissions APIs, adds 8 APIs, adds 5 fields, changes and removes a few fields, and changes a few methods.. For details, see the changelog for version 0.19.0.

Databricks SDK for Python updated to version 0.8.0 (Beta)

September 4, 2023

Databricks SDK for Python version 0.8.0 renames permissions APIs, adds a half-dozen fields, adds a half-dozen dataclasses, and introduces several changes and removals. For details, see the changelog for version 0.8.0.

Delegate allowlist privileges in Unity Catalog

September 1, 2023

You can now delegate privileges for managing items in the allowlist for Unity Catalog. See Allowlist libraries and init scripts on shared compute.

GitHub apps integration in Repos is GA

September 1, 2023

GitHub apps integration in Repos is GA. The feature allows for simpler Git credential management when you integrate with GitHub in Repos.

Important

GitHub apps are not supported on GitHub Enterprise Server.