November 2024

  • Article

These features and Azure Databricks platform improvements were released in November 2024.

Note

The release date and content listed below only corresponds to actual deployment of the Azure Public Cloud in most case.

It provide the evolution history of Azure Databricks service on Azure Public Cloud for your reference that may not be suitable for Azure operated by 21Vianet.

Note

Releases are staged. Your Azure Databricks account might not be updated until a week or more after the initial release date.

Mosaic AI Model Training serverless forecasting (Public Preview)

December 5, 2024

Mosaic AI Model Training - forecasting improves upon the existing AutoML forecasting experience with serverless compute, Unity Catalog support, access to deep learning algorithms, and an upgraded interface.

Expanded support for Python debugger in notebooks

November 18, 2024

The built-in Python debugger in Databricks notebooks is now supported on serverless compute and compute configured with shared access mode in Databricks Runtime 14.3 LTS or above. To learn more about the debugger, see Debug notebooks.

Improved web terminal experience

November 18, 2024

Databricks made the following improvements to the web terminal experience:

  • Support for configuration files: You can now set persistent configurations for your Databricks web terminal with .bashrc configuration files. See Configure your web terminal.
  • Command cycle history: You can now cycle through commands executed in previous sessions with the "Up" and "Down" arrow keys. This makes it easy to recall and reuse previous commands without retyping them.
  • Shared Cluster Support: Our improved web terminal is available on shared clusters in Databricks Runtime 15.1 and above.

To learn more about using the web terminal, see Run shell commands in Azure Databricks web terminal.

Query history and query profile support for Delta Live Tables

November 15, 2024

Azure Databricks now offers improved tools for monitoring query performance in Delta Live Tables pipelines. See Access query history for Delta Live Tables pipelines.

Mosaic AI Model Training rebrand

November 15, 2024

Mosaic AI Model Training has been rebranded to encompass existing features:

  • Foundation Model Fine-tuning , previously known as Mosaic AI Model Training, continues to offer advanced customization for foundation models.

Reuse SQL outputs as _sqldf in subsequent SQL cells

November 15, 2024

In Databricks Runtime 14.3 and above, running a SQL query in a Databricks notebook generates an implicit DataFrame (_sqldf) that you can use in downstream SQL cells. This adds to the support for using for using the _sqldf variable in Python cells introduced in Databricks Runtime 13.3. See Explore SQL cell results.

Cross-platform view sharing (Public Preview)

November 14, 2024

Databricks recipients can now query shared views using any Databricks compute resource. Previously, if a recipient's Azure Databricks account was different from the provider's account, recipients could only query a shared view using a serverless SQL warehouse. See Read shared views.

View sharing also now extends to open sharing connectors. See Read data shared using Delta Sharing open sharing (for recipients).

Databricks Runtime 16.0 is GA

November 11, 2024

Databricks Runtime 16.0 and Databricks Runtime 16.0 ML are now generally available.

See Databricks Runtime 16.0 and Databricks Runtime 16.0 for Machine Learning.

Breaking change: Hosted RStudio is end-of-life

November 11, 2024

In Databricks Runtime 16.0 and above, Databricks-hosted RStudio Server is end-of-life and unavailable on any Azure Databricks workspace. To learn more and see a list of alternatives to RStudio, see Hosted RStudio Server deprecation.

Predictive optimization enabled for all new Azure Databricks accounts

November 11, 2024

On November 11, Databricks enabled predictive optimization as the default for all new Azure Databricks accounts. Previously, it was disabled by default and could be enabled through your administrator account. When predictive optimization is enabled, Azure Databricks automatically runs maintenance operations for Unity Catalog managed tables. For more information on predictive optimization, see Predictive optimization for Unity Catalog managed tables .

Updated and unified read-only code blocks

November 8, 2024

A new read-only code block is now available across the Catalog Explorer, alerts page, and Genie.

Unity Catalog-governed access to external cloud services using service credentials (Public Preview)

November 7, 2024

Service credentials enable simple and secure authentication with your cloud tenant's services using Azure managed identities (MI) and Unity Catalog. Service credentials are in public preview and are supported on Databricks Runtime 15.4 LTS and above. See Manage access to external cloud services using service credentials.

Databricks Runtime 14.2 series support ends

November 5, 2024

Support for Databricks Runtime 14.2 and Databricks Runtime 14.2 for Machine Learning ended on November 5. See Databricks support lifecycles.