March 2022

  • Article

These features and Azure Databricks platform improvements were released in March 2022.

Note

The release date and content listed below only corresponds to actual deployment of the Azure Public Cloud in most case.

It provide the evolution history of Azure Databricks service on Azure Public Cloud for your reference that may not be suitable for Azure operated by 21Vianet.

Note

Releases are staged. Your Azure Databricks account may not be updated until a week or more after the initial release date.

Better performance and cost for your Delta Live Tables pipelines with Databricks Enhanced Autoscaling

March 31, 2022

Databricks Enhanced Autoscaling improves the existing cluster autoscaling functionality by adding streaming workload optimizations and improved support for batch workloads. Enhanced Autoscaling improvements provide more efficient cluster utilization, reduced resource usage, and lower cost. See Optimize the cluster utilization of Delta Live Tables pipelines with Enhanced Autoscaling.

Files in Repos enabled by default in new workspaces

March 30, 2022

The Files in Repos feature is now enabled by default in newly created workspaces.

Databricks Feature Store is generally available

March 22, 2022

Databricks Feature Store is generally available. Databricks Feature Store enables feature sharing and discovery across your organization and also ensures consistent feature computation code for model training and inference.

Share an experiment from the experiment page

March 21-28, 2022: Version 3.68

The Permissions button on the MLflow experiment page has been renamed Share. The dialog now includes a shareable link to the experiment. See Change permissions for experiment.

RStudio Workbench bug fix

March 21-28, 2022: Version 3.68

The latest RStudio Workbench depends on cookies when you use a license file. These cookies were not allowlisted for proxying. This issue has been fixed.

New workspace language options

March 21, 2022

Workspaces are now available in the following languages:

  • Español
  • Português (Brasil)

Databricks Runtime 10.4 LTS and 10.4 LTS ML are GA; 10.4 Photon is Public Preview

March 18, 2022

Databricks Runtime 10.4 LTS and Databricks Runtime 10.4 LTS ML are now generally available. Databricks Runtime 10.4 Photon is in Public Preview.

Note

LTS means this version is under long-term support. See Databricks Runtime LTS version lifecycle.

See Databricks Runtime 10.4 LTS and Databricks Runtime 10.4 LTS for Machine Learning.

Unity Catalog is available in Public Preview

March 10, 2022

Unity Catalog (Preview) is a fine-grained governance solution for data and AI on the Databricks lakehouse.

Unity Catalog helps simplify security and governance of your data with the following key features:

  • Define once, secure everywhere: Unity Catalog offers a single place to administer data access policies that apply across all workspaces and personas.
  • Standards-compliant security model: Unity Catalog's security model is based on standard ANSI SQL, and allows administrators to grant permissions at the level of catalogs, databases (also called schemas), tables, and views in their existing data lake using familiar syntax.
  • Built-in auditing: Unity Catalog automatically captures user-level audit logs that record access to your data.

You can access data in Unity Catalog in the same ways you access data that is local to a workspace: from a Data Science & Engineering or Databricks Mosaic AI cluster, a SQL warehouse, the Databricks SQL editor, or using the Unity Catalog REST API or the Unity Catalog command-line interface (CLI).

For more information, see What is Unity Catalog?. To participate in the Unity Catalog public preview, contact your Azure Databricks account team.

Delta Sharing is available in Public Preview

March 10, 2022

Delta Sharing (Preview) is an open protocol developed by Databricks for secure data sharing with other organizations regardless of which computing platforms they use. A Databricks user, called a "data provider", can use Delta Sharing to share data in a Unity Catalog metastore with a person or group outside of their organization, called a "data recipient". Data recipients can immediately begin working with the latest version of the shared data. For a full list of connectors and information about how to use them, see the Delta Sharing project documentation. When Delta Sharing is enabled on a Unity Catalog metastore, Unity Catalog runs a Delta Sharing server.

For more information, see What is Delta Sharing?. To participate in the Delta Sharing public preview, contact your Azure Databricks account team.

Rotate the customer-managed key for managed services

March 10, 2022

You can add a customer-managed key (CMK) for managed services. Azure Databricks uses your key to protect and control access to the key that encrypts data in the control plane for notebook source, secrets, and Databricks SQL query history. This feature is in Public Preview.

You can now rotate (update) the key.

Enhanced access control for Delta Live Tables pipelines

March 9, 2022

You now have more control over access to your Delta Live Tables pipelines, including the owner of a pipeline, who can view or manage the pipeline, and who can run the pipeline. See Delta Live Tables pipeline ACLs.

Test Delta Live Tables preview functionality with the new channel setting (Public Preview)

March 7-March 14, 2022: Version 3.67

You can test upcoming features in your Delta Live Tables pipelines with the new channel setting. To test your pipeline with upcoming features, set channel to preview. To switch back to the current release, set channel to current.

Improved error handling for Delta Live Tables Python functions (Public Preview)

March 7-March 14, 2022: Version 3.67

This release adds type checking to validate that Delta Live Tables Python functions use the string data type when setting names and values for table properties. Previously, setting table properties using non-string values could cause your pipeline to fail without an error message indicating the underlying cause.

Improvements to Databricks Repos

March 7-14, 2022: Version 3.67

The following improvements have been made to Databricks Repos.

  • Text search and replace in files in repos. The file editor now has File and Edit menus that provide search, replace, and replace all functionality.
  • Previously, Databricks recommended that the total number of notebooks and files in a repo not exceed 2000. That limit has been increased to 5000.

Diagnostic logging for cluster policy changes

March 7-14, 2022: Version 3.67

When audit logging is enabled, an audit event is now logged when you create, update, or delete a cluster policy, or when you update user permissions for a cluster policy. For more information, see Diagnostic log reference.

Databricks Runtime 10.4 (Beta)

March 4, 2022

Databricks Runtime 10.4, 10.4 Photon, and 10.4 ML are now available as Beta releases.

See the full release notes at Databricks Runtime 10.4 LTS and Databricks Runtime 10.4 LTS for Machine Learning.