March 2023
These features and Azure Databricks platform improvements were released in March 2023.
Note
The release date and content listed below only corresponds to actual deployment of the Azure Public Cloud in most case.
It provide the evolution history of Azure Databricks service on Azure Public Cloud for your reference that may not be suitable for Azure operated by 21Vianet.
Note
Releases are staged. Your Azure Databricks account might not be updated until a week or more after the initial release date.
Databricks Terraform provider updated to version 1.14.0
March 31, 2023
Version 1.14.0 adds an example for automatically rotating an access token, a run_if
condition and a file arrival trigger to the databricks_job
resource, and more. For more details, see the changelog for version 1.14.0.
Databricks Runtime 7.3 LTS ML support ends
March 31, 2023
Support for Databricks Runtime 7.3 LTS for Machine Learning ended on March 31. See Databricks support lifecycles.
Distributed training with TorchDistributor
March 27, 2023
With Databricks Runtime 13.0 ML and above, you can perform distributed training on PyTorch ML models using TorchDistributor. See Distributed training with TorchDistributor.
TorchDistributor is an open-source module in PySpark that helps users do distributed training with PyTorch on their Spark clusters, so it lets you launch PyTorch training jobs as Spark jobs.
Databricks Runtime 13.0 (Beta)
March 24, 2023
Databricks Runtime 13.0 and Databricks Runtime 13.0 ML are now available as Beta releases.
See Databricks Runtime 13.0 (EoS) and Databricks Runtime 13.0 for Machine Learning (EoS).
Improved file editor
March 24, 2023
The file editor has been updated to bring many of the notebook features to your file editing experience. Additionally, you now can directly execute Python, SQL, R, and Scala files from the file editor. For details, see Use the Databricks notebook and file editor.
Change the DBFS Root Storage Redundancy for existing Azure Databricks Workspaces
March 23, 2023
Users now have the option to change the redundancy of their DBFS root storage account. See Change workspace storage redundancy options.
.ipynb (Jupyter) notebook support in Repos (preview)
March 22, 2023
Support for Jupyter notebooks (.ipynb files) is available in Repos. You can clone repositories with .ipynb notebooks, work in Databricks UI, and then commit and push as .ipynb notebooks. Metadata such as a notebook dashboard is preserved. Admins can control whether outputs can be committed or not.
You can also:
- Create new .ipynb notebooks.
- Convert notebooks to .ipynb file format.
- View diffs as Code diff (code changes in cells) or Raw diff (code changes in JSON, including metadata).
See Allow committing .ipynb notebook output.
Support for reload4j
March 21, 2023
Reload4j 1.2.19 is replacing log4j in Databricks Runtime versions 10.4 and earlier.
New IP addresses for China East 2 region
March 15, 2023
Azure Databricks has added a new dedicated control plane for the chinaeast2
region. If you host an Azure Databricks workspace in your own VNet (also known as VNet injection) in the China East 2 region, you likely manage network traffic for your workspace using custom routes (UDR). If so, you must update those routes using the additional IP addresses and domains listed at IP addresses and domains for Azure Databricks services and assets.
Execute SQL cells in the notebook in parallel
March 15, 2023
You can now run SQL cells in Azure Databricks notebooks in parallel while attached to an interactive cluster. See Execute SQL cells in parallel.
Create job tasks using Python code stored in a Git repo
March 14, 2023
You can now retrieve Python code from a Git provider when you add a Python task to an Azure Databricks job, simplifying integration of your jobs with your CI/CD workflows. See Use Python code from a remote Git repository.
Databricks Terraform provider updated to version 1.13.0
March 14, 2023
Version 1.13.0 adds a databricks_sql_alert
resource, and more. For more details, see the changelog for version 1.13.0.
Databricks Terraform provider updated to version 1.12.0
March 9, 2023
Version 1.12.0 adds a databricks_model_serving
resource, deprecates the schedule
block within the databricks_sql_query
resource, and more. For more details, see the changelog for version 1.12.0.
SQL admin console and workspace admin console combined
March 9, 2023
The SQL admin console has been combined with the general admin settings to create a unified experience for admin users. All SQL admin settings are now accessed from the admin console.
Model Serving is GA
March 7, 2023
Model Serving, formerly Serverless Real-Time Inference, is now generally available.
Model Serving provides a highly available and low-latency service for deploying models. The service automatically scales up or down to meet demand changes within the chosen concurrency range.
Automatic feature lookup is GA
March 7, 2023
Automatic feature lookup with Model Serving is now generally available.
For new workspaces that use UDRs, Azure storage IP needs an additional UDR
March 6, 2023
Due to changes in DBFS root storage for new workspaces, new workspaces that use VNet injection and define user-defined routes (UDRs) using IP addresses instead of the recommended service tags must add an additional Azure UDR for each VNet. To configure the two UDRs, see DBFS root storage IP address. This change does not affect existing workspaces.
Azure Databricks DBFS root storage uses Azure Data Lake Storage Gen2 for new workspaces
March 6, 2023
New Azure Databricks workspaces use Azure Data Lake Storage Gen2 storage accounts for the DBFS root. Existing Azure Databricks workspaces continue to use Blob storage for the DBFS root. See Introduction to Azure Data Lake Storage Gen2.
New Catalog Explorer availability
March 6, 2023
A new version of Catalog Explorer is now available in all workspaces that use a supported version of Databricks Runtime 7.3 or later.
View frequent queries and users of a table using the Insights tab
March 6, 2023
You can now use the Insights tab in Catalog Explorer to view the most frequent queries and users of any table registered in Unity Catalog. You must have the SELECT
privilege on a table to see this data, and you only see queries that you have permission to view. See View frequent queries and users of a table.
Exact match search is available in global search
March 3, 2023
You can now search for exact phrases using quotation marks in global search. See Search for workspace objects.
View lineage information for your Azure Databricks jobs
March 3, 2023
If Unity Catalog is enabled in your workspace, you can view lineage information for your jobs in the Azure Databricks Jobs UI, including upstream tables the job reads from and downstream tables the job writes to. See View lineage information for a job.
Databricks Runtime 12.2 LTS and Databricks Runtime 12.2 LTS ML are GA
March 2, 2023
Databricks Runtime 12.2 LTS and Databricks Runtime 12.2 LTS ML are now generally available.
See Databricks Runtime 12.2 LTS and Databricks Runtime 12.2 LTS for Machine Learning.
Workspace files are now in Public Preview
March 6, 2023
You can now work with non-notebook files in Azure Databricks. Workspaces files are enabled by default in all workspaces. See What are workspace files?.