February 2021

  • Article

These features and Azure Databricks platform improvements were released in February 2021.

Note

The release date and content listed below only corresponds to actual deployment of the Azure Public Cloud in most case.

It provide the evolution history of Azure Databricks service on Azure Public Cloud for your reference that may not be suitable for Azure operated by 21Vianet.

Note

Releases are staged. Your Azure Databricks account may not be updated until a week or more after the initial release date.

New Azure Databricks Power BI connector (GA)

February 26, 2021

The new Power BI connector for Azure Databricks, released in public preview in September 2020, is now GA. It provides:

  • Simple connection configuration: the new Power BI Azure Databricks connector is integrated into Power BI, and you configure it using a simple dialog with a couple of clicks.
  • Authentication based on Microsoft Entra ID credentials—no more need for administrators to configure PAT tokens.
  • Faster imports and optimized metadata calls, thanks to the new Azure Databricks ODBC driver, which comes with significant performance improvements.
  • Access to Azure Databricks data through Power BI respects Azure Databricks table access control and Azure storage account permissions associated with your Microsoft Entra ID identity.

For more information, see Connect Power BI to Azure Databricks.

Diagnostic logging of user activities in Azure Databricks is GA

February 23 - March 2, 2021: Version 3.40

Capture a detailed audit trail of the activities that your users perform in Azure Databricks, now generally available.

See Diagnostic log reference.

Added modification_time to the DBFS REST API get-status and list responses

February 23 - March 2, 2021: Version 3.40

A file object's last modification time is now available in the DBFS REST API. The modification_time is now returned in the FileInfo response structure by both the get-status and list API endpoints. An example JSON response for a get-status call on a directory:

{"path": "/test-directory", "is_dir": true, "file_size": 0, "modification_time": 1610366751000}

See DBFS API.

Easily copy long experiment names in MLflow

February 16-23, 2021: Version 3.39

You can now copy the experiment name from the experiment page, making it easier to set the active MLflow experiment to which a notebook logs runs. See Copy experiment name.

Adjust memory size and number of cores for serving clusters

February 16-23, 2021: Version 3.39

You can now customize clusters used for MLflow model serving. See Feature store integration.

Web terminal is now GA

February 16-23, 2021: Version 3.39

Web terminal is now generally available. It provides a convenient and highly interactive way for you to run shell commands and use editors on the Spark driver node. See Run shell commands in Azure Databricks web terminal.

Apache Spark connector for Azure SQL Database and SQL Server supports Databricks Runtime 7.x and above

February 16, 2021

The Spark connector for Azure SQL Database and SQL Server now supports Spark 3 and therefore Databricks Runtime 7.x and above.

Databricks Runtime 7.2 series support ends

February 11, 2021

Support for Databricks Runtime 7.2, Databricks Runtime 7.2 for Machine Learning, and Databricks Runtime 7.2 for Genomics ended on February 11. See Databricks support lifecycles.

Databricks Runtime 7.6 GA

February 8, 2021

Databricks Runtime 7.6 and Databricks Runtime 7.6 ML are now generally available.

For information, see the full release notes at Databricks Runtime 7.6 (EoS) and Databricks Runtime 7.6 for Machine Learning (EoS).

Databricks Runtime 8.0 (Beta)

February 1, 2021

Databricks Runtime 8.0 and Databricks Runtime 8.0 ML are now available as Beta releases.

For information, see the full release notes at Databricks Runtime 8.0 (EoS) and Databricks Runtime 8.0 for ML (EoS).

Databricks Runtime for Genomics now deprecated

February 1, 2021

Databricks is no longer building new Databricks Runtime for Genomics releases and will remove support for Databricks Runtime for Genomics on September 24, 2022, when Databricks Runtime for Genomics 7.3 LTS support ends. At that point Databricks Runtime for Genomics will no longer be available for selection when you create a cluster.

For more information about the Databricks Runtime deprecation policy and schedule, see Supported Databricks Runtime LTS releases.

View more readable JSON in the MLflow run artifact display

February 4-11, 2021: Version 3.38

JSON output in the MLflow artifact panel is now in a more readable format.

Provide comments in the Model Registry using REST API

February 4-11, 2021: Version 3.38

You can now add comments in the Model Registry using the REST API.

Easily specify default cluster values in API calls

February 4-11, 2021: Version 3.38

Now you can use the applyDefaultPolicyValues field when you create and edit all-purpose clusters using the Databricks Clusters API. When true, it uses policy default values for missing cluster attributes.

Tune cluster worker configuration according to current worker allocation

February 4-11, 2021: Version 3.38

The cluster list displays the number of workers allocated to each running cluster. This number is now shown in the cluster detail page so you can easily compare current sizing of the cluster with the cluster configuration details and make configuration adjustments as needed.

Pass context specific information to a job's task with task parameter variables

February 4-11, 2021: Version 3.38

You can now pass variables to tasks containing context specific information such as the job ID or execution start time. For more information see What is a dynamic value reference?.

Error messages from job failures no longer contain possibly sensitive information

February 4-11, 2021: Version 3.38

Job failure error messages no longer include stack traces from Java exceptions. The stack traces, included in the miscMessage field, potentially exposed sensitive information to users.