October 2020

These features and Azure Databricks platform improvements were released in October 2020.

New Azure Databricks Power BI connector available in the online Power BI service (Public Preview)

October 28, 2020

The new Azure Databricks Power BI connector, released in Power BI Desktop in September (Public Preview), is now also available in the Power BI service (also known as Power BI online).

The native connector lets you connect to Azure Databricks from Power BI Desktop using Microsoft Entra ID credentials and to the Power BI service using SSO. With support for DirectQuery, the native connector lets you access data directly in Azure Databricks, querying fresh data and enforcing data lake security controls. If you use the Power BI service, SSO ensures that all report users access Azure Databricks with their own credentials—no need to duplicate security controls in Power BI. The connector is based on the Databricks ODBC driver, which has been optimized to speed up transfer of results.

For more information, see Connect Power BI to Azure Databricks.

Databricks Runtime 7.4 (Beta)

October 21, 2020

Databricks Runtime 7.4, Databricks Runtime 7.4 ML, and Databricks Runtime 7.4 for Genomics are now available as Beta releases.

For information, see the full release notes at Databricks Runtime 7.4 (EoS) and Databricks Runtime 7.4 for ML (EoS).

Use customer-managed keys for DBFS root (GA)

October 21, 2020

The ability to use your own encryption key in Azure Key Vault to encrypt the DBFS storage account is now generally available. This feature requires the Premium plan. See Customer-managed keys for DBFS root.

Expanded experiment access control (ACLs)

October 20-27, 2020: Version 3.31

The expanded experiment permissions introduced with Azure Databricks platform version 3.29 (Sept 23-29, 2020) are now enabled for all deployments.

For more information, see MLflow experiment ACLs.

High fidelity import and export of Jupyter notebook (ipynb) files

October 20-27, 2020: Version 3.31

Azure Databricks now provides high fidelity import and export of notebooks to the Jupyter notebook (ipynb) file format. When you import Jupyter notebook files that were originally exported from Azure Databricks, results and dashboards are preserved. This was previously possible only with the DBC external format. With this upgrade to Jupyter notebook handling, Azure Databricks notebooks are now compatible with tools like GitHub, nbformat, nbdime, and nbconvert.

SCIM API improvement: both indirect and direct groups returned in user record response

October 20-27, 2020: Version 3.31

User records returned by the SCIM API now comply with the SCIM RFC-7643 standard in the way that the response lists groups to which the user belongs. SCIM RFC-7643 defines two types of group membership: direct and indirect. The API was returning only direct membership. It now returns both direct and indirect membership. It also includes a new type field to distinguish between the two. See Groups API.

Azure request ID is now included in exceptions

October 20-27, 2020: Version 3.31

The Azure request ID (x-ms-client-request-id) is now included in the following logs to aid in troubleshooting:

• Cluster events
• Service logs when Azure throws exceptions
• Usage logs of AzureLaunchException

Databricks Runtime 6.5 series support ends

October 14, 2020

Support for Databricks Runtime 6.5, Databricks Runtime 6.5 for Machine Learning, and Databricks Runtime 6.5 for Genomics ended on October 14. See Databricks support lifecycles.

Secure cluster connectivity (no public IPs) is Public Preview

October 13, 2020: Version 3.30

Secure cluster connectivity lets you launch clusters in which all nodes have only private IP addresses, providing enhanced security. You can enable secure cluster connectivity for new workspaces. If you have workspaces with public IPs that you would like to migrate, you should create new workspaces enabled for secure cluster connectivity and migrate your resources to the new workspaces. Contact your Azure or Databricks account team for details.

For more information, see Enable secure cluster connectivity.

SCIM API improvement: $ref field response

October 7-13, 2020: Version 3.30

In SCIM responses, the $ref field was returning an internal hostname in the URI. In accordance with SCIM RFC-7463, SCIM responses now return a relative URI in the $ref field.

Diagnostic (audit) logs are now delivered with low latency

October 7-13, 2020: Version 3.30

Diagnostic audit logs are now delivered with low latency, with 99% of logs delivered within 15 minutes in Azure Commercial regions. No action is necessary to enable low-latency delivery. See Configure diagnostic log delivery.

Databricks Runtime 7.3, 7.3 ML, and 7.3 Genomics declared Long Term Support (LTS)

October 8, 2020

Databricks Runtime 7.3, Databricks Runtime 7.3 for Machine Learning, and Databricks Runtime 7.3 for Genomics have been declared Long Term Support (LTS) releases.

Azure Databricks provides two full years of support for LTS releases. These releases are supported until September 24, 2022.

For more information about these Databricks Runtime versions, see the Databricks Runtime 7.3 LTS (EoS), Databricks Runtime 7.3 LTS for Machine Learning (EoS), and Databricks Runtime 12.1 (EoS) release notes.

Render images at higher resolution using matplotlib

October 7-13, 2020: Version 3.30

You can now render matplotlib images in Python notebooks at double the standard resolution (also known as retina resolution), providing users of high-resolution screens with a better visualization experience. See Render images at higher resolution.

Use the Databricks CLI or the Databricks API to create Azure Key Vault backed secret scopes

October 7-13, 2020: Version 3.30

You can now use the Databricks CLI or the Databricks REST API to create Azure Key Vault-backed secret scopes. See Secret scopes and Secrets API.

Use Microsoft Entra ID tokens to authenticate to the Databricks CLI

October 7-13, 2020: Version 3.30

You can now use Microsoft Entra ID tokens to authenticate to the Databricks CLI. See Set up authentication.

New Azure regions (Public Preview)

October 1, 2020

Azure Databricks is now available in the Switzerland North, China East 2, and China North 2 regions as a Public Preview.