January 2022

  • Article

These features and Azure Databricks platform improvements were released in January 2022.

Note

The release date and content listed below only corresponds to actual deployment of the Azure Public Cloud in most case.

It provide the evolution history of Azure Databricks service on Azure Public Cloud for your reference that may not be suitable for Azure operated by 21Vianet.

Note

Releases are staged. Your Azure Databricks account may not be updated until a week or more after the initial release date.

MLflow Model Registry Webhooks on Azure Databricks (Public Preview)

January 31, 2022

You can now use webhooks to listen for MLflow Model Registry events and trigger actions based on them. For example, you can use webhooks to automate and integrate your machine learning pipeline with existing CI/CD tools and workflows, trigger CI builds when a new model version is created, or notify your team members through Slack when there is a request to transition a model to production.

Breaking change: cluster idempotency token cleared on cluster termination

January 26, 2022

Beginning February 14th, 2022, Azure Databricks cluster management will be modified to clear idempotency tokens when a cluster is terminated. This change is required to enable improvements to cluster management uptime guarantees.

You must update any clients or scripts that rely on an idempotency token to persist after terminating a cluster. If you submit a create cluster request using an idempotency token not assigned to a non-terminated cluster in the workspace, a new cluster is created.

The following illustrates the behavior before this change:

createCluster(idempotencyToken = tokenA) // launches new cluster A
terminateCluster(cluster A) // idempotency token tokenA is still associated with the terminated cluster A
startCluster(cluster = A)

createCluster(idempotencyToken = tokenA) // returns the currently running cluster A

The following illustrates the new behavior:

createCluster(idempotencyToken = tokenA) // launches new cluster A
terminateCluster(cluster A) // idempotencyToken token tokenA is cleared from the cluster A configuration
startCluster(cluster A)

createCluster(idempotencyToken = tokenA) // launches new cluster B

Databricks Runtime 10.3 (Beta)

January 26, 2022

Databricks Runtime 10.3, 10.3 Photon, and 10.3 ML are now available as Beta releases.

See the full release notes at Databricks Runtime 10.3 (EoS) and Databricks Runtime 10.3 for ML (EoS).

View information on recent job runs

January 25, 2022

You can now view in real time currently running and recently completed job runs in the Job runs page in the Azure Databricks jobs user interface. See View recent job runs across all jobs.

Use Markdown in Databricks Repos file editor

January 25-31, 2022: Version 3.64

The Databricks Repos file editor can now render markdown. For details, see Git integration for Databricks Git folders.

Improved cluster management for jobs that orchestrate multiple tasks

January 24, 2022

You can now reuse job clusters in your jobs that orchestrate multiple tasks. Cluster reuse allows you to reduce cluster resource usage by using a single cluster to run multiple tasks in a job run. See Overview of orchestration on Databricks.

Databricks Runtime 8.3 and Databricks Runtime 8.4 series support ends

January 20, 2022

Support for Databricks Runtime 8.3, Databricks Runtime 8.3 for Machine Learning, Databricks Runtime 8.4, and Databricks Runtime 8.4 for Machine Learning ended on January 20. See Databricks support lifecycles.

Databricks JDBC driver 2.6.22

January 18, 2022

We have released version 2.6.22 of the Databricks JDBC driver (download). This release upgrades the log4j library to version 2.17.1 and removes the slf4j-log4j12 dependency.

Additional metastores now available in multiple regions

January 12, 2022

The following metastores are now available in the following regions:

centralusc2:

  • consolidated-centralusc2-prod-metastore-1.mysql.database.azure.com
  • consolidated-centralusc2-prod-metastore-2.mysql.database.azure.com
  • consolidated-centralusc2-prod-metastore-3.mysql.database.azure.com

eastus2c2:

  • consolidated-eastus2c2-prod-metastore-1.mysql.database.azure.com
  • consolidated-eastus2c2-prod-metastore-2.mysql.database.azure.com
  • consolidated-eastus2c2-prod-metastore-3.mysql.database.azure.com

eastusc3:

  • consolidated-eastusc3-prod-metastore-1.mysql.database.azure.com
  • consolidated-eastusc3-prod-metastore-2.mysql.database.azure.com
  • consolidated-eastusc3-prod-metastore-3.mysql.database.azure.com

You should always include all relevant hostnames for each region you use in your user-defined routes (UDRs). See User-defined route settings for Azure Databricks.

New Share button replaces Permissions icon in notebooks

January 10-18, 2022: Version 3.63

To change permissions on a notebook, use the new Share button share button in the notebook UI, or select Permissions from the drop-down menu. The Permissions icon in the notebook toolbar Permissions icon has been removed. For more information about setting notebook permissions, see Collaborate using Databricks notebooks.

New workspace language options

January 10, 2022

Workspaces are now available in the following languages:

  • Deutsch
  • Italiano
  • 日本語
  • 한국어

Additional metastores are now available in multiple regions

January 4, 2022

The following metastores are now available in the following regions:

northeuropec2:

  • consolidated-northeuropec2-prod-metastore-1.mysql.database.azure.com
  • consolidated-northeuropec2-prod-metastore-2.mysql.database.azure.com
  • consolidated-northeuropec2-prod-metastore-3.mysql.database.azure.com

westeuropec2:

  • consolidated-westeuropec2-prod-metastore-1.mysql.database.azure.com
  • consolidated-westeuropec2-prod-metastore-2.mysql.database.azure.com
  • consolidated-westeuropec2-prod-metastore-3.mysql.database.azure.com

You should always include all relevant hostnames for each region you use in your user-defined routes (UDRs). See User-defined route settings for Azure Databricks.