July 2023

  • Article

These features and Azure Databricks platform improvements were released in July 2023.

Note

The release date and content listed below only corresponds to actual deployment of the Azure Public Cloud in most case.

It provide the evolution history of Azure Databricks service on Azure Public Cloud for your reference that may not be suitable for Azure operated by 21Vianet.

Note

Releases are staged. Your Azure Databricks workspace might not be updated until a week or more after the initial release date.

Workspace admins can now create account groups

July 31, 2023

Workspace admins can now create account groups from their identity federated workspaces. Previously workspace admins could create only workspace-local groups. Account groups can be granted access to data in a Unity Catalog metastore and access to identity-federated workspaces. Databricks recommends that you use account groups instead of workspace-local groups to take advantage of Unity Catalog and a central place to administer identity.

See Manage account groups using the workspace admin settings page.

Group manager role is in Public Preview

July 31, 2023

You can now grant Databricks users, service principals, and groups permissions to manage groups. Group managers can manage group membership. They can also assign other users the group manager role. Account admins have the group manager role on all groups in the account. Workspace admins have the group manager role on account groups that they create.

See Who can manage account groups?.

Databricks CLI updated to version 0.202.0 (Public Preview)

July 27, 2023

The Databricks command-line interface (Databricks CLI) has been updated to version 0.202.0. For details, see the changelog for version 0.202.0.

Databricks SDK for Python updated to version 0.3.0 (Beta)

July 27, 2023

Databricks SDK for Python version 0.3.0 introduces support for the Account Network Policy service; handles nested query parameters; adds, removes, and creates various methods, fields, dataclasses, and services; and more. For details, see the changelog for version 0.3.0.

Databricks SDK for Go updated to version 0.14.1 (Beta)

July 27, 2023

Databricks SDK for Go version 0.14.1 handles nested query parameters appropriately, fixing filtering in QueryHistory listings. For details, see the changelog for version 0.14.1.

Databricks SDK for Go updated to version 0.14.0 (Beta)

July 26, 2023

Databricks SDK for Go version 0.14.0 adds support for the Account Network Policy service, and adds, removes, and changes several methods, fields, and types. For details, see the changelog for version 0.14.0.

Run another job as a task in an Azure Databricks job

July 25, 2023

You can use the new Run Job task to add another job as a task in your Azure Databricks job, allowing you to decompose a large processing workflow into multiple component jobs, or create reusable components to use in multiple jobs. For information about using the task in the Jobs UI, see Schedule and orchestrate workflows. For information about using the task with the Jobs REST API, see Jobs in the REST API 2.1 reference or the Jobs API 2.0 reference.

Workspace network update gets UI support (GA)

July 24, 2023

Previously you could use an ARM template with command line or the Azure portal to enable secure cluster connectivity or enable Azure Private Link.

You can now also use the Azure portal UI (with no template) for network updates directly on your Azure Databricks workspace instance. Both types of network updates are now generally available.

Classic keyboard shortcuts mode

July 21, 2023

A new setting allows you to use features from the new editor (such as autocomplete as you type, syntax highlighting, code formatting, and more) while retaining the same familiar keyboard shortcuts from the previous editor. To access this setting, open the editor configuration and toggle Notebook editor shortcuts mode to Classic (Code Mirror).

Lakehouse Federation lets you run queries against external database providers (Public Preview)

July 21, 2023

Lakehouse Federation is the query federation platform for Azure Databricks, taking advantage of Unity Catalog to enable users and systems to run queries against multiple external data sources without needing to migrate all data to a unified system.

Lakehouse Federation is intended for use cases like:

  • Ad hoc reporting.
  • Proof-of-concept work.
  • The exploratory phase of new ETL pipelines or reports.
  • Supporting workloads during incremental migration.

In each of these scenarios, query federation gets you to insights faster, because you can query the data in place and avoid complex and time-consuming ETL processing. Plus you get the advantages of Unity Catalog interfaces and data governance, including fine-grained access control, data lineage, and search.

See What is Lakehouse Federation?.

Move to trash enabled for Repos

July 19, 2023

Deleting repos now works the same way as deleting other workspace assets. In the Repos list, right-click the repo name and select Move to Trash. After 30 days, the Trash folder is automatically deleted permanently.

See Delete an object.

Create alerts for slow-running or stuck jobs

July 18, 2023

You can now configure an expected duration for an Azure Databricks job or job task and add notifications if a job or task exceeds the expected duration. This feature lets you get alerts for slow-running jobs without the run being canceled. To configure an expected duration in the Azure Databricks Jobs UI, see Configure an expected completion time or a timeout for a job. To configure the expected duration with the Jobs 2.1 API, see Jobs in the REST API reference.

Databricks SDK for Go updated to version 0.13.0 (Beta)

July 18, 2023

Databricks SDK for Go version 0.13.0 adds support for Azure managed identities authentication for account-level operations, adds, changes, and removes several methods and fields, and more. For details, see the changelog for version 0.13.0.

Databricks SDK for Python updated to version 0.2.0 (Beta)

July 18, 2023

The Databricks SDK for Python version 0.2.0 adds a local implementation of dbutils.widgets, adds, changes, and removes several methods, fields, and dataclasses, and more. For details, see the changelog for version 0.2.0.

Databricks CLI updated to version 0.201.0 (Public Preview)

July 18, 2023

The Databricks command-line interface (Databricks CLI) version 0.201.0 improves the auth login experience, supports tab completion for referencing Azure Databricks authentication configuration profiles, makes additions, changes, and removals of several command groups and commands, and more. For details, see the changelog for version 0.201.0.

Databricks SDK for Python updated to version 0.2.1 (Beta)

July 18, 2023

Databricks SDK for Python version 0.2.1 supports older versions of urllib. For details, see the changelog for version 0.2.1.

Databricks Assistant is in Public Preview

July 18, 2023

The Databricks Assistant works as an AI-based companion pair-programmer to make you more efficient as you create notebooks, queries, and files. It can help you rapidly answer questions by generating, optimizing, completing, explaining, and fixing code and queries.

Deactivate users and service principals from your account

July 13, 2023

You can now deactivate users and service principals from your Azure Databricks account. A deactivated user cannot log in to the Azure Databricks account or identity federated workspaces. However, all of the user's permissions and workspace objects remain unchanged. For more information, see Deactivate a user in your Azure Databricks account and Deactivate a service principal in your Azure Databricks account.

Account-level SCIM provisioning now deactivates users when they are deactivated in the identity provider

July 13, 2023

Account-level SCIM provisioning now deactivates users when they are deactivated in your identity provider. Previously, when a user was deactivated in an identity provider, account-level SCIM provisioning deleted them from the Azure Databricks account. For more information, see Sync users and groups from Microsoft Entra ID and Deactivate a user in your Azure Databricks account.

Trash directory admin access

July 13, 2023

Workspace admins can now access other users' Trash directories.

Prevention of MIME type sniffing and XSS attack page rendering are now always enabled

July 12, 2023

Azure Databricks reduces the risk of MIME type sniffing and XSS attack page rendering by adding appropriate HTTP headers. These features were enabled by default and previously could be disabled. For improved security, Azure Databricks now always enables both features and there are no longer workspace admin settings to disable them. For workspaces that previously disabled one or both of these features, both features are now enabled.

Unity Catalog volumes are in Public Preview

July 12, 2023

You can now use volumes in Unity Catalog to manage access to cloud object storage locations on Databricks Runtime 13.2 and above. Volumes provide capabilities for accessing, storing, governing, and organizing files. See What are Unity Catalog volumes?.

Simplified experience for submitting product feedback from the workspace

July 11, 2023

It's now easier to send product feedback from your workspace. You can submit feedback and attach screenshots with fewer steps and without leaving the workspace UI. For more information, see Submit product feedback.

Databricks extension for Visual Studio Code updated to version 1.1.0

July 10, 2023

The Databricks extension for Visual Studio Code version 1.1.0 enables Databricks Connect integration by default, adds experimental features for Jupyter notebooks, and more. For details, see the changelog for version 1.1.0.

Functions now displayed in Catalog Explorer (Public Preview)

July 10, 2023

SQL and Python user-defined functions are now visible in Catalog Explorer. You can view function details and manage permissions using this new UI.

Databricks Terraform provider updated to version 1.21.0

July 7, 2023

Databricks Terraform provider version 1.21.0 adds support for subscriptions in dashboards and alert SQL tasks in databricks_job, defines generic Azure Databricks data utilities for defining workspace and account-level data sources, and more. For details, see the changelog for version 1.21.0.

The maximum offset for the List all jobs and List job runs API requests is now limited

July 6, 2023

The maximum offset you can specify in the List all jobs and List job runs requests in the Jobs API is now limited. To avoid this limit when you use these requests, use token-based pagination which does not have this limit. See GET /api/2.1/jobs/list and GET /api/2.1/jobs/runs/list in the REST API reference.

Databricks Runtime 13.2 is GA

July 6, 2023

Databricks Runtime 13.2 and Databricks Runtime 13.2 ML are now generally available.

See Databricks Runtime 13.2 (EoS) and Databricks Runtime 13.2 for Machine Learning (EoS).

Delta Sharing and Databricks Marketplace support view sharing (Public Preview)

July 6, 2023

You can now share views in Delta Sharing and Databricks Marketplace. See Add views to a share.

Init scripts on DBFS reach end of life on Sept 1, 2023

July 5, 2023

On Sept 1, 2023, support for init scripts on DBFS will reach end of life and the feature will no longer function. Store init scripts in workspace files to ensure that they continue to function after Sept 1, 2023.