May 2025

These features and Azure Databricks platform improvements were released in May 2025.

New alerts in Beta

May 22, 2025

A new version of Databricks SQL alerts is now in Beta. You can use alerts to periodically run queries, evaluate defined conditions, and send notifications if a condition is met. This version simplifies creating and managing alerts by consolidating query setup, conditions, schedules, and notification destinations into a single interface. You can still use legacy alerts alongside the new version. See Databricks SQL alerts.

Pipelines system table is now available (Public Preview)

May 20, 2025

The system.lakeflow.pipelines table is a slowly changing dimension table (SCD2) that tracks all pipelines created in your Azure Databricks account.

Azure Databricks supports cross-cloud read access to AWS S3 storage

May 20, 2025

Use Unity Catalog to access and govern all of your S3 data from within a secure Azure Databricks environment. There is no need to migrate or copy datasets. Access to S3 is read-only.

Databricks Asset Bundles in the workspace (Public Preview)

May 19, 2025

Collaborating on Databricks Asset Bundles with other users in your organization is now easier with bundles in the workspace, which allows workspace users to edit, commit, test, and deploy bundle updates through the UI.

Workflow task repair now respects transitive dependencies

May 19, 2025

Previously, repaired tasks were unblocked once their direct dependencies completed. Now, repaired tasks wait for all transitive dependencies. For example, in a graph A → B → C, repairing A and C will block C until A finishes.

Databricks Apps (Generally Available)

May 13, 2025

Databricks Apps is now generally available (GA). This feature lets you build and run interactive full-stack applications directly in the Databricks workspace. Apps run on managed infrastructure and integrate with Delta Lake, notebooks, ML models, and Unity Catalog.

Databricks Runtime 16.4 LTS is GA

May 13, 2025

Databricks Runtime 16.4 and Databricks Runtime 16.4 ML are now generally available.

See Databricks Runtime 16.4 LTS and Databricks Runtime 16.4 LTS for Machine Learning.

Model Serving CPU workloads now support compliance security profile standards

May 13, 2025

Model Serving CPU workloads now support the following compliance standards offered through the compliance security profile:

• HIPAA
• HITRUST
• PCI-DSS
• ISMAP
• UK Cyber Essentials Plus

Databricks JDBC driver 2.7.3

May 12, 2025

The Databricks JDBC Driver version 2.7.3 is now available for download from the JDBC driver download page.

This release includes the following enhancements and new features:

• Added support for Azure Managed Identity OAuth 2.0 authentication. To enable this, set the Auth_Flow property to 3.
• Added support for OAuth Token exchange for IDPs different than host. OAuth access tokens (including BYOT) will be exchanged for a Databricks access token.
• OAuth browser (Auth_Flow=2) now supports token caching for Linux and Mac operating systems.
• Added support for VOID, Variant, and TIMESTAMP_NTZ data types in getColumns() and getTypeInfo() APIs.
• The driver now lists columns with unknown or unsupported types and maps them to SQL VARCHAR in the getColumns() metadata API.
• Added support for cloud.databricks.us and cloud.databricks.mil domains when connecting to Databricks using OAuth (AuthMech=11).
• Upgraded to netty-buffer 4.1.119 and netty-common 4.1.119 (previously 4.1.115).

This release resolves the following issues:

• Compatibility issues when deserializing Apache Arrow data with Java JVMs version 11 or higher.
• Issues with date and timestamp before the beginning of the Gregorian calendar when connecting to specific Spark versions with Arrow result set serialization.

For complete configuration information, see the Databricks JDBC Driver Guide installed with the driver download package.

Updated lineage system tables schema

May 11, 2025

The lineage system tables (system.access.column_lineage and system.access.table_lineage) have been updated to better log entity information.

• The entity_metadata column replaces entity_type, entity_run_id, and entity_id, which have been deprecated.
• The record_id column is a new primary key for the lineage record.
• The event_id column logs an identifier for lineage event, which can be shared by multiple rows if they were generated by the same event.
• The statement_id column logs the query statement ID of the query that generated the lineage event. It is a foreign key that can be joined with the system.query.history table.

You can now collaborate with multiple parties with Azure Databricks Clean Rooms

May 14, 2025

Azure Databricks Clean Rooms now supports:

• Up to 10 collaborators for more complex, multi-party data projects.
• New notebook approval workflow that enhances security and compliance, allowing designated runners, requiring explicit approval before execution
• Auto-approval options for trusted partners.
• Difference views for easy review and auditing.

These updates enable more secure, scalable, and auditable collaboration.

Azure Databricks GitHub App adds workflow scope to support authoring GitHub Actions

May 9, 2025

Azure Databricks made a change where you may get an email request for the Read and write access to Workflows scope for the Azure Databricks GitHub app. This change makes the scope of the Azure Databricks GitHub app consistent with the required scope of other supported authentication methods and allows users to commit GitHub Actions from Azure Databricks Git folders using the Azure Databricks GitHub app for authorization.

If you are the owner of an Azure Databricks account where the Azure Databricks GitHub app is installed and configured to support OAuth, you may receive the following notification in an email titled "Databricks is requesting updated permissions" from GitHub. (This is a legitimate email request from Databricks.) Accept the new permissions in order to enable committing GitHub Actions from Azure Databricks Git folders with the Databricks GitHub app.

Automatically provision users (JIT) GA

May 9, 2025

You can now enable just-in-time (JIT) provisioning to automatically creates new user accounts during first-time authentication. When a user logs in to Azure Databricks for the first time using single sign-on (SSO), Azure Databricks checks if the user already has an account. If not, Azure Databricks instantly provisions a new user account using details from the identity provider. See Automatically provision users (JIT).

Query snippets are now available in the new SQL editor, notebooks, files, and dashboards

May 9, 2025

Query snippets are segments of queries that you can share and trigger using autocomplete. You can now create query snippets through the View menu in the new SQL editor, and also in the notebook and file editors. You can use your query snippets in the SQL editor, notebook SQL cells, SQL files, and SQL datasets in dashboards.

See Query snippets.

You can now create views in ETL pipelines

May 8, 2025

The CREATE VIEW SQL command is now available in ETL pipelines. You can create a dynamic view of your data. See CREATE VIEW (DLT).

Configure Python syntax highlighting in Databricks notebooks

May 8, 2025

You can now configure Python syntax highlighting in notebooks by placing a pyproject.toml file in the notebook's ancestor path or your home folder. Through the pyproject.toml file, you can configure ruff, pylint, pyright, and flake8 linters, as well as disable Databricks-specific rules. This configuration is supported for clusters running Databricks Runtime 16.4 or above, or Client 3.0 or above.

See Configure Python syntax highlighting.

AI Builder: Model specialization for custom gen AI text agents is in Beta

May 7, 2025

AI Builder provides a simple, no-code approach to build and optimize domain-specific, high-quality AI agent systems for common AI use cases. In Beta, AI Builder supports model specialization for custom text-based tasks, such as summarization, classification, text transformation, and content generation.

Jobs and pipelines now share a single, unified view (Public Preview)

May 7, 2025

You can now view all workflows, including jobs, ETL pipelines, and ingestion pipelines, in a single unified list. See View jobs and pipelines.

Predictive optimization enabled for all existing Azure Databricks accounts

May 7, 2025

Starting May 7, 2025, Databricks enabled predictive optimization by default for all existing Azure Databricks accounts. This will roll out gradually based on your region and will be completed by July 1, 2025. When predictive optimization is enabled, Azure Databricks automatically runs maintenance operations for Unity Catalog managed tables.

File events for external locations improve file notifications in Auto Loader and file arrival triggers in jobs (Public Preview)

May 5, 2025

You can now enable file events on external locations that are defined in Unity Catalog. This makes file arrival triggers in jobs and file notifications in Auto Loader more scalable and efficient.

This feature is in Public Preview. Auto Loader support for file events requires enablement by an Azure Databricks representative. For access, reach out to your Azure Databricks account team.

For details, see the following:

• (Recommended) Enable file events for an external location
• File notification mode with and without file events enabled on external locations
• Trigger jobs when new files arrive

Mosaic AI Model Serving region expansion

May 5, 2025

Mosaic AI Model Serving is now available in the following regions:

• eastasia
• francecentral
• germanywestcentral
• koreacentral
• swedencentral
• switzerlandnorth
• uaenorth
• westus3

The jobs system tables (Public Preview) are enabled by default

May 1, 2025

The system.lakeflow schema, which contains system tables related to jobs, is now enabled by default in all Unity Catalog workspaces.