These features and Azure Databricks platform improvements were released in February 2025.
备注
The release date and content listed below only corresponds to actual deployment of the Azure Public Cloud in most case.
It provide the evolution history of Azure Databricks service on Azure Public Cloud for your reference that may not be suitable for Azure operated by 21Vianet.
备注
Releases are staged. Your Azure Databricks account might not be updated until a week or more after the initial release date.
February 28, 2025
The Databricks ODBC Driver version 2.9.1 is now available for download from the ODBC driver download page.
This release includes the following enhancements and new features:
- Added token renewal support for the token passthrough OAuth flow.
- Added a driver-side timeout option for idle statements. A timeout can be configured to prevent an application from interrupting results fetching and hanging in the middle of a query execution.
This release resolves the following issues:
- If the initial catalog is not set, the driver now defaults to using the
hive_metastorecatalog. - The driver updates the HTTP User-Agent from
thrift/0.9.0tothrift/0.17.0.
This release includes upgrades to several third-party libraries:
- OpenSSL 3.0.15 (previously 3.0.13)
- libcURL 8.10.1 (previously 8.7.1)
- ICU 74.2 (previously 71.1)
- Arrow 15.0.0 (previously 9.0.0) for OSX & Linux (Windows was on 15.0.0)
- lz4 1.10.0 (previously 1.9.4)
- Expat 2.6.3 (previously 2.6.2)
- Kerberos 1.21.3 (previously 1.21.2)
For complete configuration information, see the Databricks ODBC Driver Guide installed with the driver download package.
February 28, 2025
You can now publish event logs for DLT pipelines as tables in Unity Catalog. See Query the event log.
February 27, 2025
In Unity Catalog-enabled Azure Databricks workspaces, Databricks Assistant can provide table lineage and insights when you use simple prompts.
See Get table lineage using Databricks Assistant.
February 26, 2025
Unity Catalog-enabled Azure Databricks workspaces include an Import provider dialog for importing shares from non-Databricks Delta Sharing servers into Unity Catalog using Open-to-Databricks (O2D) sharing. This feature was released in July 2024 but was not announced at the time. When you use it to import a provider, your users have essentially the same recipient experience as they would with Databricks-to-Databricks sharing: click Create catalog to create a shared catalog, use Unity Catalog access controls to grant access to shared tables, and use standard Unity Catalog syntax to query those shares, with no need to store a credential file or specify it when you query shared data. See Azure Databricks: Read shared data using open sharing connectors.
February 26, 2025
Azure Databricks now supports OAuth Client Credentials for Open-to-Databricks (O2D) sharing, that is, from non-Databricks Delta Sharing servers to Azure Databricks workspaces that are enabled for Unity Catalog. This makes it easier to import shares from Delta Sharing provider networks like Oracle Delta Share that use OAuth Client Credentials. This support adds to the already considerable advantages of using Unity Catalog as a recipient in the O2D model, which include the Import provider UI for importing shares and the ability to use standard Unity Catalog syntax to query those shares, with no need to store a credential file or specify it when you query shared data. See Azure Databricks: Read shared data using open sharing connectors.
February 25, 2025
Streaming reads from Unity Catalog views registered against Delta tables are now generally available. See Read a Unity Catalog view as a stream.
February 25, 2025
The withEventTimeOrder option for processing Structured Streaming queries from Delta table sources is now generally available. See Process initial snapshot without data being dropped.
February 25, 2025
Asynchronous progress tracking for Structured Streaming is now generally available. See What is asynchronous progress tracking?.
February 25, 2025
The billable usage system table now includes fields to help monitor costs incurred from connecting serverless compute to your resources using private endpoints.
February 25, 2025
MLflow Tracing is now generally available. With Tracing, you can track inputs, outputs, and other metadata associated with each step of a model or agent request.
Tracing lets you pinpoint the source of bugs and unexpected behavior, compare performance across models or agents, and build new datasets to improve the quality.
February 25, 2025
Databricks Marketplace now includes an improved filter panel designed to streamline search and browsing. This update enhances discoverability, making it easier to search by product, provider, price, and more.
February 24, 2025
OAuth secrets for service principals now have a configurable lifetime. Newly created OAuth secrets default to a maximum lifetime of two years, whereas previously, they did not expire. See Authorize unattended access to Azure Databricks resources with a service principal using OAuth.
February 18, 2025
Volumes now display previews for common file formats in Catalog Explorer, including images, text files, JSON, yaml, and CSV. See Preview files in volumes.
February 18, 2025
To improve cost observability, billing records are now logged for every five minutes of model serving and provisioned throughput usage. Previously, the records were logged at one-hour intervals.
February 18, 2025
You can now enable automatic liquid clustering on Unity Catalog managed tables. Automatic liquid clustering intelligently selects clustering keys to optimize data layout for your queries. See Automatic liquid clustering.
February 10, 2025
Databricks Runtime 16.2 and Databricks Runtime 16.2 ML are now generally available.
See Databricks Runtime 16.2 and Databricks Runtime 16.2 for Machine Learning.
Support of Scala streaming foreach, foreachBatch, and flatMapGroupsWithState on standard access mode compute (formerly shared access mode)
February 7, 2025
Standard access mode compute now supports the Scala streaming function DataStreamWriter.foreach on Databricks Runtime 16.1 and above. On Databricks Runtime 16.2 and above, the functions DataStreamWriter.foreachBatch and KeyValueGroupedDataset.flatMapGroupsWithState are supported.
February 7, 2025
Service credentials enable simple and secure authentication with your cloud tenant's services from Azure Databricks. Service credentials are generally available and now support Scala SDKs, in addition to the Python SDK supported in the Public Preview. The fully-GA feature requires compute on Databricks Runtime 16.2 or above. See Manage access to external cloud services using service credentials.
February 6, 2025
For notebooks connected to SQL warehouses, you can now download cell results that contain tabular data as an Excel file. See Download results.
February 5, 2025
The DLT sink API is in Public Preview. With DLT sinks, you can write data transformed by your pipeline to targets like event streaming services such as Apache Kafka or Azure Event Hubs, and external tables managed by Unity Catalog or the Hive metastore. See Stream records to external services with DLT sinks.
February 5, 2025
You can now add users, service principals, and groups from Microsoft Entra ID into Azure Databricks without configuring an application in Microsoft Entra ID using automatic identity management. When automatic identity management is enabled, you can directly search in identity federated workspaces for Microsoft Entra ID users, service principals, and groups and add them to your workspace. Databricks uses Microsoft Entra ID as the source of record, so any changes to group memberships are respected in Azure Databricks. Automatic identity management also supports nested groups.
See Sync users and groups automatically from Microsoft Entra ID.