August 2024
These features and Azure Databricks platform improvements were released in August 2024.
Note
The release date and content listed below only corresponds to actual deployment of the Azure Public Cloud in most case.
It provide the evolution history of Azure Databricks service on Azure Public Cloud for your reference that may not be suitable for Azure operated by 21Vianet.
Note
Releases are staged. Your Azure Databricks account might not be updated until a week or more after the initial release date.
Importable cost management dashboard added to account console (Public Preview)
August 28, 2024
Account admins can now import a customizable cost management dashboard from the account console into any Unity Catalog workspace in their account. There are two versions of the dashboard: one that monitors account-wide usage, and another for single-workspace usage.
Delta Sharing: More Delta Lake features now supported by the Python and Power BI connectors
August 21, 2024
Delta Sharing Python connector 1.1.0+ and Power BI v2.132.908.0+ now support:
- Column mapping name mode
- Deletion vectors
- Uniform format
These Delta Lake features were already supported on Databricks Runtime 14.1+ and open-source Delta Sharing Apache Spark connector 3.1+.
See Delta Lake feature support matrix.
Delta Sharing adds support for TimestampNTZ
August 21, 2024
Delta Sharing adds support for TimestampNTZ on Databricks Runtime 14.1 and above and open-source Delta Sharing Apache Spark connector 3.3 and above.
See Delta Lake feature support matrix.
The Azure Databricks Jobs For each
task is GA
August 21, 2024
The For each
task is now generally available. You can use the For each
task to run another task in a loop, passing a different set of parameters to each iteration of the task. The For each
task can iterate over any of the standard job tasks, such as a notebook, JAR, Python script, or SQL task. See Run a parameterized Azure Databricks job task in a loop.
Azure RBAC permission for workspace admins
August 20, 2024
Previously, the built-in Azure RBAC (role-based access control) Contributor or Owner role was required to grant a user the Azure Databricks workspace admin role from Azure. You can now create a custom Azure RBAC role that has the permission Microsoft.Databricks/workspaces/assignWorkspaceAdmin/action
to grant a user the Azure Databricks workspace admin role. These users can manage their Azure Databricks service and configure diagnostic logging. For more information, see Required Azure admin permissions.
Databricks Runtime 15.4 LTS is GA
August 19, 2024
Databricks Runtime 15.4 LTS and Databricks Runtime 15.4 LTS ML are now generally available.
See Databricks Runtime 15.4 LTS and Databricks Runtime 15.4 LTS for Machine Learning.
Personalized notebook autocomplete
August 19, 2024
Notebook autocomplete now prioritizes suggestions based on your individual Unity Catalog metadata and usage, providing custom suggestion rankings for each user. See Personalized autocomplete
Configure your workspace's default access mode for jobs compute
August 16, 2024
Workspace admins can now configure the default access mode for jobs compute in their workspace. This default access mode is applied to compute resources without a defined access mode. For more information, see Default access mode for jobs compute.
New slash commands for Databricks Assistant
August 14, 2024
Databricks Assistant has added the following slash commands as shortcuts for common tasks:
/findTables
: Searches for relevant tables based on Unity Catalog metadata./findQueries
: Searches for relevant queries based on Unity Catalog metadata./prettify
: Formats code for readability./rename
: Suggests updated names to notebook cells and other elements, depending on the context./settings
: Adjusts your notebook settings directly from Assistant.
Workspace search now supports volumes
August 14, 2024
Volumes are now included in search results. See Search for workspace objects.
Meta Llama 3.1 405B models supported in Mosaic AI Model Training
August 14, 2024
Meta Llama 3.1 405B models are now supported in Mosaic AI Model Training.
Databricks JDBC driver 2.6.40
August 13, 2024
Databricks JDBC Driver version 2.6.40 is now available from JDBC driver download. This release removes redundant WARNING
log messages to increase logging usability and security.
This release includes the following enhancements and new features:
- OIDC discovery endpoint support. The driver can now set an OIDC discovery endpoint to fetch a token and retrieve an authorization endpoint.
- Updated Arrow support. The driver now uses Apache Arrow version 14.0.2. Earlier versions of the driver used Apache Arrow version 9.0.0.
ProxyIgnoreList
support. The driver now supportsProxyIgnoreList
property whenUseProxy
is set to1
.- Refresh token support. The driver now supports an optional refresh token. It saves the access token and reuses it for new connections as long as it is valid. If the driver cannot renew the access token using the refresh token, it will sign in again.
- Updated authentication support. The driver now supports browser-based (U2M) and client credentials (M2M) authentication on Google Cloud.
- Added unified default OAuth options.
- You can now configure the OAuth redirect port. To do this, set the
OAuth2RedirectUrlPort
property to your port.
For complete configuration information, see the Databricks JDBC Driver Guide installed with the driver download package.
Databricks personal access tokens revoked if unused after 90 days
August 13, 2024
Databricks now automatically revokes any personal access tokens (PATs) that have not been used in 90 or more days. For more details, see Automatic revocation of old access tokens.
Clusters API now supports partial configuration updates
August 13, 2024
A new API call allows you to partially update a cluster configuration, requiring you to specify only the attributes you want to update. See Update cluster configuration (partial) in the REST API reference.
Wrap lines in notebook cells
August 12, 2024
You can now enable or disable line wrapping in notebook cells, allowing text to either wrap onto multiple lines or remain on a single line with horizontal scrolling. See Line wrapping.
Files can no longer have identical names in workspace folders
August 9, 2024
Databricks now prevents you from creating or renaming assets in workspace folders when an asset's name matches another file's name exactly, accounting for the asset's file extension. For example, you can no longer create a file named test.py
if there is already a notebook with a base name of test
with an extension of .py
in the same workspace folder.
For more details, see Naming assets in workspace folders.
Compute policy enforcement now available
August 8, 2024
Policy compliance enforcement enables workspace admins to update their workspace's compute resources to comply with the latest version of a policy. This feature can be used in the UI or through the Cluster Policies API.
See Enforce policy compliance or Cluster Policies API.
Collaborate on data projects securely and privately using Databricks Clean Rooms (Public Preview)
August 6, 2024
Databricks Clean Rooms uses Delta Sharing and serverless compute to provide a secure and privacy-protecting environment where multiple parties can share sensitive enterprise data and collaborate without direct access to each other's data.
With Clean Rooms, users from other Databricks accounts can collaborate to generate unique insights about shared projects, such as advertising campaigns, investment decisions, or research and development, without moving or exposing sensitive data. Run complex workloads in an ephemeral environment using any language supported by Databricks notebooks, including Python, which provides native support for machine learning workloads.
To try it out, reach out to your Azure Databricks representative.
Mosaic AI Vector Search is now HIPAA-compliant
August 6, 2024
Mosaic AI Vector Search is now HIPAA-compliant in all regions.
Row filters and column masks are now GA, with improvements
August 6, 2024
The ability to apply row filters and column masks to tables is now generally available on Databricks Runtime 12.2 and above. Row filters and colum masks prevent access to sensitive data by specified users. These filters and masks are implemented as SQL user-defined functions (UDFs). GA brings support for the following functionality that was not available in the public preview:
Constant expressions in policy parameters (strings, numeric, intervals, booleans, nulls).
Materialized views and streaming tables (Public Preview).
MERGE statements.
Table sampling.
The separate release of support for fine-grained access control on single-user compute also expands compute options for working on tables with row filters and column masks applied.
See Filter sensitive table data using row filters and column masks.
Lakehouse Federation is generally available (GA)
August 1, 2024
In Databricks Runtime 15.2 and later and Databricks SQL version 2024.30 and later, Lakehouse Federation connectors across the following database types are generally available (GA):
- MySQL
- PostgreSQL
- Amazon Redshift
- Snowflake
- Microsoft SQL Server
- Azure Synapse (SQL Data Warehouse)
- Databricks
This release also introduces the following improvements:
Support for single sign-on (SSO) authentication in the Snowflake and Microsoft SQL Server connectors.
Support for additional pushdowns (string, math, and miscellaneous functions).
Improved pushdown success rate across different query shapes.
Additional pushdown debugging capabilities:
- The
EXPLAIN FORMATTED
output displays the pushed-down query text. - The query profile UI displays the pushed-down query text, federated node identifiers, and JDBC query execution times (in verbose mode). See View system-generated federated queries.
- The