June 2024
These features and Azure Databricks platform improvements were released in June 2024.
Note
Releases are staged. Your Azure Databricks account might not be updated until a week or more after the initial release date.
Notebooks - out of order syntax highlighting
June 16, 2024
Python syntax error highlighting in notebooks now warns users when variables are used before they are declared and suggests a fix.
Notebooks Assistant automatically diagnoses errors
June 16, 2024
The Databricks Assistant now automatically diagnoses code errors in notebooks. This uses generative AI to analyze your code and the error message to suggest a fix directly in your notebook.
Catalog Explorer improvements
June 28, 2024
The following improvements have been made in Catalog Explorer:
Added a Quick access section that displays recent Unity Catalog objects, favorite Unity Catalog objects, and catalogs and schemas for browsing.
Removed the top bar and accordion structure, and moved the entry points for the following to a gear icon menu in the sidebar with the schema browser for simpler navigation:
- Compute selection
- Delta Sharing
- Storage credentials
- External locations
- Connections
- Browse DBFS
You can also access Delta Sharing and External data at the top of Catalog Explorer above the Quick access section.
Serve multiple external models from a single model serving endpoint
June 27, 2024
Mosaic AI Model Serving now supports serving multiple external models per model serving endpoint.
Databricks Assistant is now GA
June 27, 2024
Databricks Assistant is released as GA. Assistant capabilities include:
- Generate, debug, optimize, and explain code.
- Create visualizations from data. See Create visualizations with Databricks Assistant.
- Debug jobs (Public Preview feature).
- Code and edit SQL queries.
- Find relevant help in Azure Databricks documentation and knowledge base.
Databricks Assistant now gives titles to threads
June 27, 2024
To more easily navigate help threads, Databricks Assistant titles the threads based on the context of your prompt.
Notebooks debugging console
June 27, 2024
With the notebook debug console, you can now inspect live variable values and execute code snippets during a notebook debugging session.
Quickly add a periodic schedule to an Azure Databricks workflow
June 28, 2024
You can now add a simple schedule to an Azure Databricks workflow by specifying only an interval and unit of time (for example, run the workflow every 4 hours). This new schedule type complements the existing advanced schedule type by allowing you to add a schedule with just a few clicks. See Run jobs on a schedule.
Databricks Runtime 15.3 is GA
June 24, 2024
Databricks Runtime 15.3 and Databricks Runtime 15.3 ML are now generally available.
See Databricks Runtime 15.3 and Databricks Runtime 15.3 for Machine Learning.
Azure Geographies is GA
June 24, 2024
Azure Geographies is now generally available. Azure Geographies manages data residency when processing customer content for some AI capabilities.
New improved catalog creation UI
June 18, 2024
The new catalog creation UI takes you stepwise through the entire process of creating and configuring a catalog, providing helpful prompts and separating required from optional configurations. You can use the same user interface to create standard, shared, and foreign catalogs.
See Create catalogs.
Updates to subnets in network connectivity configurations
June 18, 2024
Databricks is updating the subnets in network connectivity configurations (NCCs) in all regions. The new subnet IDs are automatically added to existing Azure Storage firewall rules that allow access from the existing subnet IDs to ensure uninterrupted access from Azure Databricks.
If you use the Network Connectivity Configurations API to get the subnet IDs and add them to storage network rules for new Azure Storage accounts, you will not experience any interruptions. However, if you hardcode the existing set of subnet IDs, you might face disruptions. Databricks recommends using the API for the latest information. See the Network Connectivity Configurations API.
You might experience downtime for the Network Connectivity Configurations API while this update occurs between 8:00 PM and 12:00 AM PDT on June 20, 2024.
The billing system tables are enabled by default
June 18, 2024
The system.billing
schema is now enabled by default in all Unity Catalog workspaces.
Lakehouse Monitoring is GA
June 13, 2024
Databricks Lakehouse Monitoring is now generally available.
Volume sharing using Delta Sharing is now generally available
June 13, 2024
Delta Sharing support for sharing non-tabular data as a volume is now GA. You can share volumes between Databricks workspaces on different Unity Catalog metastores (including workspaces on different Databricks accounts and different clouds).
Volumes are Unity Catalog objects that represent a logical volume of storage in a cloud object storage location. They are intended primarily to provide governance over non-tabular data assets. Delta Sharing on Databricks provides a native integration with Unity Catalog that allows you to manage, govern, audit, and track usage of shared volumes data.
Customer managed keys (CMK) for Databricks Vector Search are now GA
June 12, 2024
Customer Managed Keys (CMK) on vector search endpoints are now generally available.
Mosaic AI Agent Framework (Public Preview)
June 12, 2024
Mosaic AI Agent Framework is a set of tools on Databricks designed to help developers build, deploy, and evaluate production-quality Retrieval Augmented Generation (RAG) applications. Agent Framework enables you to create or optimize LLM applications on a single platform using your own data.
Agent Framework includes the following capabilities:
- Create and log agents and chains.
- Parameterize agents and chains to experiment and iterate quickly.
- Built-in metrics to compare runs across retrieval of relevant documents, response accuracy, and cost and latency.
- Custom LLM judges for retrieval and response accuracy.
- Deploy AI applications with automated request and response logging.
- Built-in review app to get user feedback.
- Agent Evaluation to evaluate the quality, latency, and cost of generative AI applications.
- MLflow Tracing to analyze and compare traces across code execution.
Predictive optimization is now GA
June 11, 2024
Predictive optimization for Unity Catalog managed tables is now generally available. Predictive optimization now runs OPTIMIZE
on tables with liquid clustering enabled to apply clustering.
Route optimization is available for serving endpoints
June 11, 2024
You can now create route optimized serving endpoints for your model serving or feature serving workflows.
Model Serving support with a workspace storage account firewall
June 11, 2024
Model Serving is now supported with firewall support for the workspace storage account.
Column mapping is now GA
June 10, 2024
Delta Lake column mapping is now generally available. You can now use DROP FEATURE
to disable column mapping and downgrade table protocol. See Rename and drop columns with Delta Lake column mapping.
Workflow system tables are now available (Public Preview)
June 10, 2024
The system.lakeflow
schema contains four new tables you can use to monitor jobs in your account.
If you don't have access to these system tables, ensure you have enabled the system.lakeflow
schema in your account.
Function calling is Public Preview
June 10, 2024
Function calling is now available in Public Preview. This functionality is available using Foundation Model APIs pay-per-token models: DBRX Instruct and Meta-Llama-3-70B-Instruct.
Query a vector search index using vector_search()
June 10, 2024
AI Functions now supports vector_search()
which allows you to query a Mosaic AI Vector Search index using SQL.
Mosaic AI Vector Search now supports hybrid search
June 7, 2024
Mosaic AI Vector Search now supports hybrid keyword-similarity search, which combines vector-based embedding search with traditional keyword-based search techniques. This approach matches exact words in the query while also using a vector-based similarity search to capture the semantic relationships and context of the query. By integrating these two techniques, hybrid keyword-similarity search retrieves documents that contain not only the exact keywords but also those that are conceptually similar, providing more comprehensive and relevant search results.
GTE is now supported in Model Serving
June 7, 2024
Mosaic AI Model Serving now supports General Text Embedding (GTE), a text embedding model that can map text to an embedding vector. GTE is supported as part of Foundation Model APIs and is available in pay-per-token serving endpoint regions.
Databricks Assistant: Threads & queries experience enabled by default
June 6, 2024
Databricks Assistant (Public Preview) now tracks query threads and history throughout editor contexts in your session experience by default. You can manage enablement for all workspaces in an account or individual workspaces.
Improved notebook Markdown editor
June 5, 2024
You can now see a live preview of your Markdown cells in notebooks and access a toolbar for common Markdown elements like headers, lists, and links. Spend more time communicating your ideas and less time remembering syntax.
Quickly access Catalog Explorer tables from notebooks
June 5, 2024
You can now easily access Catalog Explorer tables directly from notebooks. Hover over a table name in a notebook cell to get a link to its catalog explorer page to see details like columns, permissions, and more.
Databricks ODBC driver 2.8.2
June 4, 2024
Databricks ODBC Driver (download) version 2.8.2 is now available. This release adds the following new features and enhancements:
- Customers can now use extended SQL
PUT
,GET
, andREMOVE
commands to ingest and manage large datasets in Unity Catalog volumes. - Default OAuth configuration values are now updated and compliant. The updated configuration options include the authorization endpoint, OAuth client ID, OAuth user-to-machine scope, and OAuth machine-to-machine scope. For more details, see the release notes.
- Third-party libraries have been updated, including expat 2.6.2 (previously 2.6.0) and libCURL 8.7.1 (previously 8.6.0).
- The driver now accepts a private signed certification authority (CA) if it is added to the Windows Trusted Root CA certificate store.
This release also fixes the following issues:
- The client previously returned an error if a table comment's length exceeded
MaxCommentLen
. The default behavior now truncates the table comment to conform to the limit specified inMaxCommentLen
. - The client previously returned an error when issuing
SQLCancel
under certain circumstances.
The new Databricks Notebook UI is now generally available
June 4, 2024
The new Databricks Notebooks UI provides developers with a streamlined, data-focused authoring experience. Key features include:
- Modern UX: Enjoy a streamlined coding experience with an updated UI and other features to enhance Notebook organization.
- New results table: Perform no-code data exploration with search and filtering capabilities directly on result outputs.