Azure Databricks platform release notes
Note
The release date and content listed below only corresponds to actual deployment of the Azure Public Cloud in most case.
It provide the evolution history of Azure Databricks service on Azure Public Cloud for your reference that may not be suitable for Azure operated by 21Vianet.
-
DBRX Base and DBRX Instruct are now available in Model Serving
Provisioned throughput in Foundation Model APIs is GA and HIPAA compliant
The Jobs UI is updated to better manage jobs deployed by Databricks Asset Bundles
Self-service sign-up for private exchange providers in Marketplace
Delegate the ability to view an object's metadata in Unity Catalog (Public Preview)
-
- Use Delta Live Tables in Feature Engineering (Public Preview)
- Source IPv4 addresses for PrivateLink connections in audit logs
- Restrict changing a job owner and the run as setting
- Support for Databricks managed service principals
- Enable automatic cluster update (Public Preview)
- Admins can enable enhanced security and compliance features (Public Preview)
- Support for Cloudflare R2 storage to avoid cross-region egress fees (Public Preview)
- Add data UI supports XML file format
- Use AI Functions to invoke a generative AI model from Foundation Model APIs
- Unity Catalog volumes are GA
- Full-page AI-powered search
- Run SQL notebook jobs on a SQL warehouse
- File arrival triggers in Azure Databricks Workflows is GA
- Search for machine learning models in Unity Catalog using global workspace search
- Databricks Git server proxy is GA
- Databricks Git server proxy no longer requires CAN_ATTACH_TO permissions
- Workspace file support for the dbt and SQL file tasks is GA
- Databricks Connect is GA for Scala
- Create tables from files in volumes
- Databricks Runtime 14.3 LTS is GA
- Delta Sharing supports tables that use deletion vectors (Public Preview)
-
- Native XML file format support (Public Preview)
- Share AI models using Databricks Marketplace (Public Preview)
- Updates to network security group rules
- Workspace path update
- Support for Azure Storage firewall from serverless compute is GA
- Streamlined creation of Azure Databricks jobs
- Monitor GPU model serving workloads using inference tables
- URI path-based access to Unity Catalog external volumes
- Access controls lists can be enabled on upgraded workspaces
- Marketplace listing events system table now available (Public Preview)
- Updated UI for notebook cells (Public Preview)
- Quick Fix help with syntax errors in the notebook
- Databricks Runtime 14.3 LTS (Beta)
- Share AI models using Delta Sharing (Public Preview)
- Databricks Marketplace supports volume sharing
- Create widgets from the Databricks UI
- Warehouse events system table is now available (Public Preview)
- Workspace file size limit is now 500MB
- Feature removal notice for legacy Git integration in Databricks
- Databricks ODBC driver 2.7.7
- Databricks Runtime 13.2 series support ends
- AI assistive features are enabled by default
-
- Share dynamic views using Delta Sharing (Public Preview)
- Share volumes using Delta Sharing (Public Preview)
- Entity Relationship Diagram for primary keys and foreign keys
- Unity Catalog volume file upload size limit increase
- New notebook cell results rendering available (Public Preview)
- Notebook editor themes available
- Repos & Git Integration Settings UI now correctly notes support for GitHub Enterprise Server
- Databricks JDBC driver 2.6.36
- Support for referencing workspace files from init scripts
- Feature & Function Serving is Public Preview
- Foundation Model APIs is Public Preview
- New unified admin settings UI
- Init scripts on DBFS are end-of-life
- Legacy global and cluster-named init scripts are end-of-life
-
- Databricks Vector Search is Public Preview
- Access controls lists can no longer be disabled
- New behaviors and actions in Catalog Explorer for volumes
- Databricks Runtime 14.2 is GA
- Databricks SQL Connector for Python version 3.0.0
- Libraries in workspace files supported on no-isolation shared clusters
- Deprecation of workspace libraries
- Support for Azure confidential computing VM types is now GA
- Search for Databricks Marketplace listings using global workspace search
- Consume data products in Databricks Marketplace using external platforms
- Support for serverless SQL warehouses added for several Azure regions
- Automatic enablement of Unity Catalog for new workspaces
- Databricks Runtime 14.2 (beta)
- Databricks Marketplace includes Databricks Solution Accelerators
- Lakehouse Federation adds support for Google BigQuery
-
- View the YAML source for an Azure Databricks job
- Add conditional logic to your Azure Databricks workflows
- Configure parameters on an Azure Databricks job that can be referenced by all job tasks
- Auto-enable deletion vectors
- Unity Catalog support for
UNDROP TABLEis GA
- Databricks AutoML generated notebooks are now saved as MLflow artifacts
- New region: Qatar Central
- Unity Catalog volumes supported with Azure Firewall and Private Link
- Predictive optimization (Public Preview)
- Compute system tables are now available (Public Preview)
- On-demand feature computation is GA
- Feature Engineering in Unity Catalog is GA
- AI-generated table comments (Public Preview)
- Models in Unity Catalog is GA
- Libraries now supported in compute policies (Public Preview)
- Partner Connect supports Monte Carlo
- Semantic search (Public Preview)
- Enable Databricks Assistant at the workspace level
- New Photon defaults
- Databricks Runtime 14.1 is GA
- Developer tools release notes have moved
- Databricks extension for Visual Studio Code updated to version 1.1.5
- Predictive I/O for updates is GA
- Deletion vectors are GA
- Partner Connect supports RudderStack
- Databricks CLI updated to version 0.207.0 (Public Preview)
- Run selected cells in a notebook
- Use workspace-catalog binding to give read-only access to a catalog
- Databricks extension for Visual Studio Code updated to version 1.1.4
- Databricks SDK for Python updated to version 0.10.0 (Beta)
- Databricks SDK for Go updated to version 0.22.0 (Beta)
- Unity Catalog volumes now in Public Preview for Azure China
-
- Lakehouse Federation available on single-user clusters
- Delegate the ability to create a storage credential in Unity Catalog
- Databricks Runtime 14.1 (Beta)
- GPU model serving optimized for LLMs in Public Preview
- Prevent runs of a job from being skipped when you reach concurrency limits
- Databricks Terraform provider updated to version 1.27.0
- Databricks CLI updated to version 0.206.0 (Public Preview)
- Databricks SDK for Go updated to version 0.21.0 (Beta)
- Inference tables for model serving endpoints is Public Preview
- Databricks ODBC driver 2.7.5
- Databricks extension for Visual Studio Code updated to version 1.1.3
- Running jobs as a service principal is GA
- Databricks CLI updated to version 0.205.2 (Public Preview)
- Databricks CLI updated to version 0.205.1 (Public Preview)
- Databricks SDK for Go updated to version 0.20.0 (Beta)
- Databricks SDK for Python updated to version 0.9.0 (Beta)
- Databricks SDK for Python updated to version 0.7.1 (Beta)
- Databricks Terraform provider updated to version 1.26.0
- Databricks Connect for Databricks Runtime 14.0
- Lakeview dashboards in Public Preview
- Databricks Connect V2 is Public Preview for Scala
- GPU model serving in Public Preview
- On-demand feature computation now available in Unity Catalog (Public Preview)
- Databricks Runtime 14.0 is GA
- Account email notifications upgrade
- Databricks Terraform provider updated to version 1.25.1
- Databricks SDK for Go updated to version 0.19.2 (Beta)
- Databricks CLI updated to version 0.205.0 (Public Preview)
- Reserved location for
/Workspace - Databricks Asset Bundles (Public Preview)
- Databricks Terraform provider updated to version 1.25.0
- Databricks extension for Visual Studio Code updated to version 1.1.2
- Databricks CLI updated to version 0.204.1 (Public Preview)
- Filter sensitive data with row filters and column masks (Public Preview)
- System tables now include Marketplace schema (Public Preview)
- Pricing system table is now available (Public Preview)
- Databricks SDK for Go updated to version 0.19.1 (Beta)
- Databricks ODBC driver 2.7.3
- Databricks CLI updated to version 0.204.0 (Public Preview)
- Data Explorer is now Catalog Explorer
- Databricks SDK for Go updated to version 0.19.0 (Beta)
- Databricks SDK for Python updated to version 0.8.0 (Beta)
- Delegate allowlist privileges in Unity Catalog
- GitHub apps integration in Repos is GA
-
- Tables now appear in navigational search
- Databricks CLI updated to version 0.203.3 (Public Preview)
- Databricks JDBC driver 2.6.34
- Databricks SDK for Go updated to version 0.18.0 (Beta)
- Databricks SDK for Python updated to version 0.7.0 (Beta)
- Databricks Terraform provider updated to version 1.24.1
- Init scripts on DBFS end of life extended to Dec 1, 2023
- Databricks Runtime 14.0 (Beta)
- Unified navigation experience is GA
- Databricks Terraform provider updated to version 1.24.0
- Databricks Runtime for Genomics setting removed from the workspace admin settings page
- Container Services setting removed from the workspace admin settings page
- Databricks CLI updated to version 0.203.2 (Public Preview)
- Go to definition for functions and variables in Python notebooks
- Unified schema browser is GA
- Databricks Runtime 13.3 LTS is GA
- Introducing tags with Unity Catalog
- Allowlist for init scripts, JARs, and Maven coordinates on Unity Catalog shared clusters is in Public Preview
- Volumes support for init scripts and JARs is in Public Preview
- Easier Databricks Repos .ipynb file output commits
- IPYNB notebook support in Databricks Repos is GA
- Databricks SDK for Go updated to version 0.17.0 (Beta)
- Databricks SDK for Python updated to version 0.6.0 (Beta)
- Databricks CLI updated to version 0.203.1 (Public Preview)
- Unified Schema Browser is now GA
- Databricks SDK for Python updated to version 0.5.0 (Beta)
- Programmatic write support for workspace files
- Databricks CLI updated to version 0.203.0 (Public Preview)
- Databricks Terraform provider updated to version 1.23.0
- Groups can now be renamed
- Databricks SDK for Go updated to version 0.15.0 (Beta)
- Databricks SDK for Python updated to version 0.4.0 (Beta)
- Third-party iFraming prevention configuration setting was removed
- Databricks extension for Visual Studio Code updated to version 1.1.1
- LangChain available in 13.1 and above
- Feature Engineering in Unity Catalog is Public Preview
- Improved error handling for repeated continuous job failures
- Share schemas using Delta Sharing (Public Preview)
- Run tasks conditionally in your Azure Databricks jobs
- Databricks Terraform provider updated to version 1.22.0
- Enhanced security and compliance features
- Lakehouse Monitoring is Public Preview
- Databricks Runtime 13.3 LTS (Beta)
- New Git operations are generally available: Merge branches, rebase and pull with conflict resolution
-
- Workspace admins can now create account groups
- Group manager role is in Public Preview
- Databricks CLI updated to version 0.202.0 (Public Preview)
- Databricks SDK for Python updated to version 0.3.0 (Beta)
- Databricks SDK for Go updated to version 0.14.1 (Beta)
- Databricks SDK for Go updated to version 0.14.0 (Beta)
- Run another job as a task in an Azure Databricks job
- Workspace network update gets UI support (GA)
- All users can access data products in Databricks Marketplace by default
- Classic keyboard shortcuts mode
- Lakehouse Federation lets you run queries against external database providers (Public Preview)
- Move to trash enabled for Repos
- Create alerts for slow-running or stuck jobs
- Databricks SDK for Go updated to version 0.13.0 (Beta)
- Databricks SDK for Python updated to version 0.2.0 (Beta)
- Databricks CLI updated to version 0.201.0 (Public Preview)
- Databricks SDK for Python updated to version 0.2.1 (Beta)
- Databricks Assistant is in Public Preview
- Deactivate users and service principals from your account
- Account-level SCIM provisioning now deactivates users when they are deactivated in the identity provider
- Trash directory admin access
- Prevention of MIME type sniffing and XSS attack page rendering are now always enabled
- Unity Catalog volumes are in Public Preview
- Simplified experience for submitting product feedback from the workspace
- Databricks extension for Visual Studio Code updated to version 1.1.0
- Functions now displayed in Catalog Explorer (Public Preview)
- Databricks Terraform provider updated to version 1.21.0
- The maximum offset for the
List all jobsandList job runsAPI requests is now limited - Databricks Runtime 13.2 is GA
- Delta Sharing and Databricks Marketplace support view sharing (Public Preview)
- Init scripts on DBFS reach end of life on Sept 1, 2023
-
- Databricks Terraform provider updated to version 1.20.0
- Databricks CLI updated to version 0.200.1 (Public Preview)
- Databricks SDK for Go updated to version 0.12.0 (Beta)
- Databricks SDK for Go updated to version 0.11.0 (Beta)
- Databricks SDK for Python (Beta)
- Databricks SDK for Go (Beta)
- Access audit log, billable usage, and lineage system tables (Public Preview)
- Models in Unity Catalog (Public Preview)
- Databricks Marketplace is now GA
- Databricks Runtime 13.2 (Beta)
- Databricks extension for Visual Studio Code (General Availability)
- Enable customer-managed keys using Azure CLI and Powershell without a template
- See a visual overview of completed job runs in the Azure Databricks Jobs UI
- Databricks CLI (Public Preview)
- Improved pagination of results from
List all jobsandList job runsAPI requests - Full-page workspace browser includes Repos
- Databricks Terraform provider updated to version 1.19.0
- Run jobs as a service principal (Public Preview)
- New service principal UI provides better management experience
- Home folders restored when users are re-added to workspaces
- Databricks Marketplace: Private exchanges are now available
- Databricks Marketplace: Consumers can uninstall data products using the UI
- Databricks Marketplace: providers can create their own profiles
- Databricks Connect V2 is GA for Python
- Databricks Terraform provider updated to version 1.18.0
- New Databricks Marketplace providers
- Databricks Runtime 13.1 is GA
- View data from notebooks, SQL editor, and Catalog Explorer with the unified schema browser (Public Preview)
-
- Create or modify table UI supports Avro, Parquet, and text file uploads
- Use the add data UI to load data using Unity Catalog external locations (Public Preview)
- Run Databricks notebooks on SQL warehouses (Public Preview)
- Prevent Enter key from accepting autocomplete suggestion
- New region: China East 2 3
- Support for Azure confidential computing VM types (Public Preview)
- Databricks Terraform provider updated to version 1.17.0
- Upload data UI supports new column data types
- Bind Unity Catalog catalogs to specific workspaces
- New default theme for editor
- Databricks Terraform provider updated to version 1.16.0
- Unity Catalog support for Azure managed identities is now GA
- Databricks Runtime 13.1 (Beta)
- Run file-based SQL queries in an Azure Databricks workflow
- Databricks Terraform provider updated to version 1.15.0
- Azure operated by 21Vianet regions now support Unity Catalog
- Account nicknames now available in the account console
- Share notebooks using Delta Sharing
- Azure Databricks no longer creates a serverless starter SQL warehouse
- In SQL Warehouses API, enabling serverless compute now must be explicit
- Changes for workspace settings for serverless SQL warehouses
- Changes for serverless compute settings for accounts and workspaces
-
- Databricks Marketplace (Public Preview): an open marketplace for data, analytics, and AI
- Configure the Python formatter
- Workspace files are GA
- Cluster-scoped init scripts can now be stored in workspace files
- Create an Azure Key Vault-backed secret scope with a service principal
- New Delta Sharing privileges enable delegation of share, recipient, and provider management tasks
- Databricks Connect V2 (Public Preview)
- New cluster metrics UI
- Databricks Runtime 13.0 is GA
- Control access to the account console by IP address ranges (Public Preview)
- Databricks Terraform provider updated to version 1.14.3
- Diagnostic log entries for changed admin settings for workspace
- For initial customer-managed key for managed disks, Azure Databricks now terminates compute resources
- Databricks Terraform provider updated to versions 1.14.1 and 1.14.2
- Combined SQL user settings and general user settings
- Legacy notebook visualizations deprecated
- Create or modify table from file upload page supports JSON file uploads
-
- Databricks Terraform provider updated to version 1.14.0
- Databricks Runtime 7.3 LTS ML support ends
- Distributed training with TorchDistributor
- Databricks Runtime 13.0 (Beta)
- Improved file editor
- Change the DBFS Root Storage Redundancy for existing Azure Databricks Workspaces
- .ipynb (Jupyter) notebook support in Repos (preview)
- Support for reload4j
- New IP addresses for East Asia region
- Execute SQL cells in the notebook in parallel
- Create job tasks using Python code stored in a Git repo
- Databricks Terraform provider updated to version 1.13.0
- Databricks Terraform provider updated to version 1.12.0
- SQL admin console and workspace admin console combined
- Model Serving is GA
- Automatic feature lookup is GA
- For new workspaces that use UDRs, Azure storage IP needs an additional UDR
- Azure Databricks DBFS root storage uses Azure Data Lake Storage Gen2 for new workspaces
- New Catalog Explorer availability
- View frequent queries and users of a table using the Insights tab
- Exact match search is available in global search
- View lineage information for your Azure Databricks jobs
- Databricks Runtime 12.2 LTS and Databricks Runtime 12.2 LTS ML are GA
- Workspace files are now in Public Preview
-
- Support for ND A100 v4 series instances (Public Preview)
- Ray on Azure Databricks (Public Preview)
- Notebook cell output results limit increased
- Azure Databricks Jobs now supports running continuous jobs
- Trigger your Azure Databricks job when new files arrive
- Databricks Terraform provider updated to version 1.10.0
- Legacy global init scripts and cluster-named init scripts disabled
- Improvements to the MLflow experiment UI
- Databricks Runtime 12.2 (Beta)
- Use user-assigned managed identities to access storage in Unity Catalog (Public Preview)
- Databricks extension for Visual Studio Code (Public Preview)
-
- Enable private connectivity with Azure Private Link (GA)
- Account SCIM is now GA
- Easier creation and editing of Azure Databricks jobs in the UI
- Improvements to the Azure Databricks Jobs UI when viewing job runs
- New account console home screen provides better account management experience
- Databricks Terraform provider updated to version 1.9.1
- Account users can update email preferences in the account console
- Region support consolidated onto one page
- Databricks Runtime 12.1 and Databricks Runtime 12.1 ML are GA
- Cluster policies now support limiting the max number of clusters per user
- Databricks Terraform provider updated to version 1.9.0
-
- Databricks Terraform provider updated to version 1.7.0
- Databricks Runtime 12.0 and 12.0 ML are GA
- Jobs are now available in global search (Public Preview)
- Billable usage graphs can now aggregate by individual tags
- Use SQL to specify schema- and catalog-level storage locations for Unity Catalog managed tables
- Capturing lineage data with Unity Catalog is now generally available
- Databricks ODBC driver 2.6.29
- Databricks JDBC driver 2.6.32
-
- Enhanced notifications for your Azure Databricks jobs (Public Preview)
- Databricks Runtime 12.0 (Beta)
- Work with large repositories with Sparse Checkout
- Databricks Terraform provider updated to version 1.6.5
- Databricks Terraform provider updated to versions 1.6.3 and 1.6.4
- Specify a cloud storage location for Unity Catalog managed tables at the catalog and schema levels
- Access recent objects from the search field in the top bar of your workspace
- Create or modify table from file upload page now supports multiple files
- Create or modify table from file upload page now supports overwrite
- Search for jobs by name with the Jobs API 2.1
- Databricks Terraform provider updated to version 1.6.2
- Search for tables in Unity Catalog is GA
-
- GA: Repos support for non-notebook files
- Customer-managed keys for managed disk encryption
- Deploy models for streaming inference with Delta Live Tables notebooks
- Connect to Fivetran from the add data UI
- Databricks SQL Driver for Node.js is Generally Available
- Databricks Feature Store support for Azure Cosmos DB (Public Preview)
- Enforce user isolation cluster types on a workspace
- Databricks Runtime 11.3 LTS and 11.3 LTS ML are GA
- Format Python code in notebooks (Public Preview)
- Service principals provide API-only access to the Azure Databricks account
- Personal Compute cluster policy is available by default to all users
- Add data UI provides a central UI for loading data to Azure Databricks
- Create or modify table from file upload page unifies experience for small file upload to Delta Lake
-
- New reference solution for natural language processing
- More regions for Unity Catalog
- Top navigation bar in the UI
- Audit logs now include events for web terminal
- Audit logs now include events for managing credentials for Git repos
- Select cluster policies directly in the Delta Live Tables UI
- New data trasformation card on workspace landing pages
- Orchestrate Databricks SQL tasks in your Azure Databricks workflows (Public Preview)
- Delta cache is now disk cache
- Capture and view lineage data with Unity Catalog (Public Preview)
- Search for tables using Catalog Explorer (Public Preview)
- View and organize assets in the workspace browser across personas
- Databricks Runtime 11.2 and 11.2 ML are GA
-
- Account users can access the account console
- Databricks ODBC driver 2.6.26
- Databricks JDBC driver 2.6.29
- Databricks Feature Store client now available on PyPI
- Unity Catalog is GA
- Delta Sharing is GA
- Databricks Runtime 11.2 (Beta)
- Reduced message volume in the Delta Live Tables UI for continuous pipelines
- Easier cluster configuration for your Delta Live Tables pipelines
- Orchestrate dbt tasks in your Databricks workflows (Public Preview)
- Identity federation is GA
- Additional data type support for Databricks Feature Store automatic feature lookup
- Bring your own key: Encrypt Git credentials
- Cluster UI preview and access mode replaces security mode
- Unity Catalog limitations (Public Preview)
-
- Databricks Runtime 11.1 and 11.1 ML are GA
- Photon is GA
- Notification upon notebook completion
- Increased limit for the number of jobs in your Azure Databricks workspaces
- Verbose audit logs now record when Databricks SQL queries are run
- Databricks JDBC driver 2.6.27
- Databricks ODBC driver 2.6.25
- Databricks Runtime 11.1 (Beta)
- Improved notebook visualizations
-
ALTER TABLEpermission changes for Unity Catalog- Databricks Runtime 6.4 Extended Support reaches end of support
- Databricks Runtime 10.2 series support ends
- Databricks ODBC driver 2.6.24
- Databricks Terraform provider is now GA
- Databricks Runtime 11.0 and 11.0 ML are GA; 11.0 Photon is Public Preview
- Change to Repos default working directory in Databricks Runtime 11.0
- Databricks Runtime 10.1 series support ends
- Delta Live Tables now supports SCD type 2
- Create Delta Live Tables pipelines directly in the Azure Databricks UI
- Select the Delta Live Tables channel when you create or edit a pipeline
- Communicate between tasks in your Azure Databricks jobs with task values
- Enable account switching in the Databricks UI
-
- Copy and paste notebook cells between tabs and windows
- Additional data type support for Databricks Feature Store automatic feature lookup
- Databricks Runtime 11.0 (Beta)
- Improved workspace search (Public Preview)
- Explore SQL cell results in Python notebooks natively using Python
- Databricks Repos: Support for more files in a repo
- Databricks Repos: Fix to issue with MLflow experiment data loss
- Upgrade wizard makes it easier to copy databases and multiple tables to Unity Catalog (Public Preview)
- Power BI Desktop system-wide HTTP proxy support
- Audit logs can now record when a notebook command is run
- Databricks Runtime 10.5 and 10.5 ML are GA; 10.5 Photon is Public Preview
- Databricks JDBC driver 2.6.25
- See the user a pipeline runs as in the Delta Live Tables UI
-
- New Azure region: Sweden Central
- Use tags to better manage your Azure Databricks jobs
- Databricks Runtime 10.0 series support ends
- Get a visual overview of your job runs with the new jobs matrix view
- Save time and resources when your Azure Databricks job runs are unsuccessful
- View the run history for job tasks
- Assign a new cluster in the jobs UI when the Single User access no longer exists
- Databricks Runtime 10.5 (Beta)
- The Delta Live Tables UI is enhanced to disable unauthorized actions
- Databricks AutoML is generally available
- Use datasets from Unity Catalog with AutoML
- Delta Live Tables is GA on AWS and Azure, and in Public Preview on GCP
- Delta Live Tables SQL interface: non-breaking change to table names
-
- Better performance and cost for your Delta Live Tables pipelines with Databricks Enhanced Autoscaling
- Files in Repos enabled by default in new workspaces
- Databricks Feature Store is generally available
- Share an experiment from the experiment page
- RStudio Workbench bug fix
- New workspace language options
- Databricks Runtime 10.4 LTS and 10.4 LTS ML are GA; 10.4 Photon is Public Preview
- Unity Catalog is available in Public Preview
- Delta Sharing is available in Public Preview
- Rotate the customer-managed key for managed services
- Enhanced access control for Delta Live Tables pipelines
- Test Delta Live Tables preview functionality with the new
channelsetting (Public Preview) - Improved error handling for Delta Live Tables Python functions (Public Preview)
- Improvements to Databricks Repos
- Diagnostic logging for cluster policy changes
- Databricks Runtime 10.4 (Beta)
-
- Easier scheduling for your Delta Live Tables pipelines (Public Preview)
- Easily browse the history of your Delta Live Tables pipeline updates (Public Preview)
- Ensure job idempotency for the Jobs API Run now request
- Jobs service stability and scalability improvements
- Compare MLflow runs from different experiments
- Improvements to MLflow compare runs display
- Improved visibility into job run owners in the clusters UI
- Drop dataset columns in AutoML
- Experiments page is GA
- Support for temporary tables in the Delta Live Tables Python interface
- User interface improvements for Delta Live Tables (Public Preview)
- Databricks Runtime 9.0 series support ends
- Data Science & Engineering landing page updates
- Databricks Repos now supports AWS CodeCommit for Git integration
- Improved visualization for your Delta Live Tables pipelines (Public Preview)
- Updated Markdown parser
- Delta Live Tables now supports change data capture processing (Public Preview)
- Additional metastores now available in multiple regions
- Select algorithm frameworks to use with AutoML
- Databricks hosted MLflow models can now look up features from online stores
- Databricks Runtime 10.3 and 10.3 ML are GA; 10.3 Photon is Public Preview
-
- MLflow Model Registry Webhooks on Azure Databricks (Public Preview)
- Breaking change: cluster idempotency token cleared on cluster termination
- Databricks Runtime 10.3 (Beta)
- View information on recent job runs
- Use Markdown in Databricks Repos file editor
- Improved cluster management for jobs that orchestrate multiple tasks
- Databricks Runtime 8.3 and Databricks Runtime 8.4 series support ends
- Databricks JDBC driver 2.6.22
- Additional metastores now available in multiple regions
- New Share button replaces Permissions icon in notebooks
- New workspace language options
- Additional metastores are now available in multiple regions
-
- Databricks Runtime 6.4 Extended Support series end-of-support date extended
- Databricks Runtime 5.5 Extended Support series reaches end of support
- Databricks JDBC driver 2.6.21
- Databricks Connector for Tableau 2021.4
- Databricks Runtime 10.2 and 10.2 ML are GA; 10.2 Photon is Public Preview
- Switzerland West region now available
- Updated Markdown parser
- User interface improvements for Delta Live Tables
- Additional metastores coming soon to multiple regions
- Update SCC relay for Australia Southeast region
- Databricks Runtime 8.3 series support extended
- Databricks Runtime 10.2 (Beta)
- Revert of recent breaking change that removed escaping and quotes from $ in environment variable values for cluster creation
-
- Create tags for feature tables (Public Preview)
- Syntax highlighting and autocomplete for SQL commands in Python cells
- Rename, delete, and change permissions for MLflow experiments from experiment page (Public Preview)
- New data profiles in notebooks: tabular and graphic summaries of your data (Public Preview)
- Improved logging when schemas evolve while running a Delta Live Tables pipeline
- Breaking change: remove escaping and quotes from $ in environment variable values for cluster creation
- Ease of use improvements for Files in Repos
- Support for legacy SQL widgets ends on January 15, 2022
- User interface improvements for Azure Databricks jobs
- Delta Sharing Connector for Power BI
- Databricks ODBC driver 2.6.19
- Databricks Runtime 10.1 and 10.1 ML are GA; 10.1 Photon is Public Preview
- Databricks Runtime 10.1 (Beta)
- Rename and delete MLflow experiments (Public Preview)
- You can now create a cluster policy by cloning an existing policy
- Change the default language of notebooks and notebook cells more easily
- Use Files in Repos from the web terminal
-
- Databricks Runtime 8.2 series support ends
- Databricks Runtime 10.0 and 10.0 ML are GA; 10.0 Photon is Public Preview
- User interface improvements for Delta Live Tables (Public Preview)
- Specify a fixed-size cluster when you create a new pipeline in Delta Live Tables (Public Preview)
- View data quality metrics for tables in Delta Live Tables triggered pipelines (Public Preview)
- Jobs orchestration is now GA
- Databricks Connector for Power BI
- Repos now supports arbitrary file types
- More detailed job run output with the Jobs API
- Improved readability of notebook paths in the Jobs UI
- Open your Delta Live Tables pipeline in a new tab or window
- New escape sequence for
$in legacy input widgets in SQL - Faster model deployment with automatically generated batch inference notebook
-
- Databricks Runtime 10.0 (Beta)
- Databricks ODBC driver 2.6.18
- Use customer managed keys for encryption with existing workspaces
- Databricks Runtime 9.1 LTS and 9.1 LTS ML are GA; 9.1 LTS Photon is Public Preview
- Databricks Runtime 8.1 series support ends
- Databricks JDBC driver 2.6.19
- Share feature tables across workspaces
- General Availability of several Azure GPU instance types
- Repos now supports
.gitignore - Enhanced jobs UI is now standard for all workspaces
- Databricks Runtime 9.1 (Beta)
- Streamlined management of settings for Azure Databricks jobs
- Databricks SQL Public Preview available in all workspaces
- Delete feature tables from Feature Store
- Grant view pipeline permissions in the Delta Live Tables UI (Public Preview)
- Reduce cluster resource usage with Delta Live Tables (Public Preview)
- Use MLflow models in your Delta Live Tables pipelines (Public Preview)
- Find Delta Live Tables pipelines by name (Public Preview)
- PyTorch TorchScript and other third-party libraries are now supported in Azure Databricks jobs
- Databricks Runtime 8.0 series support ends
-
- Databricks Repos GA
- Increased timeout for long-running notebook jobs
- Jobs service stability and scalability improvements
- User entitlements granted by group membership are displayed in the admin console
- Manage MLflow experiment permissions (Public Preview)
- Improved job creation from notebooks
- Ddsv4-series VMs (Public Preview)
- Improved support for collapsing notebook headings
- Databricks Runtime 9.0 and 9.0 ML are GA; 9.0 Photon is Public Preview
- Databricks Runtime 9.0 (Beta)
- Manage repos programmatically with the Databricks CLI (Public Preview)
- Manage repos programmatically with the Databricks REST API (Public Preview)
- Databricks Runtime 7.6 series support ends
- Audit events are logged when you interact with Databricks Repos
- Improved job creation and management workflow
- Photon runtimes now available (Public Preview)
- Simplified instructions for setting Git credentials (Public Preview)
- Import multiple notebooks in
.htmlformat - Usability improvements for Delta Live Tables
-
- Manage MLflow experiment permissions with the Databricks REST API
- Azure Databricks web interface is localized in Portuguese and French (Public Preview)
- Databricks Runtime 5.5 LTS for Machine Learning support ends, replaced by Extended Support version
- Databricks Light 2.4 support ends September 5, replaced by Extended Support version
- New Azure region: Germany West Central
- Feature freshness information available in Databricks Feature Store UI (Public Preview)
- Display up to 10,000 result rows
- Bulk import and export notebooks in a folder as source files
- Autocomplete in SQL notebooks now uses all-caps for SQL keywords
- Reorderable and resizable widgets in notebooks
- Azure Databricks UI usability fixes
- Quickly define pipeline settings when you create a new Delta Live Tables pipeline
- Databricks Runtime 8.4 and 8.4 ML are GA; 8.4 Photon is Public Preview
- Use Spark SQL with the Delta Live Tables Python API
- Enhanced data processing and analysis with Azure Databricks jobs (Public Preview)
- Reduced cost for Delta Live Tables default clusters (Public Preview)
- Sort pipelines by name in the Delta Live Tables UI (Public Preview)
- Changes to Compute page
- Databricks Runtime 5.5 LTS support ends, replaced by Databricks Runtime 5.5 Extended Support through the end of 2021
- Repos API (Public Preview)
- Databricks Runtime 8.4 (Beta)
-
- Correction: Repos for Git is enabled by default in new and existing workspaces in some regions
- Change to Feature Store permissions
- Improved access to results in the MLflow runs table
- Better cost visibility for Delta Live Tables
- Enhanced data quality constraints for Delta Live Tables
- API changes for updating and replacing IP address lists
- Databricks ODBC driver 2.6.17
- Databricks Runtime 7.5 series support ends
- Optimize performance and control costs by using different pools for the driver node and worker nodes
- Registry-wide permissions for Model Registry
- A user's home directory is no longer protected when you delete a user using the SCIM API
- Databricks Runtime 8.3 and 8.3 ML are GA; 8.3 Photon is Public Preview
- Python and SQL table access control (GA)
- Jobs UI and API now show the owner of a job run
- Protect sensitive Spark configuration properties and environment variables using secrets (Public Preview)
- Repos for Git is enabled by default in new and existing workspaces in some regions
- Redesigned Workspace Settings UI
- Updates to
ListTokensandListAllTokensdatabase queries expired tokens - Confirmation now required when granting or revoking Admin permissions
- Changes to keyboard shortcuts in the web UI
-
- Databricks Machine Learning: a data-native and collaborative solution for the full ML lifecycle
- SQL Analytics is renamed to Databricks SQL
- Create and manage ETL pipelines using Delta Live Tables (Public Preview)
- Azure Spot VMs are GA
- Encrypt Databricks SQL queries and query history using your own key (Public Preview)
- Increased limit for the number of terminated all-purpose clusters
- Increased limit for the number of pinned clusters
- Manage where notebook results are stored (Public Preview)
- Encrypt notebook and secret data in the control plane with your own key (Public Preview)
- Databricks Runtime 7.4 series support ends
- Repos users can now integrate with Azure DevOps using personal access tokens
-
- Databricks Runtime 8.2 (GA)
- Easier job management with the enhanced jobs user interface
- Cluster policy changes are applied automatically to existing clusters at restart and edit
- Track retries in your job tasks when task attempts fail
- Quickly view cluster details when you create a new cluster
- MLflow sidebar reflects the most recent experiment
- Change to default channel for
conda.yamlfiles in MLflow - Streamlined authentication from Tableau Online to Azure Databricks using Microsoft Entra ID
- Databricks Runtime 8.2 (Beta)
- New extended infrastructure requires update to custom routes and egress firewalls
- User and group limits
- Easier monitoring of job run status
- Global init scripts no longer run on model serving clusters
- Receive email notification about activity in Model Registry
- Databricks Runtime 6.4 series support ends
-
- Azure Databricks now supports dark mode for viewing notebooks
- Save your Azure Databricks TCO with Azure Spot VMs (Public Preview)
- Databricks Runtime 8.1 (GA)
- Easier job creation and management with the enhanced jobs user interface (Public Preview)
- Track job retry attempts with a new sequential value returned for each job run attempt
- Increased limit for the number of saved jobs in Premium workspaces
- Easier way to connect to Azure Databricks from your favorite BI tools and SQL clients
- Azure Databricks Repos let you use Git repositories to integrate Databricks with CI/CD systems
- Automatic retries for failed job clusters reverted
- Streamlined authentication from Tableau Desktop to Azure Databricks using Microsoft Entra ID
- Databricks Runtime 8.1 (Beta)
- Model Serving now supports additional model types
- New options for searching Model Registry
- Increased limit for the number of terminated all-purpose clusters
- Increased limit for the number of pinned clusters in a workspace
- Databricks Runtime 8.0 (GA)
-
- New Azure Databricks Power BI connector (GA)
- Diagnostic logging of user activities in Azure Databricks is GA
- Added modification_time to the DBFS REST API get-status and list responses
- Easily copy long experiment names in MLflow
- Adjust memory size and number of cores for serving clusters
- Web terminal is now GA
- Apache Spark connector for Azure SQL Database and SQL Server supports Databricks Runtime 7.x and above
- Databricks Runtime 7.2 series support ends
- Databricks Runtime 7.6 GA
- Databricks Runtime 8.0 (Beta)
- Databricks Runtime for Genomics now deprecated
- View more readable JSON in the MLflow run artifact display
- Provide comments in the Model Registry using REST API
- Easily specify default cluster values in API calls
- Tune cluster worker configuration according to current worker allocation
- Pass context specific information to a job's task with task parameter variables
- Error messages from job failures no longer contain possibly sensitive information
-
- Secure cluster connectivity (no public IPs) is GA
- Databricks Runtime 7.1 series support ends
- Start clusters faster with Docker images preloaded into instance pools
- Notebook find and replace now supports changing all occurrences of a match
- Single Node clusters (GA)
- Free form cluster policy type renamed to Unrestricted
- Cluster policy field not shown if a user only has access to one policy
- Databricks Runtime 7.0 series support ends
-
- Databricks Runtime 7.5 GA
- Jobs API now supports updating existing jobs
- New global init script framework is GA
- Databricks Runtime 7.5 (Beta)
- Jobs API end_time field now uses epoch time
- Find DBFS files using new visual browser
- Visibility controls for jobs, clusters, notebooks, and other workspace objects are now enabled by default on new workspaces
- Improved display of nested runs in MLflow
- Admins can now lock user accounts (Public Preview)
- Updated NVIDIA driver
-
- Databricks Runtime 6.6 series support ends
- MLflow Model Registry GA
- Filter experiment runs based on whether a registered model is associated
- Partner integrations gallery now available through the Data tab
- Cluster policies now use allowlist and blocklist as policy type names
- Automatic retries when the creation of a job cluster fails
- Navigate notebooks using the table of contents
- Databricks SQL (Public Preview)
- Single Node clusters now support Databricks Container Services
- Databricks Runtime 7.4 GA
- Databricks JDBC driver update
- Databricks Connect 7.3 (Beta)
-
- New Azure Databricks Power BI connector available in the online Power BI service (Public Preview)
- Databricks Runtime 7.4 (Beta)
- Use customer-managed keys for DBFS root (GA)
- Expanded experiment access control (ACLs)
- High fidelity import and export of Jupyter notebook (ipynb) files
- SCIM API improvement: both indirect and direct groups returned in user record response
- Azure request ID is now included in exceptions
- Databricks Runtime 6.5 series support ends
- Secure cluster connectivity (no public IPs) is Public Preview
- SCIM API improvement:
$reffield response - Diagnostic (audit) logs are now delivered with low latency
- Databricks Runtime 7.3, 7.3 ML, and 7.3 Genomics declared Long Term Support (LTS)
- Render images at higher resolution using matplotlib
- Use the Databricks CLI or the Databricks API to create Azure Key Vault backed secret scopes
- Use Microsoft Entra ID tokens to authenticate to the Databricks CLI
- New Azure regions (Public Preview)
-
- Databricks Runtime 7.3, 7.3 ML, and 7.3 Genomics are now GA
- Single Node clusters (Public Preview)
- DBFS REST API rate limiting
- New sidebar icons
- Running jobs limit increase
- Artifact access control lists (ACLs) in MLflow
- MLflow usability improvements
- New Azure Databricks Power BI connector (Public Preview)
- Use customer-managed keys for DBFS root (Public Preview)
- New JDBC and ODBC drivers bring faster and lower latency BI
- MLflow Model Serving (Public Preview)
- Clusters UI improvements
- Visibility controls for jobs, clusters, notebooks, and other workspace objects
- Ability to create tokens no longer permitted by default
- MLflow Model Registry supports sharing of models across workspaces
- Databricks Runtime 7.3 (Beta)
- Azure Databricks workload type name change
-
- Token Management API is GA and admins can use the Admin Console to grant and revoke user access to tokens
- Message size limits for Shiny apps increased
- Improved instructions for setting up a cluster in local mode
- View version of notebook associated with a run
- Databricks Runtime 7.2 GA
- Databricks Runtime 7.2 ML GA
- Databricks Runtime 7.2 Genomics GA
- Permissions API (Public Preview)
- Databricks Connect 7.1 (GA)
- Repeatable installation order for cluster libraries
- Create model from MLflow registered models page (Public Preview)
- Databricks Container Services supports GPU images
-
- Web terminal (Public Preview)
- New, more secure global init script framework (Public Preview)
- IP access lists now GA
- New file upload dialog
- SCIM API filter and sort improvements
- Databricks Runtime 7.1 GA
- Databricks Runtime 7.1 ML GA
- Databricks Runtime 7.1 Genomics GA
- Databricks Connect 7.1 (Public Preview)
- IP Access List API updates
- Python notebooks now support multiple outputs per cell
- View notebook code and results cells side by side
- Pause job schedules
- Jobs API endpoints validate run ID
- Microsoft Entra ID tokens to authorize to the Databricks REST API GA
- Format SQL in notebooks automatically
- Reproducible order of installation for Maven and CRAN libraries
- Take control of your users' personal access tokens with the Token Management API (Public Preview)
- Restore cut notebook cells
- Assign jobs CAN MANAGE permission to non-admin users
- Non-admin Azure Databricks users can view and filter by username using the SCIM API
- Link to view cluster specification when you view job run details
-
- Databricks Connect now supports Databricks Runtime 6.6
- Databricks Runtime 7.0 ML GA
- Databricks Runtime 7.0 GA, powered by Apache Spark 3.0
- Databricks Runtime 7.0 for Genomics GA
- Stage-dependent access controls for MLflow models
- Notebooks now support disabling auto-scroll
- Metastore IP addresses to change on June 30, 2020
- Internet Explorer 11 support ends on August 15
- Databricks Runtime 6.2 series support ends
- Simplify and control cluster creation using cluster policies (Public Preview)
- SCIM Me endpoint now returns SCIM compliant response
- Restrict access to Azure Databricks using IP access lists (Public Preview)
-
- Easv4-series VMs (Beta)
- Databricks Runtime 6.6 for Genomics GA
- Databricks Runtime 6.6 ML GA
- Databricks Runtime 6.6 GA
- DBFS REST API delete endpoint size limit
- Easily view large numbers of MLflow registered models
- Libraries configured to be installed on all clusters are not installed on clusters running Databricks Runtime 7.0 and above
- Databricks Runtime 7.0 for Genomics (Beta)
- Databricks Runtime 7.0 ML (Beta)
- Databricks Runtime 6.6 for Genomics (Beta)
- Databricks Runtime 6.6 ML (Beta)
- Databricks Runtime 6.6 (Beta)
- Job clusters now tagged with job name and ID
- Restore deleted notebook cells
- Jobs pending queue limit
-
- Unique URLs for each Azure Databricks workspace
- MLflow tracking UI enhancement
- Notebook usability improvements
- Use tokens to authorize to the Databricks REST API (Public Preview)
- Single user can view the Apache Spark logs on a passthrough cluster
- Databricks Connect now supports Databricks Runtime 6.5
- Databricks Runtime 6.1 and 6.1 ML support ends
- Databricks Runtime 6.5 GA
- Databricks Runtime 6.5 for Machine Learning GA
- Databricks Runtime 6.5 for Genomics GA
- Databricks Runtime 6.0 and 6.0 ML support ends
-
- Managed MLflow Model Registry collaborative hub available (Public Preview)
- Workspace, pool, and cluster tags propagate to DBU usage details and Azure VMs for better cost management reporting
- Databricks Runtime 7.0 (Beta) previews Apache Spark 3.0
- Databricks Runtime 6.5 ML (Beta)
- Databricks Runtime 6.5 (Beta)
- Azure Databricks feedback now goes directly to Azure Databricks feedback portal
- Develop and test Shiny applications inside RStudio Server
- Change the default language of a notebook
- Databricks Connect now supports Databricks Runtime 6.4
- Databricks Connect now supports Databricks Runtime 6.3
-
- Databricks Runtime 6.4 for Genomics GA
- Databricks Runtime 6.4 ML GA
- Databricks Runtime 6.4 GA
- New interactive charts offer rich client-side interactions
- New data ingestion network adds partner integrations with Delta Lake (Public Preview)
- Workspace creator automatically added as an Azure Databricks admin
- Flags to manage workspace security and notebook features now available
-
- Coming soon: workspace, pool, and cluster tags propagate to DBU usage details and Azure VMs for better cost management reporting
- Azure Databricks and Azure Lighthouse can now live in the same subscription
- Databricks Runtime 6.3 for Genomics GA
- Databricks Runtime 6.3 ML GA
- Databricks Runtime 6.3 GA
- Disk caching enabled by default
- Cluster standard autoscaling step is now configurable
- SCIM API supports pagination for Get Users and Get Groups (Public Preview)
- File browser swimlane widths increased to 240px
- Databricks Runtime 3.5 LTS support ends
-
- Databricks Connect now supports Databricks Runtime 6.2
- Configure clusters with your own container image using Databricks Container Services (GA)
- Databricks Runtime 6.2 for Genomics GA
- Azure Databricks SCIM provisioning connector available in the app gallery
- Databricks Runtime 5.3 and 5.4 support ends
- Databricks Runtime 6.2 ML GA
- Databricks Runtime 6.2 GA
- Databricks Connect now supports Databricks Runtime 6.1
-
- Supportability metrics moved to Azure Event Hubs
- Azure Data Lake Storage credential passthrough on standard clusters and Scala is GA
- Databricks Runtime 6.1 for Genomics GA
- Databricks Runtime 6.1 for Machine Learning GA
- MLflow API calls are now rate limited
- Pools of instances for quick cluster launch generally available
- Databricks Runtime 6.1 GA
- Databricks Runtime 6.0 for Genomics GA
- Ability to deploy an Azure Databricks workspace to your own virtual network, also known as VNet injection, is GA
- Non-admin Azure Databricks users can read user and group names and IDs using SCIM API
- Workspace API returns notebook and folder object IDs
- Databricks Runtime 6.0 ML GA
-
- Databricks Runtime 5.2 support ends
- Launch pool-backed automated clusters that use Databricks Light (Public Preview)
- Azure SQL Database gateway IP addresses will change on October 14, 2019
- Azure Data Lake Storage credential passthrough now supported on standard clusters and Scala (Public Preview)
- pandas DataFrames now render in notebooks without scaling
- Python version selector display now dynamic
- Databricks Runtime 6.0 Beta
-
- Coming soon: Databricks 6.0 will not support Python 2
- Preload the Databricks Runtime version on pool idle instances
- Custom cluster tags and pool tags play better together
- MLflow 1.1 brings several UI and API improvements
- pandas DataFrame display renders like it does in Jupyter
- New regions
- Databricks Runtime 5.5 with Conda (Beta)
- Updated metastore connection limit
- Set permissions on pools (Public Preview)
- Databricks Runtime 5.5 for Machine Learning
- Databricks Runtime 5.5
- Keep a pool of instances on standby for quick cluster launch (Public Preview)
- Ganglia metrics
- Global series color
-
- Upcoming change: Python 3 to become the default when you create clusters
- Databricks Runtime 5.2 for Machine Learning (Beta) release
- Cluster configuration JSON view
- Cluster UI
- Deploy Azure Databricks in your own Azure virtual network (VNet injection)
- Library UI
- Cluster Events
- Notebook Version Control using Azure DevOps Services
-
- Libraries API supports Python wheels
- IPython notebook export
- Azure Key Vault-backed secret scopes
- Trial Premium workspaces
- Cluster mode and High Concurrency clusters
- Table access control
- Unavailable cluster node types grayed out
- R Markdown support
- Home page redesign, with ability to drop files to import data
- Widget default behavior
- Table creation UI
- Multi-line JSON data import
-
- General Data Protection Regulation (GDPR)
- Azure Databricks users must belong to Microsoft Entra ID tenant
- HorovodEstimator
- MLeap ML Model Export
- Even more GPU cluster types
- Notebook cells: hide and show
- Doc site search
- Databricks Runtime 4.1 ML for Machine Learning (Beta)
- Databricks Delta
- Display() support for image data types
- GPU cluster types
- Secret management GA
- Secrets API endpoint and CLI command changes
- Cluster pinning
- Cluster autostart
- Workspace purging
- Databricks CLI 0.7.1
-
- Command execution details
- Databricks CLI supports
--profile - ACLs enabled by default for new Premium SKU customers
- Azure Databricks is now generally available
- New doc site theme
- Autoscaling local storage
- Virtual network (VNet) peering
- Cluster event log
- Databricks CLI: 0.6.0 release
- Job run management
- Edit cluster permissions now requires edit mode
- Databricks ML Model Export
-
- New line chart supports time-series data
- More visualization improvements
- Delete job runs using Job API
- KaTeX math rendering library updated
- Databricks CLI: 0.5.0 release
- DBUtils API library
- Filter for your jobs only
- Spark-submit from the Create Job page
- Select Python 3 from the Create Cluster page
- Workspace UI improvements
- Autocomplete for SQL commands and database names