January 2024

These features and Azure Databricks platform improvements were released in January 2024.

Native XML file format support (Public Preview)

January 31, 2024

Native XML file format support is now in Public Preview. XML file format support enables ingestion, querying, and parsing of XML data for batch processing or streaming. It can automatically infer and evolve schema and data types, supports SQL expressions like from_xml, and can generate XML documents. It doesn't require external jars and works seamlessly with Auto Loader, read_files, COPY INTO, and Delta Live Tables. For more information, see Read and write XML files.

Share AI models using Databricks Marketplace (Public Preview)

January 31, 2024

You can now use Marketplace to share models that are registered in Unity Catalog. Both the provider and consumer workspaces must be enabled for Unity Catalog to participate in model sharing.

See Add models to a share.

Updates to network security group rules

January 31, 2023

To improve platform scalability, the Azure Databricks control plane now requires new endpoints for the classic data plane to connect to. For workspaces deployed in your own Azure virtual network, the network ports 3306 and 8443-8451 are now added to the existing security rule to the AzureDatabricks destination, enforcing outbound traffic on port 443.

For more information, see Network security group rules.

Workspace path update

Historically, users were required to include the /Workspace path prefix for some Databricks APIs (%sh) but not for others (%run, REST API inputs). Now, users can provide workspace paths with the /Workspace prefix everywhere in Databricks. Old references to paths without the /Workspace prefix are redirected and will continue to work. Databricks recommends that all workspace paths use the /Workspace prefix to differentiate them from Volume and DBFS paths.

Streamlined creation of Azure Databricks jobs

January 29, 2024

An empty job is now created immediately after you click the button to create a job, allowing you to edit job-level settings such as tags, schedule, or notification settings before adding the first task. Previously, you had to create the first task before getting access to job-level settings. See Create and run Azure Databricks Jobs.

URI path-based access to Unity Catalog external volumes

January 25, 2024

You can now use cloud storage URIs for path-based access to data governed by Unity Catalog stored in external volumes. See What path is used for accessing files in a volume?.

Access controls lists can be enabled on upgraded workspaces

January 24, 2024

Access control settings were enabled on all Premium plan workspaces in November 2023.

For workspaces that upgrade from the Standard plan to the Premium plan, access control settings are disabled by default. Workspace admins can now enable the following access control settings on upgraded workspaces:

• Workspace access control: Enables users to configure permissions to workspace objects.
• Cluster, pool, jobs access control: Enables users to configure permissions to clusters, pools, and jobs.
• Cluster visibility control: Prevents clusters from being visible to users who have no permissions on them.
• Jobs visibility control: Prevents jobs from being visible to users who have no permissions on them.
• Workspace visibility control setting: Prevents notebooks and folders from being visible to users who have no permissions on them.

After an access control setting is enabled, it cannot be disabled. For more information, see Access control lists.

Marketplace listing events system table now available (Public Preview)

January 24, 2024>

The Marketplace system schema now includes a listing funnel events table (system.marketplace.listing_funnel_events) where you can monitor consumer actions on your listings.

Updated UI for notebook cells (Public Preview)

January 24, 2024

An updated notebook cell design is available. To enable the new cell design, open the editor settings page in the workspace. In the sidebar, click Developer. Under Experimental features, toggle New cell UI. For details about working with notebook cells see Databricks notebook interface and controls.

Quick Fix help with syntax errors in the notebook

January 24, 2024

You can get help from Databricks Assistant for syntax errors by using Quick Fix. For details, see Python error highlighting.

Databricks Runtime 14.3 LTS (Beta)

January 22, 2024

Databricks Runtime 14.3 LTS and Databricks Runtime 14.3 LTS ML are now available as Beta releases.

See Databricks Runtime 14.3 LTS and Databricks Runtime 14.3 LTS for Machine Learning.

Share AI models using Delta Sharing (Public Preview)

January 19, 2024

You can now use Delta Sharing to share models that are registered in Unity Catalog. Both the provider and recipient workspaces must be enabled for Unity Catalog to share models.

See Add models to a share.

Databricks Marketplace supports volume sharing

January 18, 2023

Databricks Marketplace now gives you access to volumes (non-tabular data) as long as you have a Unity Catalog-enabled Databricks workspace. Volumes are Unity Catalog objects that represent a logical volume of storage in a cloud object storage location, and they provide a way to manage governance and sharing of non-tabular data assets.

Create widgets from the Databricks UI

January 18, 2024

You can now create Databricks Widgets using the Databricks UI. For details, see Create widgets using the UI.

Warehouse events system table is now available (Public Preview)

January 18, 2024

The system.compute.warehouse_events table records when SQL warehouses start, stop, scale up and down. You can use the table to monitor and manage the SQL warehouses in your workspaces.

Workspace file size limit is now 500MB

January 15, 2023

Individual workspace files are now limited to 500 MB. Previously, the limit was 200 MB. For more details, read Limitations.

Feature removal notice for legacy Git integration in Databricks

January 12, 2024

After January 31st, 2024, Databricks will remove legacy notebook Git integrations. This feature has been in legacy status for more than two years, and a deprecation notice has been displayed in the product UI since November 2023.

For details on migrating to Databricks Git folders (Repos), read Switching to Databricks Repos from Legacy Git integration. If this removal impacts you and you need an extension, contact your Databricks Account Executive.

Databricks ODBC driver 2.7.7

January 9, 2024

We have released version 2.7.7 of the Databricks ODBC driver (download). This release adds the following new features and enhancements:

• Azure Managed Identity authentication.
• Support for asynchronous metadata operations. The server needs to support SPARK_CLI_SERVICE_PROTOCOL_V9 and above.
• Support for Parameterized Query in the Native Query Mode. The server needs to support SPARK_CLI_SERVICE_PROTOCOL_V8 and above.
• OAuth enhancement: The connector can now disable the token cache for the browser-based authentication flow.

This release also resolves the following issues in 2.7.7:

• The connector ignores private root CA when using the Windows Trust Store.
• When an operation fails, the connector does not close the heartbeat threads.
• The connector does not support the optional refresh token.
• When querying tables with foreign names, the connector fails to read them.
• When using OAuth authentication, if the string length of the HOST is less than 20, the connector terminates unexpectedly.

Databricks Runtime 13.2 series support ends

January 6, 2024

Support for Databricks Runtime 13.2 and Databricks Runtime 13.2 for Machine Learning ended on January 6. See Databricks support lifecycles.

AI assistive features are enabled by default

January 2-8, 2024

AI assistive features are now enabled by default for workspaces without the compliance security profile enabled. These include:

• AI-generated comments for Unity Catalog managed tables

• In-product help chatbot

• Databricks Assistant

Admins can opt out of AI assistive features for all workspaces in their account or a subset of workspaces.