Delta Live Tables release notes and the release upgrade process

Article
12/20/2024

This article explains the Delta Live Tables release process, how the Delta Live Tables runtime is managed, and provides links to release notes for each Delta Live Tables release.

Delta Live Tables runtime channels

Note

To see the Databricks Runtime versions used with a Delta Live Tables release, see the release notes for that release.

Delta Live Tables clusters use runtimes based on Databricks Runtime release notes versions and compatibility. Databricks automatically upgrades the Delta Live Tables runtimes to support enhancements and upgrades to the platform. You can use the channel field in the Delta Live Tables pipeline settings to control the Delta Live Tables runtime version that runs your pipeline. The supported values are:

current to use the current runtime version.
preview to test your pipeline with upcoming changes to the runtime version.

By default, your pipelines run using the current runtime version. Databricks recommends using the current runtime for production workloads. To learn how to use the preview setting to test your pipelines with the next runtime version, see Automate testing of your pipelines with the next runtime version.

Important

Features marked as generally available or Public Preview are available in the current channel.

For more information about Delta Live Tables channels, see the channel field in the Delta Live Tables pipeline settings.

To understand how Delta Live Tables manages the upgrade process for each release, see How do Delta Live Tables upgrades work?.

How do I find the Databricks Runtime version for a pipeline update?

You can query the Delta Live Tables event log to find the Databricks Runtime version for a pipeline update. See Runtime information.

Delta Live Tables release notes

Delta Live Tables release notes are organized by year and week-of-year. Because Delta Live Tables is versionless, both workspace and runtime changes take place automatically. The following release notes provide an overview of changes and bug fixes in each release:

How do Delta Live Tables upgrades work?

Delta Live Tables is considered to be a versionless product, which means that Databricks automatically upgrades the Delta Live Tables runtime to support enhancements and upgrades to the platform. Databricks recommends limiting external dependencies for Delta Live Tables pipelines.

Databricks proactively works to prevent automatic upgrades from introducing errors or issues to production Delta Live Tables pipelines. See Delta Live Tables upgrade process.

Especially for users that deploy Delta Live Tables pipelines with external dependencies, Databricks recommends proactively testing pipelines with preview channels. See Automate testing of your pipelines with the next runtime version.

Delta Live Tables upgrade process

Databricks manages the Databricks Runtime used by Delta Live Tables compute resources. Delta Live Tables automatically upgrades the runtime in your Azure Databricks workspaces and monitors the health of your pipelines after the upgrade.

If Delta Live Tables detects that a pipeline cannot start because of an upgrade, the runtime version for the pipeline reverts to the previous version that is known to be stable, and the following steps are triggered automatically:

The pipeline's Delta Live Tables runtime is pinned to the previous known-good version.
Databricks support is notified of the issue.
- If the issue is related to a regression in the runtime, Databricks resolves the issue.
- If the issue is caused by a custom library or package used by the pipeline, Databricks contacts you to resolve the issue.
When the issue is resolved, Databricks initiates the upgrade again.

Important

Delta Live Tables only reverts pipelines running in production mode with the channel set to current.

Automate testing of your pipelines with the next runtime version

To ensure changes in the next Delta Live Tables runtime version do not impact your pipelines, use the Delta Live Tables channels feature:

Create a staging pipeline and set the channel to preview.
In the Delta Live Tables UI, create a schedule to run the pipeline weekly and enable alerts to receive an email notification for pipeline failures. Databricks recommends scheduling weekly test runs of pipelines, especially if you use custom pipeline dependencies.
If you receive a notification of a failure and are unable to resolve it, open a support ticket with Databricks.

Pipeline dependencies

Delta Live Tables supports external dependencies in your pipelines; for example, you can install any Python package using the %pip install command. Delta Live Tables also supports using global and cluster-scoped init scripts. However, these external dependencies, particularly init scripts, increase the risk of issues with runtime upgrades. To mitigate these risks, minimize using init scripts in your pipelines. If your processing requires init scripts, automate testing of your pipeline to detect problems early; see Automate testing of your pipelines with the next runtime version. If you use init scripts, Databricks recommends increasing your testing frequency.