Delta Live Tables release 2022.40
September 28 - October 5, 2022
These features and improvements were released with the 2022.40 release of Delta Live Tables.
Databricks Runtime versions used by this release
Channel:
- CURRENT (default): Databricks Runtime 11.0.7
- PREVIEW: Databricks Runtime 11.0.7
New features and improvements in this release
Using a cluster label other than
default
ormaintenance
in your pipeline settings causes failures during pipeline creation or edit. For more information about cluster configuration for Delta Live Tables pipelines, see Configure your compute settings.You can now open or run pipelines directly from your Delta Live Tables notebooks. See Open or run a Delta Live Tables pipeline from a notebook.
This release includes a breaking change in the pipeline settings data structure returned by Delta Live Tables API requests. This change returns the
edition
field as an upper case value. Previously, this value was returned as mixed case. You must update your clients that read theedition
value in API responses to ensure the correct processing of the return value. However, theedition
value in API requests is case insensitive; your clients that submit requests including this field do not require updating.
Bug Fixes in this release
This release fixes a bug that causes the Delta Live Tables UI to show a pipeline in a
RUNNING
state even after update failure.This update fixes a bug in the APPLY_CHANGES interface when re-processing input data where the data contains duplicate
DELETE
operations with the same value in theSEQUENCE
column. This scenario results inNULL
values in the target table and removal of theDeleteVersion
value. This is equivalent to marking the record asNOT-DELETED
. With this change, the duplicateDELETE
operations are handled idempotently and always mark the record asDELETED
.This release fixes a bug causing rows to be hidden during SCD type 2 processing when a previous historical record is deleted containing the same columns as the next historical record.
This update fixes a bug causing a view with an
expect_or_fail
expectation to remain in theRUNNING
state even if the expectation fails. With this fix, the pipeline fails and transitions to theFAILED
state.