Delta Live Tables release 2022.40

  • Article

September 28 - October 5, 2022

These features and improvements were released with the 2022.40 release of Delta Live Tables.

Databricks Runtime versions used by this release

Channel:

  • CURRENT (default): Databricks Runtime 11.0.7
  • PREVIEW: Databricks Runtime 11.0.7

New features and improvements in this release

  • Using a cluster label other than default or maintenance in your pipeline settings causes failures during pipeline creation or edit. For more information about cluster configuration for Delta Live Tables pipelines, see Configure compute for a Delta Live Tables pipeline.

  • You can now open or run pipelines directly from your Delta Live Tables notebooks. See Develop and debug Delta Live Tables pipelines in notebooks.

  • This release includes a breaking change in the pipeline settings data structure returned by Delta Live Tables API requests. This change returns the edition field as an upper case value. Previously, this value was returned as mixed case. You must update your clients that read the edition value in API responses to ensure the correct processing of the return value. However, the edition value in API requests is case insensitive; your clients that submit requests including this field do not require updating.

Bug Fixes in this release

  • This release fixes a bug that causes the Delta Live Tables UI to show a pipeline in a RUNNING state even after update failure.

  • This update fixes a bug in the APPLY_CHANGES interface when re-processing input data where the data contains duplicate DELETE operations with the same value in the SEQUENCE column. This scenario results in NULL values in the target table and removal of the DeleteVersion value. This is equivalent to marking the record as NOT-DELETED. With this change, the duplicate DELETE operations are handled idempotently and always mark the record as DELETED.

  • This release fixes a bug causing rows to be hidden during SCD type 2 processing when a previous historical record is deleted containing the same columns as the next historical record.

  • This update fixes a bug causing a view with an expect_or_fail expectation to remain in the RUNNING state even if the expectation fails. With this fix, the pipeline fails and transitions to the FAILED state.