Lakeflow Declarative Pipelines release 2023.50

December 18 - 22, 2023

These features and improvements were released with the 2023.50 release of Lakeflow Declarative Pipelines.

Note

Releases are staged. Your Lakeflow Declarative Pipelines release, including Databricks Runtime versions, might not be updated until a week or more after the initial release date.

Databricks Runtime versions used by this release

Channel:

  • CURRENT (default): Databricks Runtime 12.2.18
  • PREVIEW: Databricks Runtime 14.1.3

Bug fixes in this release

  • This release fixes a bug causing incorrect recording of lineage for streaming tables.

  • With this release, streaming flows are started after all dependencies have completed processing. Previously, a streaming flow might begin processing before its dependencies are complete, leading to data loss in some cases.

  • This release includes changes to fix a bug causing maintenance jobs to fail for some pipelines. In continuous pipelines, for an AUTO CDC ... INTO target without z-order columns specified, the OPTIMIZE operation is skipped when the maintenance tasks are performed, and auto-compact is enabled.