Lakeflow Declarative Pipelines release 2023.16

April 24 - 28, 2023

These features and improvements were released with the 2023.16 release of Lakeflow Declarative Pipelines.

Databricks Runtime versions used by this release

Channel:

Lakeflow Declarative Pipelines support for Unity Catalog is in Public Preview. You can use Unity Catalog with your Lakeflow Declarative Pipelines to:
- Define a catalog in Unity Catalog where your pipeline will persist tables.
- Read data from Unity Catalog tables.
See Use Unity Catalog with your Lakeflow Declarative Pipelines.
By default, streaming tables require append-only sources. This release introduces the skipChangeCommits flag that can be set on a target streaming table to ignore changes in a source streaming table when the source table requires updates or deletes. See Configure a streaming table to ignore changes in a source streaming table
With this release, the limit for concurrent updates in a workspace is increased to 100. The previous concurrent updates limit was 20.
Lakeflow Declarative Pipelines is now supported in Azure operated by 21Vianet regions.
When you run a Lakeflow declarative pipeline from a pipeline notebook, you can now view the event log in the notebook sidebar. See Develop and debug ETL pipelines with a notebook in Lakeflow Declarative Pipelines.

This release fixes a bug causing the pipelines list page to crash when the "%" character was entered in the search box.