Delta Live Tables release 2022.37
September 14 - 22, 2022
These features and improvements were released with the 2022.37 release of Delta Live Tables.
Databricks Runtime versions used by this release
Channel:
- CURRENT (default): Databricks Runtime 10.3.7
- PREVIEW: Databricks Runtime 11.0.5
New features and improvements in this release
- The Start a pipeline update API request now returns the
request_id
field in the response body. Therequest_id
is a stable identifier for the original request starting the update. If an update is retried or restarted, the new update inherits therequest_id
.
{
"update_id": "the ID of the update that was started",
"request_id": "The ID of the request that started this update"
}
The new requests
API request (GET /pipelines/{pipeline_id}/requests/{request_id}
) returns the status of the pipeline update associated with request_id
. The response includes information about the latest update.
{
"status": "ACTIVE",
"latest_update": {
}
}
Your Python code can now call
spark.sql
operations outside ofdlt.table()
ordlt.view()
functions, as long as the operation is not reading from a live table.Event log entries now contain the
maturity
property to indicate the stability of the event schema. Possible values arestable
,evolving
, anddeprecated
. For more information about the Delta Live Tables event log, see What is the Delta Live Tables event log?.The error message is improved when incompatible changes are made to source tables used by a streaming table.
You can now select a cluster policy in the Delta Live Tables UI when you create or edit a pipeline. Previously, setting the cluster policy for a pipeline required editing the pipeline's JSON settings.
Faster pipeline startup. This release includes enhancements that speed up the
SETTING_UP_TABLES
step when a pipeline is starting.
Bug Fixes in this release
- This release fixes a bug that prevents Enhanced Autoscaling from scaling up when no idle cluster instances are available.