Delta Live Tables CLI (legacy)
Important
This documentation has been retired and might not be updated.
This information applies to legacy Databricks CLI versions 0.18 and below. Databricks recommends that you use newer Databricks CLI version 0.205 or above instead. See What is the Databricks CLI?. To find your version of the Databricks CLI, run databricks -v.
To migrate from Databricks CLI version 0.18 or below to Databricks CLI version 0.205 or above, see Databricks CLI migration.
You run Databricks Delta Live Tables CLI subcommands by appending them to databricks pipelines. These subcommands call the Delta Live Tables API.
databricks pipelines -h
Usage: databricks pipelines [OPTIONS] COMMAND [ARGS]...
Utility to interact with Databricks Delta Live Tables Pipelines.
Options:
-v, --version 0.18.0
--debug Debug Mode. Shows full stack trace on error.
--profile TEXT CLI connection profile to use. The default profile is
"DEFAULT".
-h, --help Show this message and exit.
Commands:
create Creates a pipeline.
delete Deletes the pipeline and cancels any active updates.
deploy [Deprecated] This command is deprecated, use create and edit
commands instead. Creates or edits a pipeline specified by the
pipeline settings.
edit Edits a pipeline.
get Gets a pipeline's current settings and status.
list Lists all pipelines and their statuses.
reset [Deprecated] Use the "start --full-refresh" command instead. Resets
a pipeline so that data can be reprocessed from the beginning.
run [Deprecated] Use the "start" command instead. Starts a pipeline
update.
start Starts a pipeline update.
stop Stops the pipeline by cancelling any active update.
Create a pipeline
To display usage documentation, run databricks pipelines create --help.
Usage
databricks pipelines create --settings create-pipeline.json
create-pipeline.json:
{
"name": "Wikipedia pipeline (SQL)",
"storage": "/Users/username/data",
"clusters": [
{
"label": "default",
"autoscale": {
"min_workers": 1,
"max_workers": 5
}
}
],
"libraries": [
{
"notebook": {
"path": "/Users/username/Delta Live Tables quickstart (SQL)"
}
}
],
"target": "wikipedia_analysis",
"continuous": false
}
Successfully created pipeline: https://<databricks-instance>/#joblist/pipelines/4ee6b5c2-7017-11ec-90d6-0242ac120003 with ID: 4ee6b5c2-7017-11ec-90d6-0242ac120003.
Edit a pipeline
To display usage documentation, run databricks pipelines edit --help.
Usage
databricks pipelines edit --settings edit-pipeline.json
edit-pipeline.json:
{
"id": "4ee6b5c2-7017-11ec-90d6-0242ac120003",
"name": "Wikipedia pipeline (SQL)",
"storage": "/Users/username/data",
"clusters": [
{
"label": "default",
"autoscale": {
"min_workers": 1,
"max_workers": 3
}
}
],
"libraries": [
{
"notebook": {
"path": "/Users/username/Delta Live Tables quickstart (SQL)"
}
}
],
"target": "wikipedia_quickstart_data",
"continuous": false
}
Successfully edited pipeline settings: https://<databricks-instance>/#joblist/pipelines/4ee6b5c2-7017-11ec-90d6-0242ac120003.
Delete a pipeline
To display usage documentation, run databricks pipelines delete --help.
Usage
databricks pipelines delete --pipeline-id 4ee6b5c2-7017-11ec-90d6-0242ac120003
Pipeline 4ee6b5c2-7017-11ec-90d6-0242ac120003 deleted
Get information about a pipeline
To display usage documentation, run databricks pipelines get --help.
Usage
databricks pipelines get --pipeline-id 4ee6b5c2-7017-11ec-90d6-0242ac120003
{
"pipeline_id": "4ee6b5c2-7017-11ec-90d6-0242ac120003",
"spec": {
"id": "4ee6b5c2-7017-11ec-90d6-0242ac120003",
"name": "Wikipedia pipeline (SQL)",
"storage": "/Users/username/data",
"clusters": [
{
"label": "default",
"autoscale": {
"min_workers": 1,
"max_workers": 5
}
}
],
"libraries": [
{
"notebook": {
"path": "/Users/username/Delta Live Tables quickstart (SQL)"
}
}
],
"target": "wikipedia_analysis",
"continuous": false
},
"state": "IDLE",
"name": "Wikipedia pipeline (SQL)",
"creator_user_name": "username@databricks.com",
"last_modified": 1641600056265
}
List all pipelines and information on their status
To display usage documentation, run databricks pipelines list --help.
Usage
databricks pipelines list
[
{
"pipeline_id": "8a93e1b4-7024-11ec-90d6-0242ac120003",
"state": "IDLE",
"name": "Retail sales pipeline (SQL)",
"latest_updates": [
{
"update_id": "9df03924-7024-11ec-90d6-0242ac120003",
"state": "COMPLETED",
"creation_time": "2021-10-29T16:18:57.982Z"
},
{
"update_id": "a57e601c-7024-11ec-90d6-0242ac120003",
"state": "COMPLETED",
"creation_time": "2021-10-28T18:19:30.371Z"
}
],
"creator_user_name": "user@databricks.com"
},
{
"pipeline_id": "b46e2670-7024-11ec-90d6-0242ac120003",
"state": "IDLE",
"name": "DLT quickstart (Python)",
"latest_updates": [
{
"update_id": "bcd8fa2e-7024-11ec-90d6-0242ac120003",
"state": "COMPLETED",
"creation_time": "2021-12-16T18:19:25.827Z"
},
{
"update_id": "c2c7a2c8-7024-11ec-90d6-0242ac120003",
"state": "COMPLETED",
"creation_time": "2021-10-29T22:22:32.586Z"
},
{
"update_id": "c88d2818-7024-11ec-90d6-0242ac120003",
"state": "COMPLETED",
"creation_time": "2021-10-29T22:16:18.258Z"
}
],
"creator_user_name": "user@databricks.com"
}
]
Start a pipeline update
To display usage documentation, run databricks pipelines start --help.
databricks pipelines start --pipeline-id 4ee6b5c2-7017-11ec-90d6-0242ac120003
Started an update c6462e2e-beb7-4c24-a9d3-962eaa19781b for pipeline 4ee6b5c2-7017-11ec-90d6-0242ac120003.
Start a pipeline update with full refresh
To display usage documentation, run databricks pipelines start --help.
Usage
databricks pipelines start --pipeline-id 4ee6b5c2-7017-11ec-90d6-0242ac120003 --full-refresh
Started an update ed885d3d-4fc0-44fd-b5dd-59471c789327 with full refresh for pipeline 4ee6b5c2-7017-11ec-90d6-0242ac120003.
Stop a pipeline update
To display usage documentation, run databricks pipelines stop --help.
Usage
databricks pipelines stop --pipeline-id 4ee6b5c2-7017-11ec-90d6-0242ac120003
Stopped pipeline 4ee6b5c2-7017-11ec-90d6-0242ac120003.