Runs CLI (legacy)

  • Article

Important

This documentation has been retired and might not be updated.

This information applies to legacy Databricks CLI versions 0.18 and below. Databricks recommends that you use newer Databricks CLI version 0.205 or above instead. See What is the Databricks CLI?. To find your version of the Databricks CLI, run databricks -v.

To migrate from Databricks CLI version 0.18 or below to Databricks CLI version 0.205 or above, see Databricks CLI migration.

You run Databricks job runs CLI subcommands by appending them to databricks runs and Databricks jobs CLI subcommands by appending them to databricks jobs. For Databricks jobs CLI subcommands, see the Jobs CLI (legacy). Together, these subcommands call the Jobs API and Jobs API 2.0.

Important

The Databricks job runs CLI supports calls to two versions of the Databricks Jobs REST API: versions 2.1 and 2.0. (Job runs functionality is part of the Jobs REST API.) Version 2.1 adds support for orchestration of jobs with multiple tasks; see Create and run Azure Databricks Jobs and Updating from Jobs API 2.0 to 2.1. Databricks recommends that you call version 2.1, unless you have legacy scripts that rely on version 2.0 and cannot be migrated.

Unless otherwise specified, the programmatic behaviors that are described in this article apply equally to versions 2.1 and 2.0.

Note

If you receive a 500-level error when making job runs CLI requests, Databricks recommends retrying requests for up to 10 minutes (with a minimum 30-second interval between retries).

Requirements to call the Jobs REST API 2.1

To set up the Databricks job runs CLI (and jobs CLI) to call the Jobs REST API 2.1, do the following:

  1. Update the CLI to version 0.16.0 or above.

  2. Do one of the following:

    • Run the command databricks jobs configure --version=2.1. This adds the setting jobs-api-version = 2.1 to the file ~/.databrickscfg on Unix, Linux, or macOS, or %USERPROFILE%\.databrickscfg on Windows. All job runs CLI (and jobs CLI) subcommands will call the Jobs REST API 2.1 by default.
    • Manually add the setting jobs-api-version = 2.1 to the file ~/.databrickscfg on Unix, Linux, or macOS, or %USERPROFILE%\.databrickscfg on Windows. All job runs CLI (and jobs CLI) subcommands will call the Jobs REST API 2.1 by default.
    • Append the option --version=2.1 (for example, databricks runs list --version=2.1) to instruct the job runs CLI to call the Jobs REST API 2.1 for that call only.

    If you take none of the preceding actions, the job runs CLI (and jobs CLI) will call the Jobs REST API 2.0 by default.

Requirements to call the Jobs REST API 2.0

To set up the Databricks job runs CLI (and jobs CLI) to call the Jobs REST API 2.0, do one of the following:

  • Use a version of the Databricks CLI below 0.16.0, or
  • Update the CLI to version 0.16.0 or above, and then do one of the following:
    • Run the command databricks jobs configure --version=2.0. This adds the setting jobs-api-version = 2.0 to the file ~/.databrickscfg on Unix, Linux, or macOS, or %USERPROFILE%\.databrickscfg on Windows. All job runs CLI (and jobs CLI) subcommands will call the Jobs REST API 2.0 by default.
    • Manually add the setting jobs-api-version = 2.0 to the file ~/.databrickscfg on Unix, Linux, or macOS, or %USERPROFILE%\.databrickscfg on Windows. All job runs CLI (and jobs CLI) subcommands will call the Jobs REST API 2.0 by default.
    • Append the option --version=2.1 (for example, databricks runs list --version=2.0) to instruct the job runs CLI to call the Jobs REST API 2.0 for that call only.

If you take none of the preceding actions, the job runs CLI (and jobs CLI) will call the Jobs REST API 2.0 by default.

Subcommands and general usage

databricks runs --help

Usage: databricks runs [OPTIONS] COMMAND [ARGS]...

  Utility to interact with jobs runs.

Options:
  -v, --version   [VERSION]
  --debug         Debug mode. Shows full stack trace on error.
  --profile TEXT  CLI connection profile to use. The default profile is
                  "DEFAULT".

  -h, --help      Show this message and exit.

Commands:
  cancel      Cancels the specified run.
  get         Gets the metadata about a run in JSON form.
  get-output  Gets the output of a run.
  list        Lists job runs.
  submit      Submits a one-time run.

Cancel a run

To display usage documentation, run databricks runs cancel --help.

databricks runs cancel --run-id 119

{}

Get information about a run

To display usage documentation, run databricks runs get --help.

General usage

databricks runs get --run-id 2785782

Jobs CLI 2.1 usage notes and response example

See Runs get in Updating from Jobs API 2.0 to 2.1.

Jobs CLI 2.0 response example

{
  "job_id": 1269263,
  "run_id": 2785782,
  "number_in_job": 1111,
  "original_attempt_run_id": 2785782,
  "state": {
    "life_cycle_state": "TERMINATED",
    "result_state": "SUCCESS",
    "state_message": ""
  },
  "task": {
    "notebook_task": {
      "notebook_path": "/Users/someone@example.com/notebooks/my-notebook.ipynb"
    }
  },
  "cluster_spec": {
    "new_cluster": {
      "spark_version": "8.1.x-scala2.12",
      "node_type_id": "Standard_F16s",
      "enable_elastic_disk": true,
      "azure_attributes": {
        "availability": "ON_DEMAND_AZURE"
      },
      "num_workers": 8
    }
  },
  "cluster_instance": {
    "cluster_id": "1234-567890-abcd123",
    "spark_context_id": "1234567890123456789"
  },
  "start_time": 1620947196985,
  "setup_duration": 36000,
  "execution_duration": 119000,
  "cleanup_duration": 3000,
  "end_time": 1620947355499,
  "trigger": "ONE_TIME",
  "creator_user_name": "someone@example.com",
  "run_name": "my-notebook-run",
  "run_page_url": "https://adb-1234567890123456.7.databricks.azure.cn/?o=1234567890123456#job/1269263/run/1111",
  "run_type": "JOB_RUN",
  "attempt_number": 0
}

Get the output of a run

To display usage documentation, run databricks runs get-output --help.

Note

When a notebook_task returns a value from a call to dbutils.notebook.exit(), Databricks limits the returned value to the first 5 MB of data. To return a larger result, you can store job results in a cloud storage service.

General usage

databricks runs get-output --run-id 2785782

Jobs CLI 2.1 usage notes

See Runs get output in Updating from Jobs API 2.0 to 2.1.

Jobs CLI 2.0 response example

{
  "metadata": {
    "job_id": 1269263,
    "run_id": 2785782,
    "number_in_job": 1111,
    "original_attempt_run_id": 2785782,
    "state": {
      "life_cycle_state": "TERMINATED",
      "result_state": "SUCCESS",
      "state_message": ""
    },
    "task": {
      "notebook_task": {
        "notebook_path": "/Users/someone@example.com/notebooks/my-notebook.ipynb"
      }
    },
    "cluster_spec": {
      "new_cluster": {
        "spark_version": "8.1.x-scala2.12",
        "node_type_id": "Standard_F16s",
        "enable_elastic_disk": true,
        "azure_attributes": {
          "availability": "ON_DEMAND_AZURE"
        },
        "num_workers": 8
      }
    },
    "cluster_instance": {
      "cluster_id": "1234-567890-abcd123",
      "spark_context_id": "1234567890123456789"
    },
    "start_time": 1620947196985,
    "setup_duration": 36000,
    "execution_duration": 119000,
    "cleanup_duration": 3000,
    "end_time": 1620947355499,
    "trigger": "ONE_TIME",
    "creator_user_name": "someone@example.com",
    "run_name": "my-notebook-run",
    "run_page_url": "https://adb-1234567890123456.7.databricks.azure.cn/?o=1234567890123456#job/1269263/run/1111",
    "run_type": "JOB_RUN",
    "attempt_number": 0
  },
  "notebook_output": {}
}

Get information about all runs

To display usage documentation, run databricks runs list --help.

General usage

databricks runs list --output JSON

Jobs CLI 2.1 usage notes and response example

See Runs list in Updating from Jobs API 2.0 to 2.1.

Jobs CLI 2.0 response example

{
  "runs": [
    {
      "job_id": 1269263,
      "run_id": 2785782,
      "number_in_job": 1111,
      "original_attempt_run_id": 2785782,
      "state": {
         "life_cycle_state": "TERMINATED",
         "result_state": "SUCCESS",
         "state_message": ""
      },
      "task": {
        "notebook_task": {
          "notebook_path": "/Users/someone@example.com/notebooks/my-notebook.ipynb"
        }
      },
      "cluster_spec": {
        "new_cluster": {
          "spark_version": "8.1.x-scala2.12",
          "node_type_id": "Standard_F16s",
          "enable_elastic_disk": true,
          "azure_attributes": {
            "availability": "ON_DEMAND_AZURE"
          },
          "num_workers": 8
        }
      },
      "cluster_instance": {
        "cluster_id": "1234-567890-abcd123",
        "spark_context_id": "1234567890123456789"
      },
      "start_time": 1620947196985,
      "setup_duration": 36000,
      "execution_duration": 119000,
      "cleanup_duration": 3000,
      "end_time": 1620947355499,
      "trigger": "ONE_TIME",
      "creator_user_name": "someone@example.com",
      "run_name": "my-notebook-run",
      "run_page_url": "https://adb-1234567890123456.7.databricks.azure.cn/?o=1234567890123456#job/1269263/run/1111",
      "run_type": "JOB_RUN",
      "attempt_number": 0
    },
    ...
  ],
  "has_more": false
}

If has_more returns true, information about additional runs is available. Use the --offset option to return information about runs relative to the most recent run. For example, to return information starting with the tenth most recent run, specify --offset 10.

Use the --limit option to return information about a fixed number of runs. For example, to return information for up to the next 5 runs, specify --limit 5. You can specify up to 1000 runs. If not specified, the default is 20.

Submit a one-time run

To display usage documentation, run databricks runs submit --help.

General usage

databricks runs submit --json-file submit-run.json

Jobs REST API 2.1 usage notes and request example

See Runs submit in Updating from Jobs API 2.0 to 2.1.

Jobs REST API 2.0 request and response example

submit-run.json:

{
  "run_name": "my-spark-run",
  "new_cluster": {
    "spark_version": "8.1.x-scala2.12",
    "node_type_id": "Standard_F16s",
    "enable_elastic_disk": true,
    "azure_attributes": {
      "availability": "ON_DEMAND_AZURE"
    },
    "num_workers": 8
  },
  "libraries": [
    {
      "jar": "dbfs:/my-jar.jar"
    },
    {
      "maven": {
        "coordinates": "org.jsoup:jsoup:1.7.2"
      }
    }
  ],
  "spark_jar_task": {
    "main_class_name": "com.databricks.ComputeModels"
  }
}

{
  "run_id": 123
}