Pass context about job runs into job tasks

  • Article

You can use dynamic value references to pass context about a job or task run such as the job or task name, the identifier of a run, or the start time of a job run. Dynamic value references are templated variables that are replaced with the appropriate values when the job task runs. When a job runs, the task parameter variable surrounded by double curly braces is replaced and appended to an optional string value included as part of the value. For example, to pass a parameter named MyJobId with a value of my-job-6 for any run of job ID 6, add the following task parameter:

{
  "MyJobID": "my-job-{{job.id}}"
}

The contents of the double curly braces are not evaluated as expressions, so you cannot do operations or functions in double-curly braces.

User-provided identifiers, for example, task names, task value keys, or job parameter names containing special characters must be escaped by surrounding the identifiers with backticks (` `). Only alphanumeric and underscore characters can be used without escaping.

{
  "VariableWithSpecialChars": "{{job.parameters.`param$@`}}"
}

Syntax errors in references (for example, a missing brace) are ignored and the value is treated as a literal string. For example, {{my.value} is passed as the string "{{my.value}". However, entering an invalid reference that belongs to a known namespace (for example, {{job.naem}}) is not allowed. An error message is displayed if an invalid reference belonging to a known namespace is entered in the UI.

After a task completes, you can see resolved values for parameters under Parameters on the run details page.

Supported value references

The following dynamic value references are supported:

Reference Description
{{job.id}} The unique identifier assigned to the job.
{{job.name}} The name of the job at the time of the job run.
{{job.run_id}} The unique identifier assigned to the job run.
{{job.repair_count}} The number of repair attempts on the current job run.
{{job.start_time.[argument]}} A value based on the time (in UTC timezone) that the job run started. The return value is based on the argument option. See Options for date and time values.
{{job.parameters.[name]}} The value of the job-level parameter with the key [name].
{{job.trigger.type}} The trigger type of the job run. The possible values are periodic, onetime, run_job_task, file_arrival, continuous, and table.
{{job.trigger.file_arrival.location}} If a file arrival trigger is configured for this job, the value of the storage location.
{{job.trigger.time.[argument]}} A value based on the time (in UTC timezone) that the job run was triggered, rounded down to the closest minute for jobs with a cron schedule. The return value is based on the argument option. See Options for date and time values.
{{task.name}} The name of the current task.
{{task.run_id}} The unique identifier of the current task run.
{{task.execution_count}} The number of times the current task was run (including retries and repairs).
{{task.notebook_path}} The notebook path of the current notebook task.
{{tasks.[task_name].run_id}} The unique identifier assigned to the task run for [task_name].
{{tasks.[task_name].result_state}} The result state of task [task_name]. The possible values are success, failed, excluded, canceled, evicted, timedout, upstream_canceled, upstream_evicted, and upstream_failed.
{{tasks.[task_name].error_code}} The error code for task [task_name] if an error occurred running the task. Examples of possible values are RunExecutionError, ResourceNotFound, and UnauthorizedError. For successful tasks, this evaluates to an empty string.
{{tasks.[task_name].execution_count}} The number of times the task [task_name] was run (including retries and repairs).
{{tasks.[task_name].notebook_path}} The path to the notebook for the notebook task [task_name].
{{tasks.[task_name].values.[value_name]}} The task value with the key [value_name] that was set by task [task_name].
{{workspace.id}} The unique identifier assigned to the workspace.
{{workspace.url}} The URL of the workspace.

You can set these references with any task when you Create a job, Edit a job, or Run a job with different parameters.

You can also pass parameters between tasks in a job with task values. See Share information between tasks in an Azure Databricks job.

Options for date and time values

Use the following arguments to specify the return value from time based parameter variables. All return values are based on a timestamp in UTC timezone.

Argument Description
iso_weekday Returns a digit from 1 to 7, representing the day of the week of the timestamp.
is_weekday Returns true if the timestamp is on a weekday.
iso_date Returns the date in ISO format.
iso_datetime Returns the date and time in ISO format.
year Returns the year part of the timestamp.
month Returns the month part of the timestamp.
day Returns the day part of the timestamp.
hour Returns the hour part of the timestamp.
minute Returns the minute part of the timestamp.
second Returns the second part of the timestamp.
timestamp_ms Returns the timestamp in milliseconds.

Deprecated parameter variables

The following parameter variables are deprecated. Although they are still supported, any new jobs or updates to existing jobs should use the supported value references. The recommended replacement reference is included in the description of each variable.

Variable Description
{{job_id}} The unique identifier assigned to a job. Use job.id instead.
{{run_id}} The unique identifier assigned to a task run. Use task.run_id instead.
{{start_date}} The date a task run started. The format is yyyy-MM-dd in UTC timezone. Use job.start_time.[argument] instead.
{{start_time}} The timestamp of the run's start of execution after the cluster is created and ready. The format is milliseconds since UNIX epoch in UTC timezone, as returned by System.currentTimeMillis(). Use job.start_time.[format] instead.
{{task_retry_count}} The number of retries that have been attempted to run a task if the first attempt fails. The value is 0 for the first attempt and increments with each retry. Use task.execution_count instead.
{{parent_run_id}} The unique identifier assigned to the run of a job with multiple tasks. Use job.run_id instead.
{{task_key}} The unique name assigned to a task that's part of a job with multiple tasks. Use task.name instead.