az synapse spark job

  • Reference

Manage Synapse Spark batch jobs.

Commands

az synapse spark job cancel

Cancel a Spark job.
az synapse spark job list

List all Spark jobs.
az synapse spark job show

Get a Spark job.
az synapse spark job submit

Submit a Spark job.

az synapse spark job cancel

Edit

Cancel a Spark job.

az synapse spark job cancel --livy-id
                            --spark-pool-name
                            --workspace-name
                            [--subscription]
                            [--yes]

Examples

Cancel a Spark job.

az synapse spark job cancel --livy-id 1 --workspace-name testsynapseworkspace --spark-pool-name testsparkpool

Required Parameters

--livy-id

The id of the Spark job.

--spark-pool-name

The name of the Spark pool.

--workspace-name

The name of the workspace.

Optional Parameters

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--yes -y

Do not prompt for confirmation.

az synapse spark job list

Edit

List all Spark jobs.

az synapse spark job list --spark-pool-name
                          --workspace-name
                          [--from-index]
                          [--query-examples]
                          [--size]
                          [--subscription]

Examples

List all Spark jobs.

az synapse spark job list --workspace-name testsynapseworkspace --spark-pool-name testsparkpool

Required Parameters

--spark-pool-name

The name of the Spark pool.

--workspace-name

The name of the workspace.

Optional Parameters

--from-index

Optional parameter specifying which index the list should begin from.

--query-examples

Recommend JMESPath string for you. You can copy one of the query and paste it after --query parameter within double quotation marks to see the results. You can add one or more positional keywords so that we can give suggestions based on these key words.

--size

The size of the returned list.By default it is 20 and that is the maximum.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

az synapse spark job show

Edit

Get a Spark job.

az synapse spark job show --livy-id
                          --spark-pool-name
                          --workspace-name
                          [--query-examples]
                          [--subscription]

Examples

Get a Spark job.

az synapse spark job show --livy-id 1 --workspace-name testsynapseworkspace --spark-pool-name testsparkpool

Required Parameters

--livy-id

The id of the Spark job.

--spark-pool-name

The name of the Spark pool.

--workspace-name

The name of the workspace.

Optional Parameters

--query-examples

Recommend JMESPath string for you. You can copy one of the query and paste it after --query parameter within double quotation marks to see the results. You can add one or more positional keywords so that we can give suggestions based on these key words.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

az synapse spark job submit

Edit

Submit a Spark job.

az synapse spark job submit --executor-size {Large, Medium, Small}
                            --executors
                            --main-class-name
                            --main-definition-file
                            --name
                            --spark-pool-name
                            --workspace-name
                            [--archives]
                            [--arguments]
                            [--configuration]
                            [--language {CSharp, PySpark, Python, Scala, Spark, SparkDotNet}]
                            [--reference-files]
                            [--subscription]
                            [--tags]

Examples

Submit a Java Spark job.

az synapse spark job submit --name WordCount_Java --workspace-name testsynapseworkspace \
--spark-pool-name testsparkpool \
--main-definition-file abfss://testfilesystem@testadlsgen2.dfs.core.windows.net/samples/java/wordcount/wordcount.jar \
--main-class-name WordCount \
--arguments abfss://testfilesystem@testadlsgen2.dfs.core.windows.net/samples/java/wordcount/shakespeare.txt \
abfss://testfilesystem@testadlsgen2.dfs.core.windows.net/samples/java/wordcount/result/ \
--executors 2 --executor-size Small

Required Parameters

--executor-size

The executor size.

accepted values: Large, Medium, Small
--executors

The number of executors.

--main-class-name

The fully-qualified identifier or the main class that is in the main definition file.

--main-definition-file

The main file used for the job.

--name -n

The Spark job name.

--spark-pool-name

The name of the Spark pool.

--workspace-name

The name of the workspace.

Optional Parameters

--archives

The array of archives.

--arguments

Optional arguments to the job (Note: please use storage URIs for file arguments).

--configuration

The configuration of Spark job.

--language

The Spark job language.

accepted values: CSharp, PySpark, Python, Scala, Spark, SparkDotNet
default value: Scala
--reference-files

Additional files used for reference in the main definition file.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--tags

Space-separated tags: key[=value] [key[=value] ...]. Use "" to clear existing tags.