Quickstart: Use the Azure CLI to create a Batch account and run a job

This quickstart shows you how to get started with Azure Batch by using Azure CLI commands and scripts to create and manage Batch resources. You create a Batch account that has a pool of virtual machines, or compute nodes. You then create and run a job with tasks that run on the pool nodes.

After you complete this quickstart, you understand the key concepts of the Batch service and are ready to use Batch with more realistic, larger scale workloads.

If you don't have an Azure subscription, create a trial account before you begin.

• If you prefer to run CLI reference commands locally, install the Azure CLI. If you're running on Windows or macOS, consider running Azure CLI in a Docker container. For more information, see How to run the Azure CLI in a Docker container.

  • If you're using a local installation, sign in to the Azure CLI by using the az login command. To finish the authentication process, follow the steps displayed in your terminal. For other sign-in options, see Sign in with the Azure CLI.

  • When you're prompted, install the Azure CLI extension on first use. For more information about extensions, see Use extensions with the Azure CLI.

  • Run az version to find the version and dependent libraries that are installed. To upgrade to the latest version, run az upgrade.

• This quickstart requires version 2.0.20 or later of the Azure CLI.

Run the following az group create command to create an Azure resource group. The resource group is a logical container that holds the Azure resources for this quickstart.

export RANDOM_SUFFIX=$(openssl rand -hex 3)
export REGION="canadacentral"
export RESOURCE_GROUP="qsBatch$RANDOM_SUFFIX"

az group create \
    --name $RESOURCE_GROUP \
    --location $REGION

Results:

{
    "id": "/subscriptions/xxxxx/resourceGroups/qsBatchxxx",
    "location": "chinaeast2",
    "managedBy": null,
    "name": "qsBatchxxx",
    "properties": {
         "provisioningState": "Succeeded"
    },
    "tags": null,
    "type": "Microsoft.Resources/resourceGroups"
}

Use the az storage account create command to create an Azure Storage account to link to your Batch account. Although this quickstart doesn't use the storage account, most real-world Batch workloads use a linked storage account to deploy applications and store input and output data.

Run the following command to create a Standard_LRS SKU storage account in your resource group:

export STORAGE_ACCOUNT="mybatchstorage$RANDOM_SUFFIX"

az storage account create \
    --resource-group $RESOURCE_GROUP \
    --name $STORAGE_ACCOUNT \
    --location $REGION \
    --sku Standard_LRS

Run the following az batch account create command to create a Batch account in your resource group and link it with the storage account.

export BATCH_ACCOUNT="mybatchaccount$RANDOM_SUFFIX"

az batch account create \
    --name $BATCH_ACCOUNT \
    --storage-account $STORAGE_ACCOUNT \
    --resource-group $RESOURCE_GROUP \
    --location $REGION

Sign in to the new Batch account by running the az batch account login command. Once you authenticate your account with Batch, subsequent az batch commands in this session use this account context.

az batch account login \
    --name $BATCH_ACCOUNT \
    --resource-group $RESOURCE_GROUP \
    --shared-key-auth

Run the az batch pool create command to create a pool of Linux compute nodes in your Batch account. The following example creates a pool that consists of two Standard_A1_v2 size VMs running Ubuntu 20.04 LTS OS. This node size offers a good balance of performance versus cost for this quickstart example.

export POOL_ID="myPool$RANDOM_SUFFIX"

az batch pool create \
    --id $POOL_ID \
    --image canonical:0001-com-ubuntu-server-focal:20_04-lts \
    --node-agent-sku-id "batch.node.ubuntu 20.04" \
    --target-dedicated-nodes 2 \
    --vm-size Standard_A1_v2

Batch creates the pool immediately, but takes a few minutes to allocate and start the compute nodes. To see the pool status, use the az batch pool show command. This command shows all the properties of the pool, and you can query for specific properties. The following command queries for the pool allocation state:

az batch pool show --pool-id $POOL_ID \
    --query "{allocationState: allocationState}"

Results:

{
    "allocationState": "resizing"
}

While Batch allocates and starts the nodes, the pool is in the resizing state. You can create a job and tasks while the pool state is still resizing. The pool is ready to run tasks when the allocation state is steady and all the nodes are running.

Use the az batch job create command to create a Batch job to run on your pool. A Batch job is a logical group of one or more tasks. The job includes settings common to the tasks, such as the pool to run on. The following example creates a job that initially has no tasks.

export JOB_ID="myJob$RANDOM_SUFFIX"

az batch job create \
    --id $JOB_ID \
    --pool-id $POOL_ID

Batch provides several ways to deploy apps and scripts to compute nodes. Use the az batch task create command to create tasks to run in the job. Each task has a command line that specifies an app or script.

The following Bash script creates four identical, parallel tasks called myTask1 through myTask4. The task command line displays the Batch environment variables on the compute node, and then waits 90 seconds.

for i in {1..4}
do
   az batch task create \
    --task-id myTask$i \
    --job-id $JOB_ID \
    --command-line "/bin/bash -c 'printenv | grep AZ_BATCH; sleep 90s'"
done

Batch distributes the tasks to the compute nodes.

After you create the tasks, Batch queues them to run on the pool. Once a node is available, a task runs on the node.

Use the az batch task show command to view the status of Batch tasks. The following example shows details about the status of myTask1:

az batch task show \
    --job-id $JOB_ID \
    --task-id myTask1

The command output includes many details. For example, an exitCode of 0 indicates that the task command completed successfully. The nodeId shows the name of the pool node that ran the task.

Use the az batch task file list command to list the files a task created on a node. The following command lists the files that myTask1 created:

# Wait for task to complete before downloading output
echo "Waiting for task to complete..."
while true; do
    STATUS=$(az batch task show --job-id $JOB_ID --task-id myTask1 --query "state" -o tsv)
    if [ "$STATUS" == "running" ]; then
        break
    fi
    sleep 10
done

az batch task file list --job-id $JOB_ID --task-id myTask1 --output table

Results are similar to the following output:

Results:

Name        URL                                                                                       Is Directory    Content Length
---------- ----------------------------------------------------------------------------------------  -------------- ----------------
stdout.txt  https://mybatchaccount.chinaeast2.batch.chinacloudapi.cn/jobs/myJob/tasks/myTask1/files/stdout.txt  False                  695
certs       https://mybatchaccount.chinaeast2.batch.chinacloudapi.cn/jobs/myJob/tasks/myTask1/files/certs       True
wd          https://mybatchaccount.chinaeast2.batch.chinacloudapi.cn/jobs/myJob/tasks/myTask1/files/wd          True
stderr.txt  https://mybatchaccount.chinaeast2.batch.chinacloudapi.cn/jobs/myJob/tasks/myTask1/files/stderr.txt  False                    0

The az batch task file download command downloads output files to a local directory. Run the following example to download the stdout.txt file:

az batch task file download \
    --job-id $JOB_ID \
    --task-id myTask1 \
    --file-path stdout.txt \
    --destination ./stdout.txt

You can view the contents of the standard output file in a text editor. The following example shows a typical stdout.txt file. The standard output from this task shows the Azure Batch environment variables that are set on the node. You can refer to these environment variables in your Batch job task command lines, and in the apps and scripts the command lines run.

AZ_BATCH_TASK_DIR=/mnt/batch/tasks/workitems/myJob/job-1/myTask1
AZ_BATCH_NODE_STARTUP_DIR=/mnt/batch/tasks/startup
AZ_BATCH_CERTIFICATES_DIR=/mnt/batch/tasks/workitems/myJob/job-1/myTask1/certs
AZ_BATCH_ACCOUNT_URL=https://mybatchaccount.chinaeast2.batch.chinacloudapi.cn/
AZ_BATCH_TASK_WORKING_DIR=/mnt/batch/tasks/workitems/myJob/job-1/myTask1/wd
AZ_BATCH_NODE_SHARED_DIR=/mnt/batch/tasks/shared
AZ_BATCH_TASK_USER=_azbatch
AZ_BATCH_NODE_ROOT_DIR=/mnt/batch/tasks
AZ_BATCH_JOB_ID=myJob
AZ_BATCH_NODE_IS_DEDICATED=true
AZ_BATCH_NODE_ID=tvm-257509324_2-20180703t215033z
AZ_BATCH_POOL_ID=myPool
AZ_BATCH_TASK_ID=myTask1
AZ_BATCH_ACCOUNT_NAME=mybatchaccount
AZ_BATCH_TASK_USER_IDENTITY=PoolNonAdmin

In this quickstart, you created a Batch account and pool, created and ran a Batch job and tasks, and viewed task output from the nodes. Now that you understand the key concepts of the Batch service, you're ready to use Batch with more realistic, larger scale workloads. To learn more about Azure Batch, continue to the Azure Batch tutorials.