az hdinsight

Manage HDInsight resources.

Commands

az hdinsight application

Manage HDInsight applications.

az hdinsight application create

Create an application for a HDInsight cluster.

az hdinsight application delete

Deletes the specified application on the HDInsight cluster.

az hdinsight application list

Lists all of the applications for the HDInsight cluster.

az hdinsight application show

Gets properties of the specified application.

az hdinsight application wait

Place the CLI in a waiting state until an operation is complete.

az hdinsight autoscale

Manage HDInsight cluster's Autoscale configuration.

az hdinsight autoscale condition

Manage schedule condition of the HDInsight cluster which enabled Schedule-based Autoscale.

az hdinsight autoscale condition create

Add a new schedule condition.

az hdinsight autoscale condition delete

Delete schedule condition.

az hdinsight autoscale condition list

List all schedule conditions.

az hdinsight autoscale condition update

Update a schedule condition.

az hdinsight autoscale condition wait

Place the CLI in a waiting state until an operation is complete.

az hdinsight autoscale create

Enable Autoscale for a running cluster.

az hdinsight autoscale delete

Disable Autoscale for a running cluster.

az hdinsight autoscale list-timezones

List the available timezone name when enabling Schedule-based Autoscale.

az hdinsight autoscale show

Get the Autoscale configuration of a specified cluster.

az hdinsight autoscale update

Update the Autoscale configuration.

az hdinsight autoscale wait

Place the CLI in a waiting state until an operation is complete.

az hdinsight create

Create a new cluster.

az hdinsight delete

Deletes the specified HDInsight cluster.

az hdinsight host

Manage HDInsight cluster's virtual hosts.

az hdinsight host list

List the hosts of the specified HDInsight cluster.

az hdinsight host restart

Restart the specific hosts of the specified HDInsight cluster.

az hdinsight list

List HDInsight clusters in a resource group or subscription.

az hdinsight list-usage

Lists the usages for the specified location.

az hdinsight monitor

Manage Azure Monitor logs integration on an HDInsight cluster.

az hdinsight monitor disable

Disable the Azure Monitor logs integration on an HDInsight cluster.

az hdinsight monitor enable

Enable the Azure Monitor logs integration on an HDInsight cluster.

az hdinsight monitor show

Get the status of Azure Monitor logs integration on an HDInsight cluster.

az hdinsight resize

Resizes the specified HDInsight cluster to the specified size.

az hdinsight rotate-disk-encryption-key

Rotate the disk encryption key of the specified HDInsight cluster.

az hdinsight script-action

Manage HDInsight script actions.

az hdinsight script-action delete

Deletes a specified persisted script action of the cluster.

az hdinsight script-action execute

Execute script actions on the specified HDInsight cluster.

az hdinsight script-action list

Lists all the persisted script actions for the specified cluster.

az hdinsight script-action list-execution-history

Lists all scripts' execution history for the specified cluster.

az hdinsight script-action promote

Promotes the specified ad-hoc script execution to a persisted script.

az hdinsight script-action show-execution-details

Gets the script execution detail for the given script execution ID.

az hdinsight show

Gets the specified cluster.

az hdinsight update

Patch HDInsight cluster with the specified parameters.

az hdinsight wait

Place the CLI in a waiting state until an operation is complete.

az hdinsight create

Create a new cluster.

az hdinsight create --name
                    --resource-group
                    --type
                    [--assign-identity]
                    [--autoscale-count]
                    [--autoscale-max-count]
                    [--autoscale-min-count]
                    [--autoscale-type {Load, Schedule}]
                    [--cluster-admin-account]
                    [--cluster-admin-password]
                    [--cluster-configurations]
                    [--cluster-tier {Premium, Standard}]
                    [--cluster-users-group-dns]
                    [--component-version]
                    [--days {Friday, Monday, Saturday, Sunday, Thursday, Tuesday, Wednesday}]
                    [--domain]
                    [--edgenode-size]
                    [--encryption-algorithm {RSA-OAEP, RSA-OAEP-256, RSA1_5}]
                    [--encryption-at-host {false, true}]
                    [--encryption-in-transit {false, true}]
                    [--encryption-key-name]
                    [--encryption-key-version]
                    [--encryption-vault-uri]
                    [--esp]
                    [--headnode-size]
                    [--http-password]
                    [--http-user]
                    [--kafka-client-group-id]
                    [--kafka-client-group-name]
                    [--kafka-management-node-count]
                    [--kafka-management-node-size]
                    [--ldaps-urls]
                    [--location]
                    [--minimal-tls-version {1.0, 1.1, 1.2}]
                    [--no-validation-timeout]
                    [--no-wait]
                    [--outbound-public-network-access-type {PublicLoadBalancer, UDR}]
                    [--public-network-access-type {InboundAndOutbound, OutboundOnly}]
                    [--ssh-password]
                    [--ssh-public-key]
                    [--ssh-user]
                    [--storage-account]
                    [--storage-account-key]
                    [--storage-account-managed-identity]
                    [--storage-container]
                    [--storage-filesystem]
                    [--subnet]
                    [--subscription]
                    [--tags]
                    [--time]
                    [--timezone]
                    [--version]
                    [--vnet-name]
                    [--workernode-count]
                    [--workernode-data-disk-size]
                    [--workernode-data-disk-storage-account-type {premium_lrs, standard_lrs}]
                    [--workernode-data-disks-per-node]
                    [--workernode-size]
                    [--zookeepernode-size]

Examples

Create a cluster with an existing storage account.

az hdinsight create -t spark -g MyResourceGroup -n MyCluster \
-p "HttpPassword1234!" \
--storage-account MyStorageAccount

Create a cluster with minimal tls version.

az hdinsight create -t spark -g MyResourceGroup -n MyCluster \
-p "HttpPassword1234!" \
--storage-account MyStorageAccount --minimal-tls-version 1.2

Create a cluster which enables encryption in transit.

az hdinsight create -t spark -g MyResourceGroup -n MyCluster \
-p "HttpPassword1234!" \
--storage-account MyStorageAccount --encryption-in-transit true

Create a cluster with encryption at host.

az hdinsight create -t spark -g MyResourceGroup -n MyCluster \
-p "HttpPassword1234!" \
--storage-account MyStorageAccount --encryption-at-host true

Create a cluster with private link settings.

az hdinsight create --esp -t spark -g MyResourceGroup -n MyCluster \
-p "HttpPassword1234!" \
--storage-account MyStorageAccount \
--subnet "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/MyRG/providers/Microsoft.Network/virtualNetworks/MyVnet/subnets/subnet1" \
--public-network-access-type OutboundOnly --outbound-public-network-access-type PublicLoadBalancer

Create a cluster with the Enterprise Security Package (ESP).

az hdinsight create --esp -t spark -g MyResourceGroup -n MyCluster \
-p "HttpPassword1234!" \
--storage-account MyStorageAccount \
--subnet "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/MyRG/providers/Microsoft.Network/virtualNetworks/MyVnet/subnets/subnet1" \
--domain "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/MyRG/providers/Microsoft.AAD/domainServices/MyDomain.onmicrosoft.com" \
--assign-identity "/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/MyMsiRG/providers/Microsoft.ManagedIdentity/userAssignedIdentities/MyMSI" \
--cluster-admin-account MyAdminAccount@MyDomain.onmicrosoft.com \
--cluster-users-group-dns MyGroup

Create a Kafka cluster with disk encryption. See https://docs.microsoft.com/azure/hdinsight/kafka/apache-kafka-byok.

az hdinsight create -t kafka -g MyResourceGroup -n MyCluster \
-p "HttpPassword1234!" --workernode-data-disks-per-node 2 \
--storage-account MyStorageAccount \
--encryption-key-name kafkaClusterKey \
--encryption-key-version 00000000000000000000000000000000 \
--encryption-vault-uri https://MyKeyVault.vault.azure.net \
--assign-identity MyMSI

Create a kafka cluster with kafka rest proxy.

az hdinsight create -t kafka -g MyResourceGroup -n MyCluster \
-p "HttpPassword1234!" --workernode-data-disks-per-node 2 \
--storage-account MyStorageAccount \
--kafka-management-node-size "Standard_D4_v2" \
--kafka-client-group-id MySecurityGroupId \
--kafka-client-group-name MySecurityGroupName
--component-version kafka=2.1

Create a cluster with Azure Data Lake Storage Gen2

az hdinsight create -t spark -g MyResourceGroup -n MyCluster \
-p "HttpPassword1234!" \
--storage-account MyStorageAccount \
--storage-account-managed-identity MyMSI

Create a cluster with configuration from JSON string.

az hdinsight create -t spark -g MyResourceGroup -n MyCluster \
-p "HttpPassword1234!" \
--storage-account MyStorageAccount \
--cluster-configuration {'gateway':{'restAuthCredential.username':'admin'}}

Create a cluster with configuration from a local file.

az hdinsight create -t spark -g MyResourceGroup -n MyCluster \
-p "HttpPassword1234!" \
--storage-account MyStorageAccount \
--cluster-configuration @config.json

Create a cluster which Load-based Autoscale.

az hdinsight create -t spark --version 3.6 -g MyResourceGroup -n MyCluster \
-p "HttpPassword1234!" --storage-account MyStorageAccount \
--autoscale-type Load --autoscale-min-workernode-count 3 --autoscale-max-workernode-count 5

Create a cluster which Schedule-based Autoscale.

az hdinsight create -t spark --version 3.6 -g MyResourceGroup -n MyCluster \
-p "HttpPassword1234!" --storage-account MyStorageAccount \
--autoscale-type Schedule --timezone "Pacific Standard Time" --days Monday \
--time 09:00 --autoscale-workernode-count 5

Required Parameters

--name -n

The name of the cluster.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--type -t

Type of HDInsight cluster, like: hadoop, interactivehive, hbase, kafka, storm, spark, rserver, mlservices. See also: https://docs.microsoft.com/azure/hdinsight/hdinsight-hadoop-provision-linux-clusters#cluster-types.

Optional Parameters

--assign-identity

The name or ID of user assigned identity.

--autoscale-count --autoscale-workernode-count

The scheduled workernode count.

--autoscale-max-count --autoscale-max-workernode-count

The max workernode count for Load-based atuoscale.

--autoscale-min-count --autoscale-min-workernode-count

The minimal workernode count for Load-based atuoscale.

--autoscale-type

The autoscale type.

accepted values: Load, Schedule
--cluster-admin-account

The domain user account that will have admin privileges on the cluster. Required only when create cluster with Enterprise Security Package.

--cluster-admin-password

The domain admin password. Required only when create cluster with Enterprise Security Package.

--cluster-configurations

Extra configurations of various components. Configurations may be supplied from a file using the @{path} syntax or a JSON string. See also: https://docs.microsoft.com/azure/hdinsight/hdinsight-hadoop-customize-cluster-bootstrap.

--cluster-tier

The tier of the cluster.

accepted values: Premium, Standard
--cluster-users-group-dns

A space-delimited list of Distinguished Names for cluster user groups. Required only when create cluster with Enterprise Security Package.

--component-version

The versions of various Hadoop components, in space-separated versions in 'component=version' format. Example: Spark=2.0 Hadoop=2.7.3 See also: https://docs.microsoft.com/azure/hdinsight/hdinsight-component-versioning#hadoop-components-available-with-different-hdinsight-versions.

--days

A space-delimited list of schedule day.

accepted values: Friday, Monday, Saturday, Sunday, Thursday, Tuesday, Wednesday
--domain

The name or resource ID of the user's Azure Active Directory Domain Service. Required only when create cluster with Enterprise Security Package.

--edgenode-size
--encryption-algorithm

Algorithm identifier for encryption.

accepted values: RSA-OAEP, RSA-OAEP-256, RSA1_5
default value: RSA-OAEP
--encryption-at-host

Indicates whether enable encryption at host or not.

accepted values: false, true
--encryption-in-transit

Indicates whether enable encryption in transit.

accepted values: false, true
--encryption-key-name

Key name that is used for enabling disk encryption.

--encryption-key-version

Key version that is used for enabling disk encryption.

--encryption-vault-uri

Base key vault URI where the customers key is located eg. https://myvault.vault.azure.net.

--esp

Specify to create cluster with Enterprise Security Package. If omitted, creating cluster with Enterprise Security Package will not not allowed.

--headnode-size
--http-password -p

HTTP password for the cluster. Will prompt if not given.

--http-user -u

HTTP username for the cluster. Default: admin.

--kafka-client-group-id

The client AAD security group id for Kafka Rest Proxy.

--kafka-client-group-name

The client AAD security group name for Kafka Rest Proxy.

--kafka-management-node-count

The number of kafka management node in the cluster.

default value: 2
--kafka-management-node-size
--ldaps-urls

A space-delimited list of LDAPS protocol URLs to communicate with the Active Directory. Required only when create cluster with Enterprise Security Package.

--location -l

Location. Values from: az account list-locations. You can configure the default location using az configure --defaults location=<location>.

--minimal-tls-version

The minimal supported TLS version.

accepted values: 1.0, 1.1, 1.2
--no-validation-timeout

Permit timeout error during argument validation phase. If omitted, validation timeout error will be permitted.

--no-wait

Do not wait for the long-running operation to finish.

--outbound-public-network-access-type

The outbound only public network access type.

accepted values: PublicLoadBalancer, UDR
--public-network-access-type

The public network access type.

accepted values: InboundAndOutbound, OutboundOnly
--ssh-password -P

SSH password for the cluster nodes. If none specified, uses the HTTP password.

--ssh-public-key -K

SSH public key for the cluster nodes.

--ssh-user -U

SSH username for the cluster nodes.

default value: sshuser
--storage-account

The name or ID of the storage account.

--storage-account-key

The storage account key. A key can be retrieved automatically if the user has access to the storage account.

--storage-account-managed-identity

User-assigned managed identity with access to the storage account filesystem. Only required when storage account type is Azure Data Lake Storage Gen2.

--storage-container

The storage container the cluster will use. Uses the cluster name if none was specified. (WASB only).

--storage-filesystem

The storage filesystem the cluster will use. Uses the cluster name if none was specified. (DFS only).

--subnet

The name or ID of subnet. If name is supplied, --vnet-name must be supplied.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--tags

Space-separated tags: key[=value] [key[=value] ...]. Use "" to clear existing tags.

--time

The 24-hour time in the form of xx:xx in days.

--timezone

The timezone for schedule autoscale type. Values from az hdinsight autoscale list-timezones.

--version -v
--vnet-name

The name of a virtual network.

--workernode-count -c

The number of worker nodes in the cluster.

default value: 3
--workernode-data-disk-size

The size of the data disk in GB, e.g. 1023.

--workernode-data-disk-storage-account-type

The type of storage account that will be used for the data disks: standard_lrs or premium_lrs.

accepted values: premium_lrs, standard_lrs
--workernode-data-disks-per-node

The number of data disks to use per worker node.

--workernode-size
--zookeepernode-size

az hdinsight delete

Deletes the specified HDInsight cluster.

az hdinsight delete --name
                    --resource-group
                    [--no-wait]
                    [--subscription]
                    [--yes]

Required Parameters

--name -n

The name of the cluster.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

Optional Parameters

--no-wait

Do not wait for the long-running operation to finish.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--yes -y

Do not prompt for confirmation.

az hdinsight list

List HDInsight clusters in a resource group or subscription.

az hdinsight list [--query-examples]
                  [--resource-group]
                  [--subscription]

Optional Parameters

--query-examples

Recommend JMESPath string for you. You can copy one of the query and paste it after --query parameter within double quotation marks to see the results. You can add one or more positional keywords so that we can give suggestions based on these key words.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

az hdinsight list-usage

Lists the usages for the specified location.

az hdinsight list-usage --location
                        [--subscription]

Required Parameters

--location -l

Location. Values from: az account list-locations. You can configure the default location using az configure --defaults location=<location>.

Optional Parameters

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

az hdinsight resize

Resizes the specified HDInsight cluster to the specified size.

az hdinsight resize --name
                    --resource-group
                    --workernode-count
                    [--no-wait]
                    [--subscription]

Required Parameters

--name -n

The name of the cluster.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

--workernode-count -c

The target worker node instance count for the operation.

Optional Parameters

--no-wait

Do not wait for the long-running operation to finish.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

az hdinsight rotate-disk-encryption-key

Rotate the disk encryption key of the specified HDInsight cluster.

az hdinsight rotate-disk-encryption-key --encryption-key-name
                                        --encryption-key-version
                                        --encryption-vault-uri
                                        --name
                                        --resource-group
                                        [--no-wait]
                                        [--subscription]

Required Parameters

--encryption-key-name

Key name that is used for enabling disk encryption.

--encryption-key-version

Key version that is used for enabling disk encryption.

--encryption-vault-uri

Base key vault URI where the customers key is located eg. https://myvault.vault.azure.net.

--name -n

The name of the cluster.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

Optional Parameters

--no-wait

Do not wait for the long-running operation to finish.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

az hdinsight show

Gets the specified cluster.

az hdinsight show --name
                  --resource-group
                  [--query-examples]
                  [--subscription]

Required Parameters

--name -n

The name of the cluster.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

Optional Parameters

--query-examples

Recommend JMESPath string for you. You can copy one of the query and paste it after --query parameter within double quotation marks to see the results. You can add one or more positional keywords so that we can give suggestions based on these key words.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

az hdinsight update

Patch HDInsight cluster with the specified parameters.

az hdinsight update --name
                    --resource-group
                    [--no-wait]
                    [--subscription]
                    [--tags]

Required Parameters

--name -n

The name of the cluster.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

Optional Parameters

--no-wait

Do not wait for the long-running operation to finish.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--tags

Space-separated tags: key[=value] [key[=value] ...]. Use "" to clear existing tags.

az hdinsight wait

Place the CLI in a waiting state until an operation is complete.

az hdinsight wait --name
                  --resource-group
                  [--created]
                  [--custom]
                  [--deleted]
                  [--exists]
                  [--interval]
                  [--subscription]
                  [--timeout]
                  [--updated]

Required Parameters

--name -n

The name of the cluster.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

Optional Parameters

--created

Wait until created with 'provisioningState' at 'Succeeded'.

--custom

Wait until the condition satisfies a custom JMESPath query. E.g. provisioningState!='InProgress', instanceView.statuses[?code=='PowerState/running'].

--deleted

Wait until deleted.

--exists

Wait until the resource exists.

--interval

Polling interval in seconds.

default value: 30
--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--timeout

Maximum wait in seconds.

default value: 3600
--updated

Wait until updated with provisioningState at 'Succeeded'.