Manage Apache Hadoop clusters in HDInsight by using Azure PowerShell
Azure PowerShell can be used to control and automate the deployment and management of your workloads in Azure. In this article, you learn how to manage Apache Hadoop clusters in Azure HDInsight by using the Azure PowerShell Az module. For the list of the HDInsight PowerShell cmdlets, see the Az.HDInsight reference.
If you don't have an Azure subscription, create a trial subscription before you begin.
Prerequisites
Note
We recommend that you use the Azure Az PowerShell module to interact with Azure. To get started, see Install Azure PowerShell. To learn how to migrate to the Az PowerShell module, see Migrate Azure PowerShell from AzureRM to Az.
The PowerShell Az Module installed.
Create clusters
See Create Linux-based clusters in HDInsight using Azure PowerShell
List clusters
Use the following command to list all clusters in the current subscription:
Get-AzHDInsightCluster
Show cluster
Use the following command to show details of a specific cluster in the current subscription:
Get-AzHDInsightCluster -ClusterName <Cluster Name>
Delete clusters
Use the following command to delete a cluster:
Remove-AzHDInsightCluster -ClusterName <Cluster Name>
You can also delete a cluster by removing the resource group that contains the cluster. Deleting a resource group deletes all the resources in the group including the default storage account.
Remove-AzResourceGroup -Name <Resource Group Name>
Scale clusters
The cluster scaling feature allows you to change the number of worker nodes used by a cluster that is running in Azure HDInsight without having to re-create the cluster. To change the Hadoop cluster size by using Azure PowerShell, run the following command from a client machine:
Set-AzHDInsightClusterSize -ClusterName <Cluster Name> -TargetInstanceCount <NewSize>
For more information about scaling clusters, see Scale HDInsight clusters.
Update HTTP user credentials
Set-AzHDInsightGatewayCredential sets the gateway HTTP credentials of an Azure HDInsight cluster.
$clusterName = "CLUSTERNAME"
$credential = Get-Credential -Message "Enter the HTTP username and password:" -UserName "admin"
Set-AzHDInsightGatewayCredential -ClusterName $clusterName -HttpCredential $credential
Find the default storage account
The following PowerShell script demonstrates how to get the default storage account name and the related information:
#Connect-AzAccount -Environment AzureChinaCloud
$clusterName = "<HDInsight Cluster Name>"
$clusterInfo = Get-AzHDInsightCluster -ClusterName $clusterName
$storageInfo = $clusterInfo.DefaultStorageAccount.split('.')
$defaultStoreageType = $storageInfo[1]
$defaultStorageName = $storageInfo[0]
echo "Default Storage account name: $defaultStorageName"
echo "Default Storage account type: $defaultStoreageType"
if ($defaultStoreageType -eq "blob")
{
$defaultBlobContainerName = $cluster.DefaultStorageContainer
$defaultStorageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $resourceGroupName -Name $defaultStorageAccountName)[0].Value
$defaultStorageAccountContext = New-AzStorageContext -StorageAccountName $defaultStorageAccountName -StorageAccountKey $defaultStorageAccountKey
echo "Default Blob container name: $defaultBlobContainerName"
echo "Default Storage account key: $defaultStorageAccountKey"
}
Find the resource group
In the Resource Manager mode, each HDInsight cluster belongs to an Azure resource group. To find the resource group:
$clusterName = "<HDInsight Cluster Name>"
$cluster = Get-AzHDInsightCluster -ClusterName $clusterName
$resourceGroupName = $cluster.ResourceGroup
Submit jobs
To submit MapReduce jobs
See Run the MapReduce examples included in HDInsight.
To submit Apache Hive jobs
See Run Apache Hive queries using PowerShell.
To submit Apache Sqoop jobs
See Use Apache Sqoop with HDInsight.
To submit Apache Oozie jobs
See Use Apache Oozie with Apache Hadoop to define and run a workflow in HDInsight.