使用 Azure PowerShell 管理 HDInsight 中的 Apache Hadoop 群集Manage Apache Hadoop clusters in HDInsight by using Azure PowerShell

Azure PowerShell 可用于在 Azure 中控制和自动执行工作负荷的部署和管理。Azure PowerShell can be used to control and automate the deployment and management of your workloads in Azure. 本文介绍了如何使用 Azure PowerShell Az 模块管理 Azure HDInsight 中的 Apache Hadoop 群集。In this article, you learn how to manage Apache Hadoop clusters in Azure HDInsight by using the Azure PowerShell Az module. 有关 HDInsight PowerShell cmdlet 的列表,请查看 Az.HDInsight 参考For the list of the HDInsight PowerShell cmdlets, see the Az.HDInsight reference.

如果没有 Azure 订阅,请在开始前创建一个试用帐户If you don�t have an Azure subscription, create a trial account before you begin.

先决条件Prerequisites

备注

本文进行了更新,以便使用新的 Azure PowerShell Az 模块。This article has been updated to use the new Azure PowerShell Az module. 你仍然可以使用 AzureRM 模块,至少在 2020 年 12 月之前,它将继续接收 bug 修补程序。You can still use the AzureRM module, which will continue to receive bug fixes until at least December 2020. 若要详细了解新的 Az 模块和 AzureRM 兼容性,请参阅新 Azure Powershell Az 模块简介To learn more about the new Az module and AzureRM compatibility, see Introducing the new Azure PowerShell Az module. 有关 Az 模块安装说明,请参阅安装 Azure PowerShellFor Az module installation instructions, see Install Azure PowerShell.

创建群集Create clusters

请参阅使用 Azure PowerShell 在 HDInsight 中创建基于 Linux 的群集See Create Linux-based clusters in HDInsight using Azure PowerShell

列出群集List clusters

使用以下命令可列出当前订阅中的所有群集:Use the following command to list all clusters in the current subscription:

Get-AzHDInsightCluster

显示群集Show cluster

使用以下命令可显示当前订阅中特定群集的详细信息:Use the following command to show details of a specific cluster in the current subscription:

Get-AzHDInsightCluster -ClusterName <Cluster Name>

删除群集Delete clusters

使用以下命令来删除群集:Use the following command to delete a cluster:

Remove-AzHDInsightCluster -ClusterName <Cluster Name>

还可以通过删除包含群集的资源组来删除群集。You can also delete a cluster by removing the resource group that contains the cluster. 删除资源群将删除组中的所有资源(包括默认存储帐户)。Deleting a resource group deletes all the resources in the group including the default storage account.

Remove-AzResourceGroup -Name <Resource Group Name>

缩放群集Scale clusters

使用群集缩放功能,可更改 Azure HDInsight 中运行的群集使用的辅助节点数,而无需重新创建群集。The cluster scaling feature allows you to change the number of worker nodes used by a cluster that is running in Azure HDInsight without having to re-create the cluster. 若要使用 Azure PowerShell 更改 Hadoop 群集大小,请从客户端计算机运行以下命令:To change the Hadoop cluster size by using Azure PowerShell, run the following command from a client machine:

Set-AzHDInsightClusterSize -ClusterName <Cluster Name> -TargetInstanceCount <NewSize>

有关缩放群集的详细信息,请参阅缩放 HDInsight 群集For more information about scaling clusters, see Scale HDInsight clusters.

更新 HTTP 用户凭据Update HTTP user credentials

Set-AzHDInsightGatewayCredential 设置 Azure HDInsight 群集的网关 HTTP 凭据。Set-AzHDInsightGatewayCredential sets the gateway HTTP credentials of an Azure HDInsight cluster.

$clusterName = "CLUSTERNAME"
$credential = Get-Credential -Message "Enter the HTTP username and password:" -UserName "admin"

Set-AzHDInsightGatewayCredential -ClusterName $clusterName -HttpCredential $credential

查找默认存储帐户Find the default storage account

以下 PowerShell 脚本演示了如何获取群集的默认存储帐户名称和相关信息:The following PowerShell script demonstrates how to get the default storage account name and the related information:

#Connect-AzAccount
$clusterName = "<HDInsight Cluster Name>"

$clusterInfo = Get-AzHDInsightCluster -ClusterName $clusterName
$storageInfo = $clusterInfo.DefaultStorageAccount.split('.')
$defaultStoreageType = $storageInfo[1]
$defaultStorageName = $storageInfo[0]

echo "Default Storage account name: $defaultStorageName"
echo "Default Storage account type: $defaultStoreageType"

if ($defaultStoreageType -eq "blob")
{
    $defaultBlobContainerName = $cluster.DefaultStorageContainer
    $defaultStorageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $resourceGroupName -Name $defaultStorageAccountName)[0].Value
    $defaultStorageAccountContext = New-AzStorageContext -StorageAccountName $defaultStorageAccountName -StorageAccountKey $defaultStorageAccountKey

    echo "Default Blob container name: $defaultBlobContainerName"
    echo "Default Storage account key: $defaultStorageAccountKey"
}

查找资源组Find the resource group

在 Resource Manager 模式下,每个 HDInsight 群集都属于一个 Azure 资源组。In the Resource Manager mode, each HDInsight cluster belongs to an Azure resource group. 若要查找资源组:To find the resource group:

$clusterName = "<HDInsight Cluster Name>"

$cluster = Get-AzHDInsightCluster -ClusterName $clusterName
$resourceGroupName = $cluster.ResourceGroup

提交作业Submit jobs

提交 MapReduce 作业To submit MapReduce jobs

请参阅运行 HDInsight 随附的 MapReduce 示例See Run the MapReduce examples included in HDInsight.

提交 Apache Hive 作业To submit Apache Hive jobs

请参阅使用 PowerShell 运行 Apache Hive 查询See Run Apache Hive queries using PowerShell.

提交 Apache Sqoop 作业To submit Apache Sqoop jobs

请参阅将 Apache Sqoop 与 HDInsight 配合使用See Use Apache Sqoop with HDInsight.

提交 Apache Oozie 作业To submit Apache Oozie jobs

请参阅在 HDInsight 中将 Apache Oozie 与 Apache Hadoop 配合使用以定义和运行工作流See Use Apache Oozie with Apache Hadoop to define and run a workflow in HDInsight.

将数据上传到 Azure Blob 存储Upload data to Azure Blob storage

请参阅将数据上传到 HDInsightSee Upload data to HDInsight.

另请参阅See Also