为 Azure HDInsight 配置虚拟网络服务终结点策略Configure virtual network service endpoint policies for Azure HDInsight

本文介绍如何使用 Azure HDInsight 在虚拟网络上实现服务终结点策略。This article provides information about how to implement service endpoint policies on virtual networks with Azure HDInsight.

背景Background

Azure HDInsight 允许你在自己的虚拟网络中创建群集。Azure HDInsight allows you to create clusters in your own virtual network. 如果需要允许来自虚拟网络的流量传出到其他 Azure 服务(如存储帐户),则可以创建 服务终结点策略If you need to allow outgoing traffic from your virtual network to other Azure services like storage accounts, you can create service endpoint policies. 但是,通过 Azure 门户创建的服务终结点策略仅允许你为单个帐户、订阅中的所有帐户或资源组中的所有帐户创建策略。Service endpoint policies that are created through the Azure portal, however, only allow you to create a policy for a single account, all accounts in a subscription, or all accounts in a resource group.

但作为一项托管服务,Azure HDInsight 会从每个区域中特定存储帐户中的每个群集收集数据和日志文件。As a managed service, however, Azure HDInsight collects data and log files from each cluster in specific storage accounts in each region. 为了使此数据从虚拟网络访问 HDInsight,必须创建可允许流量传出到由 Azure HDInsight 管理的特定数据收集点的服务终结点策略。In order for this data to reach HDInsight from your virtual network, it's necessary for you to create service endpoint policies that allow outgoing traffic to specific data collection points managed by Azure HDInsight.

适用于 HDInsight 的服务终结点策略Service endpoint policies for HDInsight

这些服务终结点策略支持以下功能:These service endpoint policies support the following functionality:

  • 收集有关群集创建、作业执行以及平台操作(例如缩放)的日志和遥测。Collection of logs and telemetry on cluster creation, job execution, and platform operations such as scaling.
  • 将虚拟硬盘 (VHD) 附加到新创建的群集节点,以在群集上预配软件和库。Attaching virtual hard disks (VHDs) to newly created cluster nodes for provisioning software and libraries on your cluster.

如果未创建服务终结点策略来启用此数据流,则群集创建可能会失败,且 Azure HDInsight 将无法为群集提供支持。If service endpoint policies are not created to enable this flow of data, cluster creation may fail and Azure HDInsight will be unable to provide support for your clusters.

为 HDInsight 创建服务终结点策略Create service endpoint policies for HDInsight

在创建新群集之前,请确保已将正确的服务终结点策略附加到虚拟网络。Ensure that the correct service endpoint policies are attached to your virtual network before creating new clusters. 否则,群集创建可能会失败或导致错误。Otherwise the cluster creation may fail or result in error.

使用以下过程创建所需的服务终结点策略:Use the following process to create the necessary service endpoint policies:

  1. 确定要在其中创建 HDInsight 群集的区域。Decide the region where you will be creating your HDInsight cluster.

  2. 可在服务终结点策略资源列表中查找该区域,其中提供了 HDInsight 管理存储帐户的所有资源组。Look up that region in the list of service endpoint policy resources, which gives all of the resource groups for HDInsight management storage accounts.

  3. 选择区域的资源组列表。Select the list of resource groups for your region. 下面为 China East 的资源示例:An example of the resources for China East is shown below:

    "China East":[
        "/subscriptions/235d341f-7fb9-435c-9bdc-034b7306c9b4/resourceGroups/Default-Storage-WestUS",
        "/subscriptions/da0c4c68-9283-4f88-9c35-18f7bd72fbdd/resourceGroups/GenevaWarmPathManageRG",
        "/subscriptions/6a853a41-3423-4167-8d9c-bcf37dc72818/resourceGroups/GenevaWarmPathManageRG",
        "/subscriptions/c8845df8-14d1-4a46-b6dd-e0c44ae400b0/resourceGroups/Default-Storage-CanadaCentral",
        "/subscriptions/c8845df8-14d1-4a46-b6dd-e0c44ae400b0/resourceGroups/cancstorage",
        "/subscriptions/c8845df8-14d1-4a46-b6dd-e0c44ae400b0/resourceGroups/GenevaWarmPathManageRG"
    ],
    
  4. 将这列资源组插入用 Azure CLI 或 Azure PowerShell 编写的安装脚本中。Insert that list of resource groups into the setup script written in Azure CLI or Azure PowerShell.

    $subscriptionId = "<subscription id>"
    $rgName="<resource group name> "
    $location="<location name>"
    $vnetName="<vnet name>"
    $subnetName="<subnet name>"
    $sepName="<service endpoint policy name>"
    $sepDefName="<service endpoint policy definition name>"
    
    # Set to the right subscription ID
    az account set --subscription $subscriptionId
    
    # setup service endpoint on the virtual network subnet
    az network vnet subnet update -g $rgName --vnet-name $vnetName -n $subnetName --service-endpoints Microsoft.Storage
    
    # Create Service Endpoint Policy
    az network service-endpoint policy create -g $rgName  -n $sepName -l $location
    
    # Insert the list of HDInsight owned resources for the region your clusters will be created in.
    [String[]]$resources = @("/subscriptions/235d341f-7fb9-435c-9bdc-034b7306c9b4/resourceGroups/Default-Storage-WestUS",`
    "/subscriptions/da0c4c68-9283-4f88-9c35-18f7bd72fbdd/resourceGroups/GenevaWarmPathManageRG",`
    "/subscriptions/6a853a41-3423-4167-8d9c-bcf37dc72818/resourceGroups/GenevaWarmPathManageRG",`
    "/subscriptions/c8845df8-14d1-4a46-b6dd-e0c44ae400b0/resourceGroups/Default-Storage-CanadaCentral",`
    "/subscriptions/c8845df8-14d1-4a46-b6dd-e0c44ae400b0/resourceGroups/cancstorage",`
    "/subscriptions/c8845df8-14d1-4a46-b6dd-e0c44ae400b0/resourceGroups/GenevaWarmPathManageRG")
    
    #Assign service resources to the SEP policy.
    az network service-endpoint policy-definition create -g $rgName --policy-name $sepName -n $sepDefName --service "Microsoft.Storage" --service-resources $resources
    
    # Associate a subnet to the service endpoint policy just created. If there is a delay in updating it to subnet, you can use the Azure portal to associate the policy with the subnet.
    az network vnet subnet update -g $rgName --vnet-name $vnetName -n $subnetName --service-endpoint-policy $sepName
    

    如果希望使用 PowerShell 设置服务终结点策略,请使用以下代码片段。If you prefer to set up your service endpoint policy using PowerShell, using the following code snippet.

    #Script to assign SEP 
    $subscriptionId = "<subscription id>"
    $rgName = "<resource group name>"
    $vnetName = "<vnet name>"
    $subnetName = "<subnet Name"
    $location = "China East"
    
    # Connect to your Azure Account
    Connect-AzAccount
    
    # Select the Subscription that you want to use
    Select-AzSubscription -SubscriptionId $subscriptionId
    
    # Retrieve VNet Config
    $vnet = Get-AzVirtualNetwork -ResourceGroupName $rgName -Name $vnetName
    
    # Retrieve Subnet Config
    $subnet = Get-AzVirtualNetworkSubnetConfig -Name $subnetName -VirtualNetwork $vnet
    
    # Insert the list of HDInsight owned resources for the region your clusters will be created in.
    [String[]]$resources = @("/subscriptions/235d341f-7fb9-435c-9bdc-034b7306c9b4/resourceGroups/Default-Storage-WestUS",
    "/subscriptions/da0c4c68-9283-4f88-9c35-18f7bd72fbdd/resourceGroups/GenevaWarmPathManageRG",
    "/subscriptions/6a853a41-3423-4167-8d9c-bcf37dc72818/resourceGroups/GenevaWarmPathManageRG",
    "/subscriptions/c8845df8-14d1-4a46-b6dd-e0c44ae400b0/resourceGroups/Default-Storage-CanadaCentral",
    "/subscriptions/c8845df8-14d1-4a46-b6dd-e0c44ae400b0/resourceGroups/cancstorage",
    "/subscriptions/c8845df8-14d1-4a46-b6dd-e0c44ae400b0/resourceGroups/GenevaWarmPathManageRG")
    
    #Declare service endpoint policy definition
    $sepDef = New-AzServiceEndpointPolicyDefinition -Name "SEPHDICanadaCentral" -Description "Service Endpoint Policy Definition" -Service "Microsoft.Storage" -ServiceResource $resources
    
    # Service Endpoint Policy
    $sep= New-AzServiceEndpointPolicy -ResourceGroupName $rgName -Name "SEPHDICanadaCentral" -Location $location -ServiceEndpointPolicyDefinition $sepDef
    
    # Associate a subnet to the service endpoint policy just created. If there is a delay in updating it to subnet, you can use the Azure portal to associate the policy with the subnet.
    Set-AzVirtualNetworkSubnetConfig -Name $subnetName -VirtualNetwork $vnet -AddressPrefix $subnet.AddressPrefix -ServiceEndpointPolicy $sep
    

后续步骤Next steps