为 Azure HDInsight 配置虚拟网络服务终结点策略

本文介绍如何使用 Azure HDInsight 在虚拟网络上实现服务终结点策略。

背景

Azure HDInsight 允许你在自己的虚拟网络中创建群集。 如果需要允许来自虚拟网络的流量传出到其他 Azure 服务（如存储帐户），则可以创建 服务终结点策略。 但是，通过 Azure 门户创建的服务终结点策略仅允许你为单个帐户、订阅中的所有帐户或资源组中的所有帐户创建策略。

但作为一项托管服务，Azure HDInsight 会从每个区域中特定存储帐户中的每个群集收集数据和日志文件。 为了使此数据从虚拟网络访问 HDInsight，必须创建可允许流量传出到由 Azure HDInsight 管理的特定数据收集点的服务终结点策略。

适用于 HDInsight 的服务终结点策略

这些服务终结点策略支持以下功能：

• 收集有关群集创建、作业执行以及平台操作（例如缩放）的日志和遥测。
• 将虚拟硬盘 (VHD) 附加到新创建的群集节点，以在群集上预配软件和库。

如果未创建服务终结点策略来启用此数据流，则群集创建可能会失败，且 Azure HDInsight 将无法为群集提供支持。

为 HDInsight 创建服务终结点策略

在创建新群集之前，请确保已将正确的服务终结点策略附加到虚拟网络。 否则，群集创建可能会失败或导致错误。

使用以下过程创建所需的服务终结点策略：

1. 确定要在其中创建 HDInsight 群集的区域。

2. 可在服务终结点策略资源列表中查找该区域，其中提供了 HDInsight 管理存储帐户的所有资源组。

3. 选择区域的资源组列表。 下面为 China North 的资源示例：

   "China North":[
       "/subscriptions/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/resourceGroups/Default-Storage-ChinaNorth",
       "/subscriptions/bbbb1b1b-cc2c-dd3d-ee4e-ffffff5f5f5f/resourceGroups/GenevaWarmPathManageRG",
       "/subscriptions/cccc2c2c-dd3d-ee4e-ff5f-aaaaaa6a6a6a/resourceGroups/GenevaWarmPathManageRG",
       "/subscriptions/dddd3d3d-ee4e-ff5f-aa6a-bbbbbb7b7b7b/resourceGroups/Default-Storage-CanadaCentral",
       "/subscriptions/dddd3d3d-ee4e-ff5f-aa6a-bbbbbb7b7b7b/resourceGroups/cancstorage",
       "/subscriptions/dddd3d3d-ee4e-ff5f-aa6a-bbbbbb7b7b7b/resourceGroups/GenevaWarmPathManageRG",
       "/subscriptions/eeee4efe-ff5f-aa6a-bb7b-cccccc8c8c8c/resourceGroups/DistroStorageRG/providers/Microsoft.Storage/storageAccounts/hdi31distrorelease",
       "/subscriptions/eeee4efe-ff5f-aa6a-bb7b-cccccc8c8c8c/resourceGroups/DistroStorageRG/providers/Microsoft.Storage/storageAccounts/bigdatadistro"
   ],
   

4. 将这列资源组插入用 Azure CLI 或 Azure PowerShell 编写的安装脚本中。

   $subscriptionId = "<subscription id>"
   $rgName="<resource group name> "
   $location="<location name>"
   $vnetName="<vnet name>"
   $subnetName="<subnet name>"
   $sepName="<service endpoint policy name>"
   $sepDefName="<service endpoint policy definition name>"
   
   # Set to the right subscription ID
   az account set --subscription $subscriptionId
   
   # setup service endpoint on the virtual network subnet
   az network vnet subnet update -g $rgName --vnet-name $vnetName -n $subnetName --service-endpoints Microsoft.Storage
   
   # Create Service Endpoint Policy
   az network service-endpoint policy create -g $rgName  -n $sepName -l $location
   
   # Insert the list of HDInsight owned resources for the region your clusters will be created in.
   # Be sure to get the most recent list of resource groups from the [list of service endpoint policy resources](https://github.com/Azure-Samples/hdinsight-enterprise-security/blob/main/hdinsight-service-endpoint-policy-resources.json)
   [String[]]$resources = @("/subscriptions/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/resourceGroups/Default-Storage-ChinaNorth",`
   "/subscriptions/bbbb1b1b-cc2c-dd3d-ee4e-ffffff5f5f5f/resourceGroups/GenevaWarmPathManageRG",`
   "/subscriptions/cccc2c2c-dd3d-ee4e-ff5f-aaaaaa6a6a6a/resourceGroups/GenevaWarmPathManageRG",`
   "/subscriptions/dddd3d3d-ee4e-ff5f-aa6a-bbbbbb7b7b7b/resourceGroups/Default-Storage-CanadaCentral",`
   "/subscriptions/dddd3d3d-ee4e-ff5f-aa6a-bbbbbb7b7b7b/resourceGroups/cancstorage",`
   "/subscriptions/dddd3d3d-ee4e-ff5f-aa6a-bbbbbb7b7b7b/resourceGroups/GenevaWarmPathManageRG",
   "/subscriptions/eeee4efe-ff5f-aa6a-bb7b-cccccc8c8c8c/resourceGroups/DistroStorageRG/providers/Microsoft.Storage/storageAccounts/hdi31distrorelease",
   "/subscriptions/eeee4efe-ff5f-aa6a-bb7b-cccccc8c8c8c/resourceGroups/DistroStorageRG/providers/Microsoft.Storage/storageAccounts/bigdatadistro")
   
   #Assign service resources to the SEP policy.
   az network service-endpoint policy-definition create -g $rgName --policy-name $sepName -n $sepDefName --service "Microsoft.Storage" --service-resources $resources
   
   # Associate a subnet to the service endpoint policy just created. If there is a delay in updating it to subnet, you can use the Azure portal to associate the policy with the subnet.
   az network vnet subnet update -g $rgName --vnet-name $vnetName -n $subnetName --service-endpoint-policy $sepName
   

   如果希望使用 PowerShell 设置服务终结点策略，请使用以下代码片段。

   #Script to assign SEP 
   $subscriptionId = "<subscription id>"
   $rgName = "<resource group name>"
   $vnetName = "<vnet name>"
   $subnetName = "<subnet Name"
   $location = "China North"
   
   # Connect to your Azure Account
   Connect-AzAccount -Environment AzureChinaCloud
   
   # Select the Subscription that you want to use
   Select-AzSubscription -SubscriptionId $subscriptionId
   
   # Retrieve VNet Config
   $vnet = Get-AzVirtualNetwork -ResourceGroupName $rgName -Name $vnetName
   
   # Retrieve Subnet Config
   $subnet = Get-AzVirtualNetworkSubnetConfig -Name $subnetName -VirtualNetwork $vnet
   
   # Insert the list of HDInsight owned resources for the region your clusters will be created in.
   # Be sure to get the most recent list of resource groups from the [list of service endpoint policy resources](https://github.com/Azure-Samples/hdinsight-enterprise-security/blob/main/hdinsight-service-endpoint-policy-resources.json)
   [String[]]$resources = @("/subscriptions/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/resourceGroups/Default-Storage-ChinaNorth",
   "/subscriptions/bbbb1b1b-cc2c-dd3d-ee4e-ffffff5f5f5f/resourceGroups/GenevaWarmPathManageRG",
   "/subscriptions/cccc2c2c-dd3d-ee4e-ff5f-aaaaaa6a6a6a/resourceGroups/GenevaWarmPathManageRG",
   "/subscriptions/dddd3d3d-ee4e-ff5f-aa6a-bbbbbb7b7b7b/resourceGroups/Default-Storage-CanadaCentral",
   "/subscriptions/dddd3d3d-ee4e-ff5f-aa6a-bbbbbb7b7b7b/resourceGroups/cancstorage",
   "/subscriptions/dddd3d3d-ee4e-ff5f-aa6a-bbbbbb7b7b7b/resourceGroups/GenevaWarmPathManageRG",
   "/subscriptions/eeee4efe-ff5f-aa6a-bb7b-cccccc8c8c8c/resourceGroups/DistroStorageRG/providers/Microsoft.Storage/storageAccounts/hdi31distrorelease",
   "/subscriptions/eeee4efe-ff5f-aa6a-bb7b-cccccc8c8c8c/resourceGroups/DistroStorageRG/providers/Microsoft.Storage/storageAccounts/bigdatadistro")
   
   #Declare service endpoint policy definition
   $sepDef = New-AzServiceEndpointPolicyDefinition -Name "SEPHDICanadaCentral" -Description "Service Endpoint Policy Definition" -Service "Microsoft.Storage" -ServiceResource $resources
   
   # Service Endpoint Policy
   $sep= New-AzServiceEndpointPolicy -ResourceGroupName $rgName -Name "SEPHDICanadaCentral" -Location $location -ServiceEndpointPolicyDefinition $sepDef
   
   # Associate a subnet to the service endpoint policy just created. If there is a delay in updating it to subnet, you can use the Azure portal to associate the policy with the subnet.
   Set-AzVirtualNetworkSubnetConfig -Name $subnetName -VirtualNetwork $vnet -AddressPrefix $subnet.AddressPrefix -ServiceEndpointPolicy $sep
   

后续步骤

• Azure HDInsight 虚拟网络体系结构
• 配置网络虚拟设备