为 Azure HDInsight 配置虚拟网络服务终结点策略

  • 项目

本文介绍如何使用 Azure HDInsight 在虚拟网络上实现服务终结点策略。

背景

Azure HDInsight 允许你在自己的虚拟网络中创建群集。 如果需要允许来自虚拟网络的流量传出到其他 Azure 服务（如存储帐户），则可以创建 服务终结点策略。 但是，通过 Azure 门户创建的服务终结点策略仅允许你为单个帐户、订阅中的所有帐户或资源组中的所有帐户创建策略。

但作为一项托管服务，Azure HDInsight 会从每个区域中特定存储帐户中的每个群集收集数据和日志文件。 为了使此数据从虚拟网络访问 HDInsight，必须创建可允许流量传出到由 Azure HDInsight 管理的特定数据收集点的服务终结点策略。

适用于 HDInsight 的服务终结点策略

这些服务终结点策略支持以下功能：

  • 收集有关群集创建、作业执行以及平台操作（例如缩放）的日志和遥测。
  • 将虚拟硬盘 (VHD) 附加到新创建的群集节点，以在群集上预配软件和库。

如果未创建服务终结点策略来启用此数据流，则群集创建可能会失败，且 Azure HDInsight 将无法为群集提供支持。

为 HDInsight 创建服务终结点策略

在创建新群集之前，请确保已将正确的服务终结点策略附加到虚拟网络。 否则，群集创建可能会失败或导致错误。

使用以下过程创建所需的服务终结点策略：

  1. 确定要在其中创建 HDInsight 群集的区域。

  2. 可在服务终结点策略资源列表中查找该区域，其中提供了 HDInsight 管理存储帐户的所有资源组。

  3. 选择区域的资源组列表。 下面为 China East 的资源示例：

    "China East":[
    "/subscriptions/235d341f-7fb9-435c-9bdc-034b7306c9b4/resourceGroups/Default-Storage-chinanorth",
    "/subscriptions/da0c4c68-9283-4f88-9c35-18f7bd72fbdd/resourceGroups/GenevaWarmPathManageRG",
    "/subscriptions/6a853a41-3423-4167-8d9c-bcf37dc72818/resourceGroups/GenevaWarmPathManageRG",
    "/subscriptions/c8845df8-14d1-4a46-b6dd-e0c44ae400b0/resourceGroups/Default-Storage-CanadaCentral",
    "/subscriptions/c8845df8-14d1-4a46-b6dd-e0c44ae400b0/resourceGroups/cancstorage",
    "/subscriptions/c8845df8-14d1-4a46-b6dd-e0c44ae400b0/resourceGroups/GenevaWarmPathManageRG",
    "/subscriptions/fb3429ab-83d0-4bed-95e9-1a8e9455252c/resourceGroups/DistroStorageRG/providers/Microsoft.Storage/storageAccounts/hdi31distrorelease",
    "/subscriptions/fb3429ab-83d0-4bed-95e9-1a8e9455252c/resourceGroups/DistroStorageRG/providers/Microsoft.Storage/storageAccounts/bigdatadistro"
],

  4. 将这列资源组插入用 Azure CLI 或 Azure PowerShell 编写的安装脚本中。

    $subscriptionId = "<subscription id>"
$rgName="<resource group name> "
$location="<location name>"
$vnetName="<vnet name>"
$subnetName="<subnet name>"
$sepName="<service endpoint policy name>"
$sepDefName="<service endpoint policy definition name>"

# Set to the right subscription ID
az account set --subscription $subscriptionId

# setup service endpoint on the virtual network subnet
az network vnet subnet update -g $rgName --vnet-name $vnetName -n $subnetName --service-endpoints Microsoft.Storage

# Create Service Endpoint Policy
az network service-endpoint policy create -g $rgName  -n $sepName -l $location

# Insert the list of HDInsight owned resources for the region your clusters will be created in.
# Be sure to get the most recent list of resource groups from the [list of service endpoint policy resources](https://github.com/Azure-Samples/hdinsight-enterprise-security/blob/main/hdinsight-service-endpoint-policy-resources.json)
[String[]]$resources = @("/subscriptions/235d341f-7fb9-435c-9bdc-034b7306c9b4/resourceGroups/Default-Storage-chinanorth",`
"/subscriptions/da0c4c68-9283-4f88-9c35-18f7bd72fbdd/resourceGroups/GenevaWarmPathManageRG",`
"/subscriptions/6a853a41-3423-4167-8d9c-bcf37dc72818/resourceGroups/GenevaWarmPathManageRG",`
"/subscriptions/c8845df8-14d1-4a46-b6dd-e0c44ae400b0/resourceGroups/Default-Storage-CanadaCentral",`
"/subscriptions/c8845df8-14d1-4a46-b6dd-e0c44ae400b0/resourceGroups/cancstorage",`
"/subscriptions/c8845df8-14d1-4a46-b6dd-e0c44ae400b0/resourceGroups/GenevaWarmPathManageRG",
"/subscriptions/fb3429ab-83d0-4bed-95e9-1a8e9455252c/resourceGroups/DistroStorageRG/providers/Microsoft.Storage/storageAccounts/hdi31distrorelease",
"/subscriptions/fb3429ab-83d0-4bed-95e9-1a8e9455252c/resourceGroups/DistroStorageRG/providers/Microsoft.Storage/storageAccounts/bigdatadistro")

#Assign service resources to the SEP policy.
az network service-endpoint policy-definition create -g $rgName --policy-name $sepName -n $sepDefName --service "Microsoft.Storage" --service-resources $resources

# Associate a subnet to the service endpoint policy just created. If there is a delay in updating it to subnet, you can use the Azure portal to associate the policy with the subnet.
az network vnet subnet update -g $rgName --vnet-name $vnetName -n $subnetName --service-endpoint-policy $sepName

    如果希望使用 PowerShell 设置服务终结点策略，请使用以下代码片段。

    #Script to assign SEP 
$subscriptionId = "<subscription id>"
$rgName = "<resource group name>"
$vnetName = "<vnet name>"
$subnetName = "<subnet Name"
$location = "China East"

# Connect to your Azure Account
Connect-AzAccount -Environment AzureChinaCloud

# Select the Subscription that you want to use
Select-AzSubscription -SubscriptionId $subscriptionId

# Retrieve VNet Config
$vnet = Get-AzVirtualNetwork -ResourceGroupName $rgName -Name $vnetName

# Retrieve Subnet Config
$subnet = Get-AzVirtualNetworkSubnetConfig -Name $subnetName -VirtualNetwork $vnet

# Insert the list of HDInsight owned resources for the region your clusters will be created in.
# Be sure to get the most recent list of resource groups from the [list of service endpoint policy resources](https://github.com/Azure-Samples/hdinsight-enterprise-security/blob/main/hdinsight-service-endpoint-policy-resources.json)
[String[]]$resources = @("/subscriptions/235d341f-7fb9-435c-9bdc-034b7306c9b4/resourceGroups/Default-Storage-chinanorth",
"/subscriptions/da0c4c68-9283-4f88-9c35-18f7bd72fbdd/resourceGroups/GenevaWarmPathManageRG",
"/subscriptions/6a853a41-3423-4167-8d9c-bcf37dc72818/resourceGroups/GenevaWarmPathManageRG",
"/subscriptions/c8845df8-14d1-4a46-b6dd-e0c44ae400b0/resourceGroups/Default-Storage-CanadaCentral",
"/subscriptions/c8845df8-14d1-4a46-b6dd-e0c44ae400b0/resourceGroups/cancstorage",
"/subscriptions/c8845df8-14d1-4a46-b6dd-e0c44ae400b0/resourceGroups/GenevaWarmPathManageRG",
"/subscriptions/fb3429ab-83d0-4bed-95e9-1a8e9455252c/resourceGroups/DistroStorageRG/providers/Microsoft.Storage/storageAccounts/hdi31distrorelease",
"/subscriptions/fb3429ab-83d0-4bed-95e9-1a8e9455252c/resourceGroups/DistroStorageRG/providers/Microsoft.Storage/storageAccounts/bigdatadistro")

#Declare service endpoint policy definition
$sepDef = New-AzServiceEndpointPolicyDefinition -Name "SEPHDICanadaCentral" -Description "Service Endpoint Policy Definition" -Service "Microsoft.Storage" -ServiceResource $resources

# Service Endpoint Policy
$sep= New-AzServiceEndpointPolicy -ResourceGroupName $rgName -Name "SEPHDICanadaCentral" -Location $location -ServiceEndpointPolicyDefinition $sepDef

# Associate a subnet to the service endpoint policy just created. If there is a delay in updating it to subnet, you can use the Azure portal to associate the policy with the subnet.
Set-AzVirtualNetworkSubnetConfig -Name $subnetName -VirtualNetwork $vnet -AddressPrefix $subnet.AddressPrefix -ServiceEndpointPolicy $sep

重要

建议你定期手动或通过自动化获取最新的服务终结点策略资源列表。 这将防止在 JSON 文件中添加或删除其他资源组时出现 CRUD 问题。

后续步骤