为 Azure HDInsight 群集创建虚拟网络Create virtual networks for Azure HDInsight clusters

本文通过操作示例与代码示例来演示如何创建和配置用于 Azure HDInsight 群集的 Azure 虚拟网络This article provides examples and code samples for creating and configuring Azure Virtual Networks for use with Azure HDInsight clusters. 其中提供了有关创建网络安全组 (NSG) 和配置 DNS 的详细示例。Detailed examples of creating network security groups (NSGs) and configuring DNS are presented.

代码示例和示例的先决条件Prerequisites for code samples and examples

在执行本文中的任何代码示例之前,应了解 TCP/IP 网络。Before executing any of the code samples in this article, ou should have an understanding of TCP/IP networking. 如果你不熟悉 TCP/IP 网络,请在修改生产网络之前咨询相关的人员。If you are not familiar with TCP/IP networking, consult someone who is before making modifications to production networks.

本文中的示例要求满足的其他先决条件包括:Other prerequisites for the samples in this article include the following:

  • 如果使用 PowerShell,则需要安装 AZ 模块If you are using PowerShell, you will need to install the AZ Module.
  • 若要使用 Azure CLI 但尚未安装它,请参阅安装 Azure CLIIf you want to use Azure CLI and have not yet installed it, see Install the Azure CLI.

Important

如果正在查找有关如何使用 Azure 虚拟网络将 HDInsight 连接到本地网络的分步指南,请参阅将 HDInsight 连接到本地网络文档。If you are looking for step by step guidance on connecting HDInsight to your on-premises network using an Azure Virtual Network, see the Connect HDInsight to your on-premises network document.

示例:网络安全组与 HDInsightExample: network security groups with HDInsight

本部分的示例演示如何创建允许 HDInsight 与 Azure 管理服务通信的网络安全组规则。The examples in this section demonstrate how to create network security group rules that allow HDInsight to communicate with the Azure management services. 在使用示例之前,请调整 IP 地址,使之与所用 Azure 区域的 IP 地址匹配。Before using the examples, adjust the IP addresses to match the ones for the Azure region you are using. HDInsight 管理 IP 地址中可找到此信息。You can find this information in HDInsight management IP addresses.

Azure 资源管理模板Azure Resource Management template

以下资源管理模板创建一个虚拟网络,该网络限制入站流量,但允许来自 HDInsight 所需的 IP 地址的流量。The following Resource Management template creates a virtual network that restricts inbound traffic, but allows traffic from the IP addresses required by HDInsight. 该模板还在虚拟网络中创建 HDInsight 群集。This template also creates an HDInsight cluster in the virtual network.

Azure PowerShellAzure PowerShell

使用以下 PowerShell 脚本创建可限制入站流量的虚拟网络,但允许来自中国北部的 IP 地址的流量。Use the following PowerShell script to create a virtual network that restricts inbound traffic and allows traffic from the IP addresses for the China North.

Important

更改此示例中的 hdirule1hdirule2 的 IP 地址,使之与要使用的 Azure 区域匹配。Change the IP addresses for hdirule1 and hdirule2 in this example to match the Azure region you are using. HDInsight 管理 IP 地址中可找到此信息。You can find this information HDInsight management IP addresses.

$vnetName = "Replace with your virtual network name"
$resourceGroupName = "Replace with the resource group the virtual network is in"
$subnetName = "Replace with the name of the subnet that you plan to use for HDInsight"

# Get the Virtual Network object
$vnet = Get-AzVirtualNetwork `
    -Name $vnetName `
    -ResourceGroupName $resourceGroupName

# Get the region the Virtual network is in.
$location = $vnet.Location

# Get the subnet object
$subnet = $vnet.Subnets | Where-Object Name -eq $subnetName

# Create a Network Security Group.
# And add exemptions for the HDInsight health and management services.
$nsg = New-AzNetworkSecurityGroup `
    -Name "hdisecure" `
    -ResourceGroupName $resourceGroupName `
    -Location $location `
    | Add-AzNetworkSecurityRuleConfig `
        -name "hdirule1" `
        -Description "HDI health and management address 52.164.210.96" `
        -Protocol "*" `
        -SourcePortRange "*" `
        -DestinationPortRange "443" `
        -SourceAddressPrefix "52.164.210.96" `
        -DestinationAddressPrefix "VirtualNetwork" `
        -Access Allow `
        -Priority 300 `
        -Direction Inbound `
    | Add-AzNetworkSecurityRuleConfig `
        -Name "hdirule2" `
        -Description "HDI health and management 13.74.153.132" `
        -Protocol "*" `
        -SourcePortRange "*" `
        -DestinationPortRange "443" `
        -SourceAddressPrefix "13.74.153.132" `
        -DestinationAddressPrefix "VirtualNetwork" `
        -Access Allow `
        -Priority 301 `
        -Direction Inbound `
    | Add-AzNetworkSecurityRuleConfig `
        -Name "hdirule3" `
        -Description "HDI health and management 168.61.49.99" `
        -Protocol "*" `
        -SourcePortRange "*" `
        -DestinationPortRange "443" `
        -SourceAddressPrefix "168.61.49.99" `
        -DestinationAddressPrefix "VirtualNetwork" `
        -Access Allow `
        -Priority 302 `
        -Direction Inbound `
    | Add-AzNetworkSecurityRuleConfig `
        -Name "hdirule4" `
        -Description "HDI health and management 23.99.5.239" `
        -Protocol "*" `
        -SourcePortRange "*" `
        -DestinationPortRange "443" `
        -SourceAddressPrefix "23.99.5.239" `
        -DestinationAddressPrefix "VirtualNetwork" `
        -Access Allow `
        -Priority 303 `
        -Direction Inbound `
    | Add-AzNetworkSecurityRuleConfig `
        -Name "hdirule5" `
        -Description "HDI health and management 168.61.48.131" `
        -Protocol "*" `
        -SourcePortRange "*" `
        -DestinationPortRange "443" `
        -SourceAddressPrefix "168.61.48.131" `
        -DestinationAddressPrefix "VirtualNetwork" `
        -Access Allow `
        -Priority 304 `
        -Direction Inbound `
    | Add-AzNetworkSecurityRuleConfig `
        -Name "hdirule6" `
        -Description "HDI health and management 138.91.141.162" `
        -Protocol "*" `
        -SourcePortRange "*" `
        -DestinationPortRange "443" `
        -SourceAddressPrefix "138.91.141.162" `
        -DestinationAddressPrefix "VirtualNetwork" `
        -Access Allow `
        -Priority 305 `
        -Direction Inbound `

# Set the changes to the security group
Set-AzNetworkSecurityGroup -NetworkSecurityGroup $nsg

# Apply the NSG to the subnet
Set-AzVirtualNetworkSubnetConfig `
    -VirtualNetwork $vnet `
    -Name $subnetName `
    -AddressPrefix $subnet.AddressPrefix `
    -NetworkSecurityGroup $nsg
$vnet | Set-AzVirtualNetwork

此示例演示如何添加规则,以便在所需的 IP 地址上允许入站流量。This example demonstrates how to add rules to allow inbound traffic on the required IP addresses. 它不包含限制从其他源进行入站访问的规则。It does not contain a rule to restrict inbound access from other sources. 以下代码演示如何允许来自 Internet 的 SSH 访问:The following code demonstrates how to enable SSH access from the Internet:

Get-AzNetworkSecurityGroup -Name hdisecure -ResourceGroupName RESOURCEGROUP |
Add-AzNetworkSecurityRuleConfig -Name "SSH" -Description "SSH" -Protocol "*" -SourcePortRange "*" -DestinationPortRange "22" -SourceAddressPrefix "*" -DestinationAddressPrefix "VirtualNetwork" -Access Allow -Priority 306 -Direction Inbound

Azure CLIAzure CLI

使用以下步骤创建一个虚拟网络,该网络限制入站流量,但允许来自 HDInsight 所需的 IP 地址的流量。Use the following steps to create a virtual network that restricts inbound traffic, but allows traffic from the IP addresses required by HDInsight.

  1. 使用以下命令创建名为 hdisecure 的新网络安全组。Use the following command to create a new network security group named hdisecure. RESOURCEGROUP 替换为包含 Azure 虚拟网络的资源组。Replace RESOURCEGROUP with the resource group that contains the Azure Virtual Network. LOCATION 替换为组创建在的位置(区域)。Replace LOCATION with the location (region) that the group was created in.

    az network nsg create -g RESOURCEGROUP -n hdisecure -l LOCATION
    

    在创建组后,会收到有关新组的信息。Once the group has been created, you receive information on the new group.

  2. 使用以下命令将规则添加新网络安全组,以允许从 Azure HDInsight 运行状况和管理服务通过端口 443 发起的入站通信。Use the following to add rules to the new network security group that allow inbound communication on port 443 from the Azure HDInsight health and management service. RESOURCEGROUP 替换为包含 Azure 虚拟网络的资源组的名称。Replace RESOURCEGROUP with the name of the resource group that contains the Azure Virtual Network.

    Important

    更改此示例中的 hdirule1hdirule2 的 IP 地址,使之与要使用的 Azure 区域匹配。Change the IP addresses for hdirule1 and hdirule2 in this example to match the Azure region you are using. HDInsight 管理 IP 地址中可找到此信息。You can find this information in HDInsight management IP addresses.

    az network nsg rule create -g RESOURCEGROUP --nsg-name hdisecure -n hdirule1 --protocol "*" --source-port-range "*" --destination-port-range "443" --source-address-prefix "52.164.210.96" --destination-address-prefix "VirtualNetwork" --access "Allow" --priority 300 --direction "Inbound"
    az network nsg rule create -g RESOURCEGROUP --nsg-name hdisecure -n hdirule2 --protocol "*" --source-port-range "*" --destination-port-range "443" --source-address-prefix "13.74.153.132" --destination-address-prefix "VirtualNetwork" --access "Allow" --priority 301 --direction "Inbound"
    az network nsg rule create -g RESOURCEGROUP --nsg-name hdisecure -n hdirule3 --protocol "*" --source-port-range "*" --destination-port-range "443" --source-address-prefix "168.61.49.99" --destination-address-prefix "VirtualNetwork" --access "Allow" --priority 302 --direction "Inbound"
    az network nsg rule create -g RESOURCEGROUP --nsg-name hdisecure -n hdirule4 --protocol "*" --source-port-range "*" --destination-port-range "443" --source-address-prefix "23.99.5.239" --destination-address-prefix "VirtualNetwork" --access "Allow" --priority 303 --direction "Inbound"
    az network nsg rule create -g RESOURCEGROUP --nsg-name hdisecure -n hdirule5 --protocol "*" --source-port-range "*" --destination-port-range "443" --source-address-prefix "168.61.48.131" --destination-address-prefix "VirtualNetwork" --access "Allow" --priority 304 --direction "Inbound"
    az network nsg rule create -g RESOURCEGROUP --nsg-name hdisecure -n hdirule6 --protocol "*" --source-port-range "*" --destination-port-range "443" --source-address-prefix "138.91.141.162" --destination-address-prefix "VirtualNetwork" --access "Allow" --priority 305 --direction "Inbound"
    
  3. 若要检索此网络安全组的唯一标识符,请使用以下命令:To retrieve the unique identifier for this network security group, use the following command:

    az network nsg show -g RESOURCEGROUP -n hdisecure --query "id"
    

    此命令返回类似于以下文本的值:This command returns a value similar to the following text:

     "/subscriptions/SUBSCRIPTIONID/resourceGroups/RESOURCEGROUP/providers/Microsoft.Network/networkSecurityGroups/hdisecure"
    
  4. 使用以下命令将网络安全组应用于子网。Use the following command to apply the network security group to a subnet. GUIDRESOURCEGROUP 值替换为从上一步骤中返回的值。Replace the GUID and RESOURCEGROUP values with the ones returned from the previous step. VNETNAMESUBNETNAME 替换为要创建的虚拟网络名称和子网名称。Replace VNETNAME and SUBNETNAME with the virtual network name and subnet name that you want to create.

    az network vnet subnet update -g RESOURCEGROUP --vnet-name VNETNAME --name SUBNETNAME --set networkSecurityGroup.id="/subscriptions/GUID/resourceGroups/RESOURCEGROUP/providers/Microsoft.Network/networkSecurityGroups/hdisecure"
    

    此命令完成后,即可将 HDInsight 安装到虚拟网络中。Once this command completes, you can install HDInsight into the Virtual Network.

这些步骤只会实现对 Azure 云中 HDInsight 运行状况和管理服务的访问。These steps only open access to the HDInsight health and management service on the Azure cloud. 任何从虚拟网络外部对 HDInsight 群集的其他访问会被阻止。Any other access to the HDInsight cluster from outside the Virtual Network is blocked. 若要从虚拟网络之外启用访问,必须添加其他的虚拟网络安全组规则。To enable access from outside the virtual network, you must add additional Network Security Group rules.

以下代码演示如何允许来自 Internet 的 SSH 访问:The following code demonstrates how to enable SSH access from the Internet:

az network nsg rule create -g RESOURCEGROUP --nsg-name hdisecure -n ssh --protocol "*" --source-port-range "*" --destination-port-range "22" --source-address-prefix "*" --destination-address-prefix "VirtualNetwork" --access "Allow" --priority 306 --direction "Inbound"

示例:DNS 配置Example: DNS configuration

在虚拟网络和连接的本地网络之间进行名称解析Name resolution between a virtual network and a connected on-premises network

此示例作出以下假设:This example makes the following assumptions:

  • 你有一个使用 VPN 网关连接到本地网络的 Azure 虚拟网络。You have an Azure Virtual Network that is connected to an on-premises network using a VPN gateway.

  • 虚拟网络中的自定义 DNS 服务器运行 Linux 或 Unix 作为操作系统。The custom DNS server in the virtual network is running Linux or Unix as the operating system.

  • Bind 安装在自定义 DNS 服务器上。Bind is installed on the custom DNS server.

在虚拟网络的自定义 DNS 服务器上,执行以下操作:On the custom DNS server in the virtual network:

  1. 使用 Azure PowerShell 或 Azure CLI 查找虚拟网络的 DNS 后缀:Use either Azure PowerShell or Azure CLI to find the DNS suffix of the virtual network:

    RESOURCEGROUP 替换为包含虚拟网络的资源组的名称,然后输入命令:Replace RESOURCEGROUP with the name of the resource group that contains the virtual network, and then enter the command:

    $NICs = Get-AzNetworkInterface -ResourceGroupName "RESOURCEGROUP"
    $NICs[0].DnsSettings.InternalDomainNameSuffix
    
    az network nic list --resource-group RESOURCEGROUP --query "[0].dnsSettings.internalDomainNameSuffix"
    
  2. 在虚拟网络的自定义 DNS 服务器上,使用以下文本作为 /etc/bind/named.conf.local 文件的内容:On the custom DNS server for the virtual network, use the following text as the contents of the /etc/bind/named.conf.local file:

    // Forward requests for the virtual network suffix to Azure recursive resolver
    zone "0owcbllr5hze3hxdja3mqlrhhe.ex.internal.chinacloudapp.cn" {
        type forward;
        forwarders {168.63.129.16;}; # Azure recursive resolver
    };
    

    0owcbllr5hze3hxdja3mqlrhhe.ex.internal.chinacloudapp.cn 值替换为虚拟网络的 DNS 后缀。Replace the 0owcbllr5hze3hxdja3mqlrhhe.ex.internal.chinacloudapp.cn value with the DNS suffix of your virtual network.

    此配置将针对虚拟网络 DNS 后缀的所有 DNS 请求路由到 Azure 递归解析程序。This configuration routes all DNS requests for the DNS suffix of the virtual network to the Azure recursive resolver.

  3. 在虚拟网络的自定义 DNS 服务器上,使用以下文本作为 /etc/bind/named.conf.options 文件的内容:On the custom DNS server for the virtual network, use the following text as the contents of the /etc/bind/named.conf.options file:

    // Clients to accept requests from
    // TODO: Add the IP range of the joined network to this list
    acl goodclients {
        10.0.0.0/16; # IP address range of the virtual network
        localhost;
        localnets;
    };
    
    options {
            directory "/var/cache/bind";
    
            recursion yes;
    
            allow-query { goodclients; };
    
            # All other requests are sent to the following
            forwarders {
                192.168.0.1; # Replace with the IP address of your on-premises DNS server
            };
    
            dnssec-validation auto;
    
            auth-nxdomain no;    # conform to RFC1035
            listen-on { any; };
    };
    
    • 10.0.0.0/16 值替换为虚拟网络的 IP 地址范围。Replace the 10.0.0.0/16 value with the IP address range of your virtual network. 此项允许该范围内的名称解析请求地址。This entry allows name resolution requests addresses within this range.

    • 将本地网络的 IP 地址范围添加到 acl goodclients { ... } 部分。Add the IP address range of the on-premises network to the acl goodclients { ... } section. 此项允许本地网络中的资源发出的名称解析请求。entry allows name resolution requests from resources in the on-premises network.

    • 将值 192.168.0.1 替换为本地 DNS 服务器的 IP 地址。Replace the value 192.168.0.1 with the IP address of your on-premises DNS server. 此项将所有其他的 DNS 请求路由到本地 DNS 服务器。This entry routes all other DNS requests to the on-premises DNS server.

  4. 若要使用配置,请重新启动 Bind。To use the configuration, restart Bind. 例如,sudo service bind9 restartFor example, sudo service bind9 restart.

  5. 向本地 DNS 服务器添加条件转发器。Add a conditional forwarder to the on-premises DNS server. 配置条件转发器,将对步骤 1 中 DNS 后者的请求发送到自定义 DNS 服务器。Configure the conditional forwarder to send requests for the DNS suffix from step 1 to the custom DNS server.

    Note

    请参阅 DNS 软件的文档,详细了解如何添加条件转发器。Consult the documentation for your DNS software for specifics on how to add a conditional forwarder.

完成这些步骤后,即可使用完全限定的域名 (FQDN) 连接到任一网络中的资源。After completing these steps, you can connect to resources in either network using fully qualified domain names (FQDN). 现在可以将 HDInsight 安装到虚拟网络。You can now install HDInsight into the virtual network.

在两个连接的虚拟网络之间进行名称解析Name resolution between two connected virtual networks

本示例做出如下假设:This example makes the following assumptions:

  • 你有两个使用 VPN 网关或对等互连进行连接的 Azure 虚拟网络。You have two Azure Virtual Networks that are connected using either a VPN gateway or peering.

  • 两个网络中的自定义 DNS 服务器运行 Linux 或 Unix 作为操作系统。The custom DNS server in both networks is running Linux or Unix as the operating system.

  • Bind 安装在自定义 DNS 服务器上。Bind is installed on the custom DNS servers.

  1. 使用 Azure PowerShell 或 Azure CLI 查找两个虚拟网络的 DNS 后缀:Use either Azure PowerShell or Azure CLI to find the DNS suffix of both virtual networks:

    RESOURCEGROUP 替换为包含虚拟网络的资源组的名称,然后输入命令:Replace RESOURCEGROUP with the name of the resource group that contains the virtual network, and then enter the command:

    $NICs = Get-AzNetworkInterface -ResourceGroupName "RESOURCEGROUP"
    $NICs[0].DnsSettings.InternalDomainNameSuffix
    
    az network nic list --resource-group RESOURCEGROUP --query "[0].dnsSettings.internalDomainNameSuffix"
    
  2. 在自定义 DNS 服务器上,使用以下文本作为 /etc/bind/named.config.local 文件的内容。Use the following text as the contents of the /etc/bind/named.config.local file on the custom DNS server. 在两个虚拟网络中对自定义 DNS 服务器进行这种更改。Make this change on the custom DNS server in both virtual networks.

    // Forward requests for the virtual network suffix to Azure recursive resolver
    zone "0owcbllr5hze3hxdja3mqlrhhe.ex.internal.chinacloudapp.cn" {
        type forward;
        forwarders {10.0.0.4;}; # The IP address of the DNS server in the other virtual network
    };
    

    0owcbllr5hze3hxdja3mqlrhhe.ex.internal.chinacloudapp.cn 值替换为另一虚拟网络的 DNS 后缀。 Replace the 0owcbllr5hze3hxdja3mqlrhhe.ex.internal.chinacloudapp.cn value with the DNS suffix of the other virtual network. 此项将对远程网络 DNS 后缀的请求路由到该网络中的自定义 DNS。This entry routes requests for the DNS suffix of the remote network to the custom DNS in that network.

  3. 在两个虚拟网络的自定义 DNS 服务器上,使用以下文本作为 /etc/bind/named.conf.options 文件的内容:On the custom DNS servers in both virtual networks, use the following text as the contents of the /etc/bind/named.conf.options file:

    // Clients to accept requests from
    acl goodclients {
        10.1.0.0/16; # The IP address range of one virtual network
        10.0.0.0/16; # The IP address range of the other virtual network
        localhost;
        localnets;
    };
    
    options {
            directory "/var/cache/bind";
    
            recursion yes;
    
            allow-query { goodclients; };
    
            forwarders {
            168.63.129.16;   # Azure recursive resolver         
            };
    
            dnssec-validation auto;
    
            auth-nxdomain no;    # conform to RFC1035
            listen-on { any; };
    };
    

    10.0.0.0/1610.1.0.0/16 值替换为虚拟网络的 IP 地址范围。Replace the 10.0.0.0/16 and 10.1.0.0/16 values with the IP address ranges of your virtual networks. 此项允许每个网络中的资源发出 DNS 服务器的请求。This entry allows resources in each network to make requests of the DNS servers.

    任何不是针对虚拟网络 DNS 后缀的请求(例如,microsoft.com)均由 Azure 递归解析程序处理。Any requests that are not for the DNS suffixes of the virtual networks (for example, microsoft.com) is handled by the Azure recursive resolver.

  4. 若要使用此配置,请重启 Bind。To use the configuration, restart Bind. 例如,两个 DNS 服务器上的 sudo service bind9 restartFor example, sudo service bind9 restart on both DNS servers.

完成这些步骤后,即可使用完全限定的域名 (FQDN) 连接到虚拟网络中的资源。After completing these steps, you can connect to resources in the virtual network using fully qualified domain names (FQDN). 现在可以将 HDInsight 安装到虚拟网络。You can now install HDInsight into the virtual network.

后续步骤Next steps