创建专用 Azure Kubernetes 服务群集Create a private Azure Kubernetes Service cluster

在专用群集中,控制平面或 API 服务器具有内部 IP 地址,这些地址在 RFC1918 - 专用 Internet 的地址分配文档中定义。In a private cluster, the control plane or API server has internal IP addresses that are defined in the RFC1918 - Address Allocation for Private Internet document. 通过使用专用群集,可以确保 API 服务器与节点池之间的网络流量仅保留在专用网络上。By using a private cluster, you can ensure network traffic between your API server and your node pools remains on the private network only.

控制平面或 API 服务器位于 Azure Kubernetes 服务 (AKS) 托管的 Azure 订阅中。The control plane or API server is in an Azure Kubernetes Service (AKS)-managed Azure subscription. 客户的群集或节点池在客户的订阅中。A customer's cluster or node pool is in the customer's subscription. 服务器与群集或节点池可以通过 API 服务器虚拟网络中的 Azure 专用链接服务以及在客户 AKS 群集的子网中公开的专用终结点相互通信。The server and the cluster or node pool can communicate with each other through the Azure Private Link service in the API server virtual network and a private endpoint that's exposed in the subnet of the customer's AKS cluster.

上市区域Region availability

专用群集在支持 AKS 的 Azure 中国世纪互联区域提供。Private cluster is available in Azure China 21Vianet regions where AKS is supported.

先决条件Prerequisites

  • Azure CLI 版本 2.2.0 或更高版本The Azure CLI version 2.2.0 or later
  • 仅标准 Azure 负载均衡器支持专用链接服务。The Private Link service is supported on Standard Azure Load Balancer only. 不支持基本 Azure 负载均衡器。Basic Azure Load Balancer isn't supported.
  • 若要使用自定义 DNS 服务器,请在自定义 DNS 服务器中将 Azure DNS IP 168.63.129.16 作为上游 DNS 服务器进行添加。To use a custom DNS server, add the Azure DNS IP 168.63.129.16 as the upstream DNS server in the custom DNS server.

创建专用 AKS 群集Create a private AKS cluster

创建资源组Create a resource group

为 AKS 群集创建资源组或使用现有资源组。Create a resource group or use an existing resource group for your AKS cluster.

az group create -l chinaeast2 -n MyResourceGroup

默认基本网络Default basic networking

az aks create -n <private-cluster-name> -g <private-cluster-resource-group> --load-balancer-sku standard --enable-private-cluster  

其中的 --enable-private-cluster 是专用群集的必需标志。Where --enable-private-cluster is a mandatory flag for a private cluster.

高级网络Advanced networking

az aks create \
    --resource-group <private-cluster-resource-group> \
    --name <private-cluster-name> \
    --load-balancer-sku standard \
    --enable-private-cluster \
    --network-plugin azure \
    --vnet-subnet-id <subnet-id> \
    --docker-bridge-address 172.17.0.1/16 \
    --dns-service-ip 10.2.0.10 \
    --service-cidr 10.2.0.0/24 

其中的 --enable-private-cluster 是专用群集的必需标志。Where --enable-private-cluster is a mandatory flag for a private cluster.

备注

如果 Docker 桥地址 CIDR (172.17.0.1/16) 与子网 CIDR 冲突,请相应地更改 Docker 桥地址。If the Docker bridge address CIDR (172.17.0.1/16) clashes with the subnet CIDR, change the Docker bridge address appropriately.

配置专用 DNS 区域Configure Private DNS Zone

如果省略 --private-dns-zone 参数,则默认值为“system”。The default value is "system", if the --private-dns-zone argument is omitted. AKS 将在节点资源组中创建专用 DNS 区域。AKS will create a Private DNS Zone in the Node Resource Group. 传递“none”参数意味着 AKS 将不会创建专用 DNS 区域。Passing the "none" parameter means AKS will not create a Private DNS Zone. 此操作依赖于自带 DNS 服务器以及专用 FQDN 的 DNS 解析的配置。This relies on Bring Your Own DNS Server and configuration of the DNS resolution for the Private FQDN. 如果未配置 DNS 解析,则 DNS 只能在代理节点内进行解析,并且会导致群集在部署之后出现问题。If you don't configure DNS resolution, DNS is only resolvable within the agent nodes and will cause cluster issues after deployment.

“无专用 DNS 区域”先决条件No Private DNS Zone Prerequisites

无 PrivateDNSZoneNo PrivateDNSZone

  • Azure CLI 0.4.67 或更高版本The Azure CLI version 0.4.67 or later
  • API 2020-11-01 或更高版本The api version 2020-11-01 or later

创建具有专用 DNS 区域的专用 AKS 群集Create a private AKS cluster with Private DNS Zone

az aks create -n <private-cluster-name> -g <private-cluster-resource-group> --load-balancer-sku standard --enable-private-cluster --private-dns-zone [none|system]

连接到专用群集的选项Options for connecting to the private cluster

API 服务器终结点没有公共 IP 地址。The API server endpoint has no public IP address. 若要管理 API 服务器,需要使用有权访问 AKS 群集的 Azure 虚拟网络 (VNet) 的 VM。To manage the API server, you'll need to use a VM that has access to the AKS cluster's Azure Virtual Network (VNet). 有几种选项可用于建立与专用群集的网络连接。There are several options for establishing network connectivity to the private cluster.

  • 在与 AKS 群集相同的 Azure 虚拟网络 (VNet) 中创建 VM。Create a VM in the same Azure Virtual Network (VNet) as the AKS cluster.
  • 在单独的网络中使用 VM 并设置虚拟网络对等互联Use a VM in a separate network and set up Virtual network peering. 有关此选项的详细信息,请参阅以下部分。See the section below for more information on this option.
  • 使用快速路由或 VPN 连接。Use an Express Route or VPN connection.

在与 AKS 群集相同的 VNET 中创建 VM 是最简单的选项。Creating a VM in the same VNET as the AKS cluster is the easiest option. 快速路由和 VPN 会增加成本,且要求额外的网络复杂性。Express Route and VPNs add costs and require additional networking complexity. 虚拟网络对等互联要求计划网络 CIDR 范围,以确保不存在重叠范围。Virtual network peering requires you to plan your network CIDR ranges to ensure there are no overlapping ranges.

虚拟网络对等互连Virtual network peering

如前所述,虚拟网络对等互连是一种访问专用群集的方法。As mentioned, virtual network peering is one way to access your private cluster. 若要使用虚拟网络对等互连,需要在虚拟网络与专用 DNS 区域之间设置链接。To use virtual network peering, you need to set up a link between virtual network and the private DNS zone.

  1. 转到 Azure 门户中的节点资源组。Go to the node resource group in the Azure portal.
  2. 选择专用 DNS 区域。Select the private DNS zone.
  3. 在左窗格中,选择“虚拟网络”链接。In the left pane, select the Virtual network link.
  4. 创建新链接,将 VM 的虚拟网络添加到专用 DNS 区域。Create a new link to add the virtual network of the VM to the private DNS zone. DNS 区域链接需要几分钟时间才能变为可用。It takes a few minutes for the DNS zone link to become available.
  5. 在 Azure 门户中,导航到包含群集虚拟网络的资源组。In the Azure portal, navigate to the resource group that contains your cluster's virtual network.
  6. 在右窗格中,选择“虚拟网络”。In the right pane, select the virtual network. 虚拟网络名称的格式为 aks-vnet-*。The virtual network name is in the form aks-vnet-*.
  7. 在左窗格中,选择“对等互连”。In the left pane, select Peerings.
  8. 选择“添加”,添加 VM 的虚拟网络,然后创建对等互连。Select Add, add the virtual network of the VM, and then create the peering.
  9. 转到你在其中具有 VM 的虚拟网络,选择“对等互连”、“AKS 虚拟网络”,然后创建对等互连。Go to the virtual network where you have the VM, select Peerings, select the AKS virtual network, and then create the peering. 如果 AKS 虚拟网络上的地址范围与 VM 的虚拟网络冲突,则对等互连将失败。If the address ranges on the AKS virtual network and the VM's virtual network clash, peering fails. 有关详细信息,请参阅虚拟网络对等互联For more information, see Virtual network peering.

具有自定义 DNS 的中心和分支Hub and spoke with custom DNS

中心和分支体系结构通常用于在 Azure 中部署网络。Hub and spoke architectures are commonly used to deploy networks in Azure. 在许多此类部署中,会将分支 VNet 中的 DNS 设置配置为引用中心 DNS 转发器,以允许本地和基于 Azure 的 DNS 解析。In many of these deployments, DNS settings in the spoke VNets are configured to reference a central DNS forwarder to allow for on-premises and Azure-based DNS resolution. 将 AKS 群集部署到此类网络环境中时,必须考虑一些特殊注意事项。When deploying an AKS cluster into such a networking environment, there are some special considerations that must be taken into account.

专用群集中心和分支

  1. 默认情况下,预配专用群集后,会在群集托管资源组中创建专用终结点 (1) 和专用 DNS 区域 (2)。By default, when a private cluster is provisioned, a private endpoint (1) and a private DNS zone (2) are created in the cluster-managed resource group. 群集使用专用区域中的 A 记录来解析专用终结点的 IP,以便与 API 服务器通信。The cluster uses an A record in the private zone to resolve the IP of the private endpoint for communication to the API server.

  2. 专用 DNS 区域仅链接到群集节点附加到的 VNet (3)。The private DNS zone is linked only to the VNet that the cluster nodes are attached to (3). 这意味着专用终结点只能由该链接 VNet 中的主机进行解析。This means that the private endpoint can only be resolved by hosts in that linked VNet. 在 VNet 上不配置任何自定义 DNS(默认设置)的情况下,这可以正常工作,因为主机指向用于 DNS 的 168.63.129.16,因此可以解析专用 DNS 区域中的记录(由于存在链接)。In scenarios where no custom DNS is configured on the VNet (default), this works without issue as hosts point at 168.63.129.16 for DNS that can resolve records in the private DNS zone because of the link.

  3. 在包含群集的 VNet 具有自定义 DNS 设置 (4) 的情况下,除非将专用 DNS 区域链接到包含自定义 DNS 解析程序的 VNet (5),否则群集部署将失败。In scenarios where the VNet containing your cluster has custom DNS settings (4), cluster deployment fails unless the private DNS zone is linked to the VNet that contains the custom DNS resolvers (5). 可以在群集预配期间创建专用区域后手动创建此链接,也可以使用基于事件的部署机制(例如,Azure 事件网格和 Azure Functions)在检测到区域已创建后通过自动化来创建此链接。This link can be created manually after the private zone is created during cluster provisioning or via automation upon detection of creation of the zone using event-based deployment mechanisms (for example, Azure Event Grid and Azure Functions).

备注

如果你将自带路由表与 kubenet 配合使用,并且将自带 DNS 与专用群集配合使用,群集创建将会失败。If you are using Bring Your Own Route Table with kubenet and Bring Your Own DNS with Private Cluster, the cluster creation will fail. 你需要在创建群集失败之后将节点资源组中的 RouteTable 关联到子网,以使创建能够成功。You will need to associate the RouteTable in the node resource group to the subnet after the cluster creation failed, in order to make the creation successful.

限制Limitations

  • IP 授权范围不能应用于专用 API 服务器终结点,它们仅适用于公共 API 服务器IP authorized ranges can't be applied to the private api server endpoint, they only apply to the public API server

  • 不支持具有专用群集的 Azure DevOps Microsoft 托管的代理。No support for Azure DevOps Microsoft-hosted Agents with private clusters.

  • 对于需要使 Azure 容器注册表能够与专用 AKS 配合使用的客户,容器注册表虚拟网络必须与代理群集虚拟网络对等互连。For customers that need to enable Azure Container Registry to work with private AKS, the Container Registry virtual network must be peered with the agent cluster virtual network.

  • 不支持将现有 AKS 群集转换为专用群集No support for converting existing AKS clusters into private clusters

  • 删除或修改客户子网中的专用终结点将导致群集停止运行。Deleting or modifying the private endpoint in the customer subnet will cause the cluster to stop functioning.

  • 当前不支持适用于容器的 Azure Monitor 实时数据。Azure Monitor for containers Live Data isn't currently supported.

  • 客户在自己的 DNS 服务器上更新 A 记录后,这些 Pod 仍会在迁移后将 apiserver FQDN 解析到较旧的 IP,直到重启这些 Pod。After customers have updated the A record on their own DNS servers, those Pods would still resolve apiserver FQDN to the older IP after migration until they're restarted. 客户需要在控制平面迁移之后重启 hostNetwork Pod 和 default-DNSPolicy Pod。Customers need to restart hostNetwork Pods and default-DNSPolicy Pods after control plane migration.

  • 如果对控制平面进行维护,AKS IP 可能会更改。In the case of maintenance on the control plane, your AKS IP might change. 在这种情况下,你必须在自定义 DNS 服务器上更新指向 API 服务器专用 IP 的 A 记录,并重启使用 hostNetwork 的任何自定义 Pod 或部署。In this case you must update the A record pointing to the API server private IP on your custom DNS server and restart any custom pods or deployments using hostNetwork.