使用用户定义的路由自定义群集出口(预览版)Customize cluster egress with a User-Defined Route (Preview)

可以根据具体的方案自定义 AKS 群集的出口。Egress from an AKS cluster can be customized to fit specific scenarios. 默认情况下,AKS 将预配一个可设置并用于出口的标准 SKU 负载均衡器。By default, AKS will provision a Standard SKU Load Balancer to be setup and used for egress. 但是,如果禁用了公共 IP 或者出口需要额外的跃点,则默认设置可能不能满足所有方案的要求。However, the default setup may not meet the requirements of all scenarios if public IPs are disallowed or additional hops are required for egress.

本文介绍了如何自定义群集的出口路由以支持自定义网络方案,例如,禁用公共 IP 并要求群集位于网络虚拟设备 (NVA) 后面。This article walks through how to customize a cluster's egress route to support custom network scenarios, such as those which disallows public IPs and requires the cluster to sit behind a network virtual appliance (NVA).

Important

AKS 预览版功能是自助服务,根据用户的选择提供。AKS preview features are self-service and are offered on an opt-in basis. 预览版按原样提供,并且仅在可用情况下提供,不享受服务级别协议 (SLA) 和有限担保。 Previews are provided as is and as available and are excluded from the service-level agreement (SLA) and limited warranty. AKS 预览版由客户支持尽力提供部分支持。AKS previews are partially covered by customer support on a best effort basis. 因此,这些功能不适用于生产用途。Therefore, the features aren't meant for production use. 有关详细信息,请参阅以下支持文章:For more information, see the following support articles:

先决条件Prerequisites

  • Azure CLI 2.0.81 或更高版本Azure CLI version 2.0.81 or greater
  • Azure CLI 预览版扩展 0.4.28 或更高版本Azure CLI Preview extension version 0.4.28 or greater
  • API 2020-01-01 或更高版本API version of 2020-01-01 or greater

安装最新的 Azure CLI AKS 预览版扩展Install the latest Azure CLI AKS Preview extension

若要设置群集的出站类型,需要安装 Azure CLI AKS 预览版扩展 0.4.18 或更高版本。To set the outbound type of a cluster, you need the Azure CLI AKS Preview extension version 0.4.18 or later. 使用 az extension add 命令安装 Azure CLI AKS 预览版扩展,然后使用以下 az extension update 命令检查是否有任何可用更新:Install the Azure CLI AKS Preview extension by using the az extension add command, and then check for any available updates by using the following az extension update command:

# Install the aks-preview extension
az extension add --name aks-preview

# Update the extension to make sure you have the latest version installed
az extension update --name aks-preview

限制Limitations

  • 在预览版期间,只能在创建群集时定义 outboundType,以后无法对其进行更新。During preview, outboundType can only be defined at cluster create time and cannot be updated afterward.
  • 在预览版期间,outboundType AKS 群集应使用 Azure CNI。During preview, outboundType AKS clusters should use Azure CNI. Kubenet 是可配置的,若要使用它,需要手动将路由表关联到 AKS 子网。Kubenet is configurable, usage requires manual associations of the route table to the AKS subnet.
  • 设置 outboundType 需要将 AKS 群集的 vm-set-type 指定为 VirtualMachineScaleSets,将 load-balancer-sku 指定为 StandardSetting outboundType requires AKS clusters with a vm-set-type of VirtualMachineScaleSets and load-balancer-sku of Standard.
  • outboundType 设置为 UDR 值需要用户定义的路由以及群集的有效出站连接。Setting outboundType to a value of UDR requires a user-defined route with valid outbound connectivity for the cluster.
  • outboundType 设置为 UDR 值意味着,路由到负载均衡器的入口源 IP 可能与群集的传出出口目标地址不匹配Setting outboundType to a value of UDR implies the ingress source IP routed to the load-balancer may not match the cluster's outgoing egress destination address.

AKS 中的出站类型概述Overview of outbound types in AKS

可以使用类型为“负载均衡器”或“用户定义的路由”的唯一 outboundType 来自定义 AKS 群集。An AKS cluster can be customized with a unique outboundType of type load balancer or user-defined routing.

Important

出站类型仅影响群集的出口流量。Outbound type impacts only the egress traffic of your cluster. 有关详细信息,请参阅设置入口控制器See setting up ingress controllers for more information.

loadBalancer 的出站类型Outbound type of loadBalancer

如果设置了 loadBalancer,AKS 将自动完成以下设置。If loadBalancer is set, AKS completes the following setup automatically. 对于经过 AKS 分配的公共 IP 的传出流量,将使用负载均衡器。The load balancer is used for egress through an AKS assigned public IP. 出站类型 loadBalancer 支持 loadBalancer 类型的 Kubernetes 服务,此类型的服务应收到由 AKS 资源提供程序创建的负载均衡器传出的流量。An outbound type of loadBalancer supports Kubernetes services of type loadBalancer, which expect egress out of the load balancer created by the AKS resource provider.

以下设置由 AKS 完成。The following setup is done by AKS.

  • 为群集出口预配一个公共 IP 地址。A public IP address is provisioned for cluster egress.
  • 将该公共 IP 地址分配给负载均衡器资源。The public IP address is assigned to the load balancer resource.
  • 为群集中的代理节点设置负载均衡器的后端池。Backend pools for the load balancer are setup for agent nodes in the cluster.

下面是默认情况下在 AKS 群集中部署的网络拓扑,该拓扑使用类型为 loadBalanceroutboundTypeBelow is a network topology deployed in AKS clusters by default, which use an outboundType of loadBalancer.

outboundtype-lb

userDefinedRouting 的出站类型Outbound type of userDefinedRouting

Note

使用出站类型是一种高级网络方案,需要正确配置网络。Using outbound type is an advanced networking scenario and requires proper network configuration.

如果设置了 userDefinedRouting,则 AKS 不会自动配置出口路径。If userDefinedRouting is set, AKS will not automatically configure egress paths. 以下设置预期由用户完成。The following is expected to be done by the user.

必须将 AKS 群集部署到具有已配置的子网的现有虚拟网络。The AKS cluster must be deployed into an existing virtual network with a subnet that has been configured. 使用标准负载均衡器 (SLB) 体系结构时,必须建立显式出口。When using standard load balancer (SLB) architecture you must establish explicit egress. 这需要向防火墙、网关、本地等设备发送出口请求,或者需要将公共 IP 分配到标准负载均衡器或给定节点以完成出口操作。This requires sending egress requests to an appliance such as a firewall, gateway, on-prem or to allow the egress to be done by a public IP assigned to the standard load balancer or a given node.

AKS 资源提供程序将部署一个标准负载均衡器 (SLB)。The AKS resource provider will deploy a standard load balancer (SLB). 不会为负载均衡器配置任何规则,且在实施规则之前,负载均衡器不会产生费用The load balancer is not configured with any rules and does not incur a charge until a rule is placed. AKS 不会自动为 SLB 前端预配公共 IP 地址。AKS will not automatically provision a public IP address for the SLB frontend. AKS 不会自动配置负载均衡器后端池。AKS will not automatically configure the load balancer backend pool.

部署出站类型为 UDR 且具有 Azure 防火墙的群集Deploy a cluster with outbound type of UDR and Azure Firewall

若要演示如何应用出站类型为用户定义的路由的群集,可以在通过 Azure 防火墙对等互连的虚拟网络中配置一个群集。To illustrate the application of a cluster with outbound type using a user-defined route, a cluster can be configured on a virtual network peered with an Azure Firewall.

锁定的拓扑

  • 强制入口流量流经防火墙筛选器Ingress is forced to flow through firewall filters
    • 一个隔离的子网包含用于路由到代理节点的内部负载均衡器An isolated subnet holds an internal load balancer for routing into agent nodes
    • 代理节点隔离在专用子网中Agent nodes are isolated in a dedicated subnet
  • 出站请求从代理节点启动并使用用户定义的路由发送到 Azure 防火墙内部 IPOutbound requests start from agent nodes to the Azure Firewall internal IP using a user-defined route
    • 来自 AKS 代理节点的请求遵循 AKS 群集所部署到的子网中已放置的 UDR。Requests from AKS agent nodes follow a UDR that has been placed on the subnet the AKS cluster was deployed into.
    • Azure 防火墙通过公共 IP 前端将流量传出虚拟网络Azure Firewall egresses out of the virtual network from a public IP frontend
    • 对 AKS 控制平面的访问由已启用防火墙前端 IP 地址的 NSG 提供保护Access to the AKS control plane is protected by an NSG, which has enabled the firewall frontend IP address
    • 对公共 Internet 或其他 Azure 服务的访问流量会流入和流出防火墙前端 IP 地址Access to the public internet or other Azure services flows to and from the firewall frontend IP address

通过环境变量设置配置Set configuration via environment variables

定义创建资源时要使用的一组环境变量。Define a set of environment variables to be used in resource creations.

PREFIX="contosofin"
RG="${PREFIX}-rg"
LOC="chinaeast2"
NAME="${PREFIX}outboundudr"
AKS_NAME="${PREFIX}aks"
VNET_NAME="${PREFIX}vnet"
AKSSUBNET_NAME="${PREFIX}akssubnet"
SVCSUBNET_NAME="${PREFIX}svcsubnet"
# DO NOT CHANGE FWSUBNET_NAME - This is currently a requirement for Azure Firewall.
FWSUBNET_NAME="AzureFirewallSubnet"
FWNAME="${PREFIX}fw"
FWPUBLICIP_NAME="${PREFIX}fwpublicip"
FWIPCONFIG_NAME="${PREFIX}fwconfig"
FWROUTE_TABLE_NAME="${PREFIX}fwrt"
FWROUTE_NAME="${PREFIX}fwrn"
FWROUTE_NAME_INTERNET="${PREFIX}fwinternet"
DEVSUBNET_NAME="${PREFIX}dev"

接下来,设置订阅 ID。Next, set subscription IDs.

# Get ARM Access Token and Subscription ID - This will be used for AuthN later.

ACCESS_TOKEN=$(az account get-access-token -o tsv --query 'accessToken')

# NOTE: Update Subscription Name
# Set Default Azure Subscription to be Used via Subscription ID

az account set -s <SUBSCRIPTION_ID_GOES_HERE>

# NOTE: Update Subscription Name for setting SUBID

SUBID=$(az account show -s '<SUBSCRIPTION_NAME_GOES_HERE>' -o tsv --query 'id')

创建包含多个子网的虚拟网络Create a virtual network with multiple subnets

预配包含三个单独子网的虚拟网络,其中一个子网用于群集,一个子网用于防火墙,一个子网用于服务入口。Provision a virtual network with three separate subnets, one for the cluster, one for the firewall, and one for service ingress.

空网络拓扑

创建一个资源组来存放所有资源。Create a resource group to hold all of the resources.

# Create Resource Group

az group create --name $RG --location $LOC

创建两个虚拟网络来托管 AKS 群集和 Azure 防火墙。Create a two virtual networks to host the AKS cluster and the Azure Firewall. 每个虚拟网络都具有自己的子网。Each will have their own subnet. 让我们从 AKS 网络开始。Let's start with the AKS network.

# Dedicated virtual network with AKS subnet

az network vnet create \
    --resource-group $RG \
    --name $VNET_NAME \
    --address-prefixes 100.64.0.0/16 \
    --subnet-name $AKSSUBNET_NAME \
    --subnet-prefix 100.64.1.0/24

# Dedicated subnet for K8s services

az network vnet subnet create \
    --resource-group $RG \
    --vnet-name $VNET_NAME \
    --name $SVCSUBNET_NAME \
    --address-prefix 100.64.2.0/24

# Dedicated subnet for Azure Firewall (Firewall name cannot be changed)

az network vnet subnet create \
    --resource-group $RG \
    --vnet-name $VNET_NAME \
    --name $FWSUBNET_NAME \
    --address-prefix 100.64.3.0/24

创建并设置具有 UDR 的 Azure 防火墙Create and setup an Azure Firewall with a UDR

必须配置 Azure 防火墙入站和出站规则。Azure Firewall inbound and outbound rules must be configured. 防火墙的主要用途是使组织能够针对传入和传出 AKS 群集的流量设置精细的规则。The main purpose of the firewall is to enable organizations to setup granular ingress and egress traffic rules into and out of the AKS Cluster.

防火墙和 UDR

创建将用作 Azure 防火墙前端地址的标准 SKU 公共 IP 资源。Create a standard SKU public IP resource which will be used as the Azure Firewall frontend address.

az network public-ip create -g $RG -n $FWPUBLICIP_NAME -l $LOC --sku "Standard"

注册预览版 CLI 扩展以创建 Azure 防火墙。Register the preview cli-extension to create an Azure Firewall.

# Install Azure Firewall preview CLI extension

az extension add --name azure-firewall

# Deploy Azure Firewall

az network firewall create -g $RG -n $FWNAME -l $LOC

现在,可将前面创建的 IP 地址分配到防火墙前端。The IP address created earlier can now be assigned to the firewall frontend.

Note

设置 Azure 防火墙的公共 IP 地址可能需要几分钟时间。Setup of the public IP address to the Azure Firewall may take a few minutes.

如果以下命令反复出现错误,请删除现有的防火墙和公共 IP,同时通过门户预配公共 IP 和 Azure 防火墙。If errors are repeatedly received on the below command, delete the existing firewall and public IP and provision the Public IP and Azure Firewall through the portal at the same time.

# Configure Firewall IP Config

az network firewall ip-config create -g $RG -f $FWNAME -n $FWIPCONFIG_NAME --public-ip-address $FWPUBLICIP_NAME --vnet-name $VNET_NAME

当前面的命令成功时,保存防火墙前端 IP 地址以便稍后进行配置。When the previous command has succeeded, save the firewall frontend IP address for configuration later.

# Capture Firewall IP Address for Later Use

FWPUBLIC_IP=$(az network public-ip show -g $RG -n $FWPUBLICIP_NAME --query "ipAddress" -o tsv)
FWPRIVATE_IP=$(az network firewall show -g $RG -n $FWNAME --query "ipConfigurations[0].privateIpAddress" -o tsv)

创建包含 Azure 防火墙跃点的 UDRCreate a UDR with a hop to Azure Firewall

Azure 自动在 Azure 子网、虚拟网络与本地网络之间路由流量。Azure automatically routes traffic between Azure subnets, virtual networks, and on-premises networks. 若要更改 Azure 的任何默认路由,可以创建一个路由表。If you want to change any of Azure's default routing, you do so by creating a route table.

创建一个要与给定子网关联的空路由表。Create an empty route table to be associated with a given subnet. 该路由表将下一跃点定义为前面创建的 Azure 防火墙。The route table will define the next hop as the Azure Firewall created above. 每个子网可以有一个与之关联的路由表,也可以没有。Each subnet can have zero or one route table associated to it.

# Create UDR and add a route for Azure Firewall

az network route-table create -g $RG --name $FWROUTE_TABLE_NAME
az network route-table route create -g $RG --name $FWROUTE_NAME --route-table-name $FWROUTE_TABLE_NAME --address-prefix 0.0.0.0/0 --next-hop-type VirtualAppliance --next-hop-ip-address $FWPRIVATE_IP --subscription $SUBID
az network route-table route create -g $RG --name $FWROUTE_NAME_INTERNET --route-table-name $FWROUTE_TABLE_NAME --address-prefix $FWPUBLIC_IP/32 --next-hop-type Internet

请参阅虚拟网络路由表文档,了解如何替代 Azure 的默认系统路由或者在子网的路由表中添加更多路由。See virtual network route table documentation about how you can override Azure's default system routes or add additional routes to a subnet's route table.

添加网络防火墙规则Adding network firewall rules

Warning

以下示例展示了如何添加防火墙规则。Below shows one example of adding a firewall rule. 应用程序防火墙规则必须启用所需的出口终结点中定义的所有出口终结点,AKS 群集才能正常运行。All egress endpoints defined in the required egress endpoints must be enabled by application firewall rules for AKS clusters to function. 如果未启用这些终结点,则群集无法正常运行。Without these endpoints enabled, your cluster cannot operate.

下面是网络和应用程序规则的示例。Below is an example of a network and application rule. 我们添加了一个允许任何协议、源地址、目标地址和目标端口的网络规则。We add a network rule which allows any protocol, source-address, destination-address, and destination-ports. 我们还为 AKS 所需的某些终结点添加了应用程序规则。We also add an application rule for some of the endpoints required by AKS.

在生产方案中,只应启用对应用程序所需终结点以及 AKS 所需出口中定义的终结点的访问。In a production scenario, you should only enable access to required endpoints for your application and those defined in AKS required egress.

# Add Network FW Rules

az network firewall network-rule create -g $RG -f $FWNAME --collection-name 'aksfwnr' -n 'netrules' --protocols 'Any' --source-addresses '*' --destination-addresses '*' --destination-ports '*' --action allow --priority 100

# Add Application FW Rules
# IMPORTANT: Add AKS required egress endpoints

az network firewall application-rule create -g $RG -f $FWNAME \
    --collection-name 'AKS_Global_Required' \
    --action allow \
    --priority 100 \
    -n 'required' \
    --source-addresses '*' \
    --protocols 'http=80' 'https=443' \
    --target-fqdns \
        'aksrepos.azurecr.cn' \
        '*blob.core.chinacloudapi.cn' \
        'mcr.microsoft.com' \
        '*cdn.mscr.io' \
        '*.data.mcr.microsoft.com' \
        'management.chinacloudapi.cn' \
        'login.chinacloudapi.cn' \
        'ntp.ubuntu.com' \
        'packages.microsoft.com' \
        'acs-mirror.azureedge.net'

请参阅 Azure 防火墙文档来详细了解 Azure 防火墙服务。See Azure Firewall documentation to learn more about the Azure Firewall service.

将路由表关联到 AKSAssociate the route table to AKS

若要将群集与防火墙相关联,群集子网的专用子网必须引用前面创建的路由表。To associate the cluster with the firewall, the dedicated subnet for the cluster's subnet must reference the route table created above. 可以通过向包含群集和防火墙的虚拟网络发出更新群集子网路由表的命令来执行关联。Association can be done by issuing a command to the virtual network holding both the cluster and firewall to update the route table of the cluster's subnet.

# Associate route table with next hop to Firewall to the AKS subnet

az network vnet subnet update -g $RG --vnet-name $VNET_NAME --name $AKSSUBNET_NAME --route-table $FWROUTE_TABLE_NAME

将出站类型为 UDR 的 AKS 部署到现有网络Deploy AKS with outbound type of UDR to the existing network

现在,可将 AKS 群集部署到现有的虚拟网络设置中。Now an AKS cluster can be deployed into the existing virtual network setup. 若要将群集出站类型设置为用户定义的路由,必须向 AKS 提供一个现有子网。In order to set a cluster outbound type to user-defined routing, an existing subnet must be provided to AKS.

aks-deploy

创建有权在现有虚拟网络中进行预配的服务主体Create a service principal with access to provision inside the existing virtual network

AKS 使用服务主体来创建群集资源。A service principal is used by AKS to create cluster resources. 创建时传递的服务主体用于创建底层 AKS 资源,例如 AKS 使用的 VM、存储和负载均衡器。The service principal passed at create time is used to create underlying AKS resources such as VMs, Storage, and Load Balancers used by AKS. 如果授予的权限太少,将无法预配 AKS 群集。If granted too few permissions, it will not be able to provision an AKS Cluster.

# Create SP and Assign Permission to Virtual Network

az ad sp create-for-rbac -n "${PREFIX}sp" --skip-assignment

现在,请将下面的 APPIDPASSWORD 替换为前一命令输出自动生成的服务主体 appid 和服务主体密码。Now replace the APPID and PASSWORD below with the service principal appid and service principal password autogenerated by the previous command output. 我们将引用 VNET 资源 ID 来向服务主体授予权限,使 AKS 能够将资源部署到其中。We will reference the VNET resource ID to grant the permissions to the service principal so AKS can deploy resources into it.

APPID="<SERVICE_PRINCIPAL_APPID_GOES_HERE>"
PASSWORD="<SERVICEPRINCIPAL_PASSWORD_GOES_HERE>"
VNETID=$(az network vnet show -g $RG --name $VNET_NAME --query id -o tsv)

# Assign SP Permission to VNET

az role assignment create --assignee $APPID --scope $VNETID --role Contributor

# View Role Assignment
az role assignment list --assignee $APPID --all -o table

部署 AKSDeploy AKS

最后,可将 AKS 群集部署到专用于群集的现有子网中。Finally, the AKS cluster can be deployed into the existing subnet we have dedicated for the cluster. 要部署到的目标子网是使用环境变量 ($SUBNETID) 定义的。The target subnet to be deployed into is defined with the environment variable, $SUBNETID. 在前面的步骤中,我们未定义 $SUBNETID 变量。We didn't define the $SUBNETID variable in the previous steps. 若要设置子网 ID 的值,可使用以下命令:To set the value for the subnet ID, you can use the following command:

SUBNETID="/subscriptions/$SUBID/resourceGroups/$RG/providers/Microsoft.Network/virtualNetworks/$VNET_NAME/subnets/$AKSSUBNET_NAME"

我们将出站类型定义为遵循子网中存在的 UDR,使 AKS 能够跳过负载均衡器(现在可以严格要求它是内部负载均衡器)设置和 IP 预配。We will define the outbound type to follow the UDR which exists on the subnet, enabling AKS to skip setup and IP provisioning for the load balancer which can now be strictly internal.

可以添加 API 服务器已授权 IP 范围 AKS 功能,以便限制 API 服务器仅访问防火墙的公共终结点。The AKS feature for API server authorized IP ranges can be added to limit API server access to only the firewall's public endpoint. 已授权 IP 范围功能在示意图中表示为 NSG,必须通过此 NSG 才能访问控制平面。The authorized IP ranges feature is denoted in the diagram as the NSG which must be passed to access the control plane. 启用已授权 IP 范围功能来限制 API 服务器访问权限时,开发人员工具必须使用防火墙虚拟网络中的 Jumpbox,或者必须将所有开发人员终结点添加到已授权 IP 范围。When enabling the authorized IP range feature to limit API server access, your developer tools must use a jumpbox from the firewall's virtual network or you must add all developer endpoints to the authorized IP range.

Tip

可向群集部署添加更多功能,例如“专用群集”。Additional features can be added to the cluster deployment such as (Private Cluster)[]. 使用已授权 IP 范围时,Jumpbox 需位于群集网络内部才能访问 API 服务器。When using authorized IP ranges, a jumpbox will be required inside of the cluster network to access the API server.

az aks create -g $RG -n $AKS_NAME -l $LOC \
  --node-count 3 \
  --network-plugin azure --generate-ssh-keys \
  --service-cidr 192.168.0.0/16 \
  --dns-service-ip 192.168.0.10 \
  --docker-bridge-address 172.22.0.1/29 \
  --vnet-subnet-id $SUBNETID \
  --service-principal $APPID \
  --client-secret $PASSWORD \
  --load-balancer-sku standard \
  --outbound-type userDefinedRouting \
  --api-server-authorized-ip-ranges $FWPUBLIC_IP

使开发人员能够访问 API 服务器Enable developer access to the API server

由于为群集设置了已授权 IP 范围,因此必须将开发人员工具 IP 地址添加到 AKS 群集的已批准 IP 范围列表,才能访问 API 服务器。Due to the authorized IP ranges setup for the cluster, you must add your developer tooling IP addresses to the AKS cluster list of approved IP ranges to access the API server. 另一种做法是在防火墙虚拟网络中的单独子网内,使用所需的工具配置 Jumpbox。Another option is to configure a jumpbox with the needed tooling inside a separate subnet in the Firewall's virtual network.

使用以下命令将另一个 IP 地址添加到已批准范围Add another IP address to the approved ranges with the following command

# Retrieve your IP address
CURRENT_IP=$(dig @resolver1.opendns.com ANY myip.opendns.com +short)

# Add to AKS approved list
az aks update -g $RG -n $AKS_NAME --api-server-authorized-ip-ranges $CURRENT_IP/32

使用 az aks get-credentials 命令将 kubectl 配置为连接到新建的 Kubernetes 群集。Use the az aks get-credentials command to configure kubectl to connect to your newly created Kubernetes cluster.

az aks get-credentials -g $RG -n $AKS_NAME

设置内部负载均衡器Setup the internal load balancer

AKS 已在群集中部署了一个负载均衡器,可将该负载均衡器设置为内部负载均衡器AKS has deployed a load balancer with the cluster which can be setup as an internal load balancer.

若要创建内部负载均衡器,请创建名为 internal-lb.yaml 的服务清单并在其中包含 LoadBalancer 服务类型和 azure-load-balancer-internal 注释,如以下示例所示:To create an internal load balancer, create a service manifest named internal-lb.yaml with the service type LoadBalancer and the azure-load-balancer-internal annotation as shown in the following example:

apiVersion: v1
kind: Service
metadata:
  name: internal-app
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
    service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "contosofinsvcsubnet"
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: internal-app

使用 kubectl apply 部署内部负载均衡器,并指定 YAML 清单的名称:Deploy the internal load balancer using the kubectl apply and specify the name of your YAML manifest:

kubectl apply -f internal-lb.yaml

部署 Kubernetes 服务Deploy a Kubernetes service

由于群集出站类型设置为 UDR,在创建群集时,AKS 不会自动完成将代理节点关联为负载均衡器后端池的操作。Since the cluster outbound type is set as UDR, associating the agent nodes as the backend pool for the load balancer is not completed automatically by AKS at cluster create time. 但是,在部署 Kubernetes 服务时,Kubernetes Azure 云提供商会处理后端池关联。However, backend pool association is handled by the Kubernetes Azure cloud provider when the Kubernetes service is deployed.

通过将以下 yaml 复制为名为 example.yaml 的文件来部署 Azure 投票应用程序。Deploy the Azure voting app application by copying the yaml below to a file named example.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: azure-vote-back
spec:
  replicas: 1
  selector:
    matchLabels:
      app: azure-vote-back
  template:
    metadata:
      labels:
        app: azure-vote-back
    spec:
      nodeSelector:
        "beta.kubernetes.io/os": linux
      containers:
      - name: azure-vote-back
        image: redis
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 250m
            memory: 256Mi
        ports:
        - containerPort: 6379
          name: redis
---
apiVersion: v1
kind: Service
metadata:
  name: azure-vote-back
spec:
  ports:
  - port: 6379
  selector:
    app: azure-vote-back
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: azure-vote-front
spec:
  replicas: 1
  selector:
    matchLabels:
      app: azure-vote-front
  template:
    metadata:
      labels:
        app: azure-vote-front
    spec:
      nodeSelector:
        "beta.kubernetes.io/os": linux
      containers:
      - name: azure-vote-front
        image: dockerhub.azk8s.cn/microsoft/azure-vote-front:v1
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 250m
            memory: 256Mi
        ports:
        - containerPort: 80
        env:
        - name: REDIS
          value: "azure-vote-back"
---
apiVersion: v1
kind: Service
metadata:
  name: azure-vote-front
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
    service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "contosofinsvcsubnet"
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: azure-vote-front

运行以下命令来部署服务:Deploy the service by running:

kubectl apply -f example.yaml

将 DNAT 规则添加到 Azure 防火墙Add a DNAT rule to Azure Firewall

若要配置入站连接,必须将一个 DNAT 规则写入到 Azure 防火墙。To configure inbound connectivity, a DNAT rule must be written to the Azure Firewall. 为了测试与群集的连接,我们为防火墙前端公共 IP 地址定义了一个规则,以便路由到内部服务公开的内部 IP。To test connectivity to our cluster, a rule is defined for the firewall frontend public IP address to route to the internal IP exposed by the internal service.

可以自定义目标地址,因为它是防火墙上要访问的端口。The destination address can be customized as it is the port on the firewall to be accessed. 转换的地址必须是内部负载均衡器的 IP 地址。The translated address must be the IP address of the internal load balancer. 转换的端口必须是 Kubernetes 服务的已公开端口。The translated port must be the exposed port for your Kubernetes service.

你需要指定分配给 Kubernetes 服务所创建的负载均衡器的内部 IP 地址。You will need to specify the internal IP address assigned to the load balancer created by the Kubernetes service. 运行以下命令来检索该地址:Retrieve the address by running:

kubectl get services

所需的 IP 地址将在“EXTERNAL-IP”列中列出,如下所示。The IP address needed will be listed in the EXTERNAL-IP column, similar to the following.

NAME               TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
azure-vote-back    ClusterIP      192.168.92.209   <none>        6379/TCP       23m
azure-vote-front   LoadBalancer   192.168.19.183   100.64.2.5    80:32106/TCP   23m
kubernetes         ClusterIP      192.168.0.1      <none>        443/TCP        4d3h
az network firewall nat-rule create --collection-name exampleset --destination-addresses $FWPUBLIC_IP --destination-ports 80 --firewall-name $FWNAME --name inboundrule --protocols Any --resource-group $RG --source-addresses '*' --translated-port 80 --action Dnat --priority 100 --translated-address <INSERT IP OF K8s SERVICE>

清理资源Clean up resources

Note

删除 Kubernetes 内部服务时,如果内部负载均衡器不再由任何服务使用,则 Azure 云提供商会删除内部负载均衡器。When deleting the Kubernetes internal service, if the internal load balancer is no longer in use by any service, the Azure cloud provider will delete the internal load balancer. 下次部署服务时,如果找不到使用所请求配置的负载均衡器,则会部署一个负载均衡器。On the next service deployment, a load balancer will be deployed if none can be found with the configuration requested.

若要清理 Azure 资源,请删除 AKS 资源组。To clean up Azure resources, delete the AKS resource group.

az group delete -g $RG

验证连接Validate connectivity

在浏览器中导航到 Azure 防火墙前端 IP 地址来验证连接。Navigate to the Azure Firewall frontend IP address in a browser to validate connectivity.

应会看到 Azure 投票应用的图像。You should see an image of the Azure voting app.

后续步骤Next steps

参阅 Azure 网络 UDR 概述See Azure networking UDR overview.

参阅如何创建、更改或删除路由表See how to create, change, or delete a route table.