在 Azure Kubernetes 服务 (AKS) 中使用公共标准负载均衡器Use a public Standard Load Balancer in Azure Kubernetes Service (AKS)

Azure 负载均衡器是开放式系统互连 (OSI) 模型的 L4,支持入站和出站场景。The Azure Load Balancer is an L4 of the Open Systems Interconnection (OSI) model that supports both inbound and outbound scenarios. 负载均衡器将抵达负载均衡器前端的入站流量分配到后端池实例。It distributes inbound flows that arrive at the load balancer's front end to the backend pool instances.

公共负载均衡器与 AKS 集成时有两个用途:A public Load Balancer when integrated with AKS serves two purposes:

  1. 提供到 AKS 虚拟网络内群集节点的出站连接。To provide outbound connections to the cluster nodes inside the AKS virtual network. 它通过将节点专用 IP 地址转换为作为其“出站池”一部分的公共 IP 地址来实现此目标。It achieves this objective by translating the nodes private IP address to a public IP address that is part of its Outbound Pool .
  2. 通过类型为 LoadBalancer 的 Kubernetes 服务提供对应用程序的访问。To provide access to applications via Kubernetes services of type LoadBalancer. 使用负载均衡器你可轻松缩放应用程序,并创建高度可用的服务。With it, you can easily scale your applications and create highly available services.

内部(或专用)负载平衡器用于仅在前端允许专用 IP 的情况。An internal (or private) load balancer is used where only private IPs are allowed as frontend. 内部负载均衡器用于对虚拟网络内部的流量进行负载均衡。Internal load balancers are used to load balance traffic inside a virtual network. 负载均衡器前端还可在混合场景中从本地网络进行访问。A load balancer frontend can also be accessed from an on-premises network in a hybrid scenario.

本文档介绍与公共负载均衡器的集成。This document covers the integration with Public Load balancer. 如需了解内部负载均衡器集成,请参阅 AKS 内部负载均衡器文档For internal Load Balancer integration, see the AKS Internal Load balancer documentation.

准备阶段Before you begin

Azure 负载均衡器以两种 SKU 提供:“基本”和“标准” 。Azure Load Balancer is available in two SKUs - Basic and Standard . 默认情况下,创建 AKS 群集时将使用标准 SKU。By default, Standard SKU is used when you create an AKS cluster. 使用标准 SKU 访问附加功能,例如大型后端池、多个节点池Use the Standard SKU to have access to added functionality, such as a larger backend pool, multiple node pools. 这是推荐的 AKS 的负载均衡器 SKU。It's the recommended Load Balancer SKU for AKS.

有关基本和标准 SKU 的详细信息,请参阅 Azure 负载均衡器 SKU 的比较For more information on the Basic and Standard SKUs, see Azure load balancer SKU comparison.

本文假定你拥有具有标准 SKU Azure 负载均衡器的 AKS 群集,并介绍如何使用和配置负载均衡器的一些功能和特性。This article assumes you have an AKS cluster with the Standard SKU Azure Load Balancer and walks through how to use and configure some of the capabilities and features of the load balancer. 如果需要 AKS 群集,请参阅 AKS 快速入门使用 Azure CLI使用 Azure 门户If you need an AKS cluster, see the AKS quickstart using the Azure CLI or using the Azure portal.

重要

如果不想利用 Azure 负载均衡器提供出站连接,而希望为此拥有自己的网关、防火墙或代理,则可以使用出站类型作为 UserDefinedRouting (UDR),跳过创建负载均衡器出站池和相应的前端 IP 的操作。If you prefer not to leverage the Azure Load Balancer to provide outbound connection and instead have your own gateway, firewall or proxy for that purpose you can skip the creation of the load balancer outbound pool and respective frontend IP by using Outbound type as UserDefinedRouting (UDR). 出站类型定义群集的出口方法,它默认为负载均衡器类型。The Outbound type defines the egress method for a cluster and it defaults to type: load balancer.

使用公共标准负载均衡器Use the public standard load balancer

创建默认出站类型为负载均衡器的 AKS 群集后,群集也可使用负载均衡器公开服务。After creating an AKS cluster with Outbound Type: Load Balancer (default), the cluster is ready to use the load balancer to expose services as well.

为此,你可创建类型为 LoadBalancer 的公共服务,如以下示例所示。For that you can create a public Service of type LoadBalancer as shown in the following example. 首先创建名为 public-svc.yaml 的服务清单:Start by creating a service manifest named public-svc.yaml:

apiVersion: v1
kind: Service
metadata:
  name: public-svc
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: public-app

使用 kubectl apply 部署公共服务清单,并指定 YAML 清单的名称:Deploy the public service manifest by using kubectl apply and specify the name of your YAML manifest:

kubectl apply -f public-svc.yaml

将为 Azure 负载均衡器配置一个支持此新服务的新公共 IP。The Azure Load Balancer will be configured with a new public IP that will front this new service. 由于 Azure 负载均衡器可以有多个前端 IP,因此部署的每个新服务都将获得唯一的专用前端 IP。Since the Azure Load Balancer can have multiple Frontend IPs, each new service deployed will get a new dedicated frontend IP to be uniquely accessed.

你可以确认你的服务已创建,并且通过运行来配置负载均衡器,例如:You can confirm your service is created and the load balancer is configured by running for example:

kubectl get service public-svc
NAMESPACE     NAME          TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)         AGE
default       public-svc    LoadBalancer   10.0.39.110    52.156.88.187   80:32068/TCP    52s

查看服务详细信息时,为负载均衡器上的该服务创建的公共 IP 地址显示在“EXTERNAL-IP”列中。When you view the service details, the public IP address created for this service on the load balancer is shown in the EXTERNAL-IP column. 可能需要一两分钟,IP 地址才会从 <pending> 更改为实际的公共 IP 地址,如以下示例所示。It may take a minute or two for the IP address to change from <pending> to an actual public IP address, as shown in the above example.

配置公共标准负载均衡器Configure the public standard load balancer

使用标准 SKU 公共负载均衡器时,可以在创建时或通过更新群集自定义一组选项。When using the Standard SKU public load balancer, there's a set of options that can be customized at creation time or by updating the cluster. 通过这些选项,你可自定义负载均衡器以满足工作负载需求,并应相应地进行审查。These options allow you to customize the Load Balancer to meet your workloads needs and should be reviewed accordingly. 使用标准负载均衡器,你可以执行以下操作:With the Standard load balancer you can:

  • 设置或缩放受管理出站 IP 的数量Set or scale the number of Managed Outbound IPs
  • 自带自定义出站 IP 或出站 IP 前缀Bring your own custom Outbound IPs or Outbound IP Prefix
  • 自定义分配给群集的每个节点的出站端口数Customize the number of allocated outbound ports to each node of the cluster
  • 为空闲连接配置超时设置Configure the timeout setting for idle connections

缩放受管理出站公共 IP 的数量Scale the number of managed outbound public IPs

除了入站连接以外,Azure 负载均衡器还提供从虚拟网络的出站连接。Azure Load Balancer provides outbound connectivity from a virtual network in addition to inbound. 使用出站规则可以更方便地配置公共标准负载均衡器的出站网络地址转换。Outbound rules make it simple to configure public Standard Load Balancer's outbound network address translation.

与所有负载均衡器规则一样,出站规则遵循负载均衡和入站 NAT 规则的类似语法:Like all Load Balancer rules, outbound rules follow the same familiar syntax as load balancing and inbound NAT rules:

*前端 IP + 参数 + 后端池*frontend IPs + parameters + backend pool _

出站规则为后端池识别的、要转换为前端的所有虚拟机配置出站 NAT。An outbound rule configures outbound NAT for all virtual machines identified by the backend pool to be translated to the frontend. 参数针对出站 NAT 算法提供更精细的控制。And parameters provide additional fine grained control over the outbound NAT algorithm.

尽管出站规则只能配合单个公共 IP 地址使用,但出站规则减轻了缩放出站 NAT 的负担。While an outbound rule can be used with just a single public IP address, outbound rules ease the configuration burden for scaling outbound NAT. 规划大规模部署场景时可以使用多个 IP 地址,并可以使用出站规则来缓解容易出现 SNAT 耗尽的模式。You can use multiple IP addresses to plan for large-scale scenarios and you can use outbound rules to mitigate SNAT exhaustion prone patterns. 前端提供的每个附加 IP 地址可提供 64,000 个临时端口,供负载均衡器用作 SNAT 端口。Each additional IP address provided by a frontend provides 64k ephemeral ports for Load Balancer to use as SNAT ports.

使用具有默认创建的托管出站公共 IP 的标准 SKU 负载均衡器时,可以使用 load-balancer-managed-ip-count 参数缩放托管出站公共 IP 的数量。When using a _Standard* SKU load balancer with managed outbound public IPs, which are created by default, you can scale the number of managed outbound public IPs using the load-balancer-managed-ip-count parameter.

若要更新现有群集,请运行以下命令。To update an existing cluster, run the following command. 还可以在创建群集时设置此参数,以获得多个托管出站公共 IP。This parameter can also be set at cluster create-time to have multiple managed outbound public IPs.

az aks update \
    --resource-group myResourceGroup \
    --name myAKSCluster \
    --load-balancer-managed-outbound-ip-count 2

以上示例将 myResourceGroupmyAKSCluster 群集的托管出站公共 IP 数量设置为 2The above example sets the number of managed outbound public IPs to 2 for the myAKSCluster cluster in myResourceGroup .

还可以在创建群集时,通过追加 --load-balancer-managed-outbound-ip-count 参数并将其设置为所需的值,使用 load-balancer-managed-ip-count 参数来设置托管出站公共 IP 的初始数量 。You can also use the load-balancer-managed-ip-count parameter to set the initial number of managed outbound public IPs when creating your cluster by appending the --load-balancer-managed-outbound-ip-count parameter and setting it to your desired value. 托管出站公共 IP 的默认数量为 1。The default number of managed outbound public IPs is 1.

提供自己的出站公共 IP 或前缀Provide your own outbound public IPs or prefixes

使用标准 SKU 负载均衡器时,默认情况下,AKS 群集会自动在 AKS 管理的基础结构资源组中创建公共 IP 并将其分配给负载均衡器出站池。When you use a Standard SKU load balancer, by default the AKS cluster automatically creates a public IP in the AKS-managed infrastructure resource group and assigns it to the load balancer outbound pool.

AKS 创建的公共 IP 被视为受 AKS 管理的资源。A public IP created by AKS is considered an AKS managed resource. 这意味着该公共 IP 的生命周期由 AKS 管理,并且用户不需要直接对公共 IP 资源执行操作。This means the lifecycle of that public IP is intended to be managed by AKS and requires no user action directly on the public IP resource. 或者,你可以在群集创建时分配自己的自定义公共 IP 或公共 IP 前缀。Alternatively, you can assign your own custom public IP or public IP prefix at cluster creation time. 还可以在现有群集的负载均衡器属性上更新自定义 IP。Your custom IPs can also be updated on an existing cluster's load balancer properties.

备注

自定义公共 IP 地址必须由用户创建和拥有。Custom public IP addresses must be created and owned by the user. 由 AKS 创建的托管公共 IP 地址不能重新用于自带的自定义 IP,因为它可能会导致管理冲突。Managed public IP addresses created by AKS cannot be reused as a bring your own custom IP as it can cause management conflicts.

在执行此操作前,请确保满足配置出站 IP 或出站 IP 前缀所需的先决条件和限制Before you do this operation, make sure you meet the pre-requisites and constraints necessary to configure Outbound IPs or Outbound IP prefixes.

使用自己的出站公共 IP 更新群集Update the cluster with your own outbound public IP

使用 az network public-ip show 命令列出公共 IP 的 ID。Use the az network public-ip show command to list the IDs of your public IPs.

az network public-ip show --resource-group myResourceGroup --name myPublicIP --query id -o tsv

以上命令显示 myResourceGroup 资源组中 myPublicIP 公共 IP 的 ID。The above command shows the ID for the myPublicIP public IP in the myResourceGroup resource group.

az aks update 命令与 load-balancer-outbound-ips 参数一起使用,以使用公共 IP 更新群集。Use the az aks update command with the load-balancer-outbound-ips parameter to update your cluster with your public IPs.

以下示例结合前一命令返回的 ID 使用 load-balancer-outbound-ips 参数。The following example uses the load-balancer-outbound-ips parameter with the IDs from the previous command.

az aks update \
    --resource-group myResourceGroup \
    --name myAKSCluster \
    --load-balancer-outbound-ips <publicIpId1>,<publicIpId2>

使用自己的出站公共 IP 前缀更新群集Update the cluster with your own outbound public IP prefix

你也可以将出口的公共 IP 前缀与标准 SKU 负载均衡器配合使用。You can also use public IP prefixes for egress with your Standard SKU load balancer. 以下示例使用 az network public-ip prefix show 命令列出公共 IP 前缀的 ID:The following example uses the az network public-ip prefix show command to list the IDs of your public IP prefixes:

az network public-ip prefix show --resource-group myResourceGroup --name myPublicIPPrefix --query id -o tsv

以上命令显示 myResourceGroup 资源组中 myPublicIPPrefix 公共 IP 前缀的 ID。The above command shows the ID for the myPublicIPPrefix public IP prefix in the myResourceGroup resource group.

以下示例将 load-balancer-outbound-ip-prefixes 参数与前一命令返回的 ID 配合使用。The following example uses the load-balancer-outbound-ip-prefixes parameter with the IDs from the previous command.

az aks update \
    --resource-group myResourceGroup \
    --name myAKSCluster \
    --load-balancer-outbound-ip-prefixes <publicIpPrefixId1>,<publicIpPrefixId2>

使用自己的公共 IP 或前缀创建群集Create the cluster with your own public IP or prefixes

你可能希望在创建群集时为出口提供自己的 IP 地址或 IP 前缀,以支持将出口终结点添加到允许列表这样的方案。You may wish to bring your own IP addresses or IP prefixes for egress at cluster creation time to support scenarios like adding egress endpoints to an allow list. 将上面所示的相同参数追加到群集创建步骤,可在群集生命周期的起始部分定义自己的公共 IP 和 IP 前缀。Append the same parameters shown above to your cluster creation step to define your own public IPs and IP prefixes at the start of a cluster's lifecycle.

结合 load-balancer-outbound-ips 参数使用 az aks create 命令可在启动时使用你的公共 IP 创建新的群集。Use the az aks create command with the load-balancer-outbound-ips parameter to create a new cluster with your public IPs at the start.

az aks create \
    --resource-group myResourceGroup \
    --name myAKSCluster \
    --load-balancer-outbound-ips <publicIpId1>,<publicIpId2>

结合 load-balancer-outbound-ip-prefixes 参数使用 az aks create 命令可在启动时使用你的公共 IP 前缀创建新的群集。Use the az aks create command with the load-balancer-outbound-ip-prefixes parameter to create a new cluster with your public IP prefixes at the start.

az aks create \
    --resource-group myResourceGroup \
    --load-balancer-outbound-ip-prefixes <publicIpPrefixId1>,<publicIpPrefixId2>

配置分配的出站端口Configure the allocated outbound ports

重要

如果你的群集上有一些应用程序,它们预期会与一小组目标集(例如,If you have applications on your cluster which are expected to establish a large number of connection to small set of destinations, eg. 许多连接到 SQL DB 的前端实例)建立大量连接,你的情况是很容易遇到 SNAT 端口耗尽(要连接的端口用完)。many frontend instances connecting to an SQL DB, you have a scenario very susceptible to encounter SNAT Port exhaustion (run out of ports to connect from). 对于这些场景,强烈建议增加负载均衡器上分配的出站端口和出站前端 IP。For these scenarios it's highly recommended to increase the allocated outbound ports and outbound frontend IPs on the load balancer. 此增加操作应考虑一 (1) 个附加 IP 地址添加 64,000 个附加端口,以便分布在所有群集节点上。The increase should consider that one (1) additional IP address adds 64k additional ports to distribute across all cluster nodes.

除非另有说明,否则 AKS 将使用标准负载均衡器在配置时所定义的分配出站端口的默认值。Unless otherwise specified, AKS will use the default value of Allocated Outbound Ports that Standard Load Balancer defines when configuring it. 在 AKS API 上此值为 null,或在 SLB API 上为 0,如以下命令所示 :This value is null on the AKS API or 0 on the SLB API as shown by the below command:

NODE_RG=$(az aks show --resource-group myResourceGroup --name myAKSCluster --query nodeResourceGroup -o tsv)
az network lb outbound-rule list --resource-group $NODE_RG --lb-name kubernetes -o table

上述命令将列出负载均衡器的出站规则,例如:The previous commands will list the outbound rule for your load balancer, for example:

AllocatedOutboundPorts    EnableTcpReset    IdleTimeoutInMinutes    Name             Protocol    ProvisioningState    ResourceGroup
------------------------ ----------------  ---------------------- ---------------  ---------- -------------------  -------------
0                         True              30                      aksOutboundRule  All         Succeeded            MC_myResourceGroup_myAKSCluster_chinaeast2  

此输出并不意味着你有 0 个端口,而是你利用基于后端池大小的自动出站端口分配,因此,举例来说,如果群集有 50 个或更少的节点,则为每个节点分配 1024 个端口,当你增加节点数时,每个节点的端口数将逐渐减少。This output does not mean that you have 0 ports but instead that you are leveraging the automatic outbound port assignment based on backend pool size, so for example if a cluster has 50 or less nodes, 1024 ports for each node are allocated, as you increase the number of nodes from there you'll gradually get fewer ports per node.

若要定义或增加分配的出站端口的数量,可以按照以下示例操作:To define or increase the number of Allocated Outbound ports, you can follow the below example:

az aks update \
    --resource-group myResourceGroup \
    --name myAKSCluster \
    --load-balancer-managed-outbound-ip-count 7 \
    --load-balancer-outbound-ports 4000

此示例假设为群集中的每个节点提供 4000 个分配的出站端口以及 7 个 IP,则你会得到以下结果:每个节点 4000 个端口 * 100 个节点 = 400,000 个总端口 < = 448,000 个总端口 = 7 个 IP * 每个 IP 64,000 端口。This example would give you 4000 Allocated Outbound Ports for each node in my cluster, and with 7 IPs you would have 4000 ports per node * 100 nodes = 400k total ports < = 448k total ports = 7 IPs * 64k ports per IP . 这样你便可安全地缩放到 100 个节点,并执行默认升级操作。This would allow you to safely scale to 100 nodes and have a default upgrade operation. 为升级和其他操作所需的其他节点分配足够的端口至关重要。It is critical to allocate sufficient ports for additional nodes needed for upgrade and other operations. AKS 默认为一个缓冲区节点用于升级,在此示例中,这要求在任何给定时间点有 4000 个可用端口。AKS defaults to one buffer node for upgrade, in this example this requires 4000 free ports at any given point in time.

要安全地超过 100 个节点,必须添加更多 IP。To safely go above 100 nodes, you'd have to add more IPs.

重要

在自定义 allocatedOutboundPort 之前,必须计算所需的配额并检查要求以避免连接或缩放问题。You must calculate your required quota and check the requirements before customizing allocatedOutboundPorts to avoid connectivity or scaling issues.

创建群集时,还可以使用 load-balancer-outbound-ports 参数,但必须同时指定 load-balancer-managed-outbound-ip-countload-balancer-outbound-ipsload-balancer-outbound-ip-prefixesYou can also use the load-balancer-outbound-ports parameters when creating a cluster, but you must also specify either load-balancer-managed-outbound-ip-count , load-balancer-outbound-ips , or load-balancer-outbound-ip-prefixes as well. 例如:For example:

az aks create \
    --resource-group myResourceGroup \
    --name myAKSCluster \
    --load-balancer-sku standard \
    --load-balancer-managed-outbound-ip-count 2 \
    --load-balancer-outbound-ports 1024 

配置负载均衡器的空闲超时Configure the load balancer idle timeout

如果 SNAT 端口资源已经耗尽,那么在现有流释放 SNAT 端口之前出站流会失败。When SNAT port resources are exhausted, outbound flows fail until existing flows release SNAT ports. 当流关闭时,负载均衡器将回收 SNAT 端口,AKS 配置的负载均衡器将使用 30 分钟空闲超时回收空闲流中的 SNAT 端口。Load Balancer reclaims SNAT ports when the flow closes and the AKS-configured load balancer uses a 30-minute idle timeout for reclaiming SNAT ports from idle flows. 还可以使用传输(例如 TCP keepalives)或 application-layer keepalives 刷新空闲流,并在必要时重置此空闲超时 。You can also use transport (for example, TCP keepalives ) or application-layer keepalives to refresh an idle flow and reset this idle timeout if necessary. 可以按照以下示例配置此超时:You can configure this timeout following the below example:

az aks update \
    --resource-group myResourceGroup \
    --name myAKSCluster \
    --load-balancer-idle-timeout 4

如果你希望具有许多短生存期的连接,而不想有长生存期且可能有长时间空闲的连接(例如利用 kubectl proxykubectl port-forward),请考虑使用低超时值(如 4 分钟)。If you expect to have numerous short lived connections, and no connections that are long lived and might have long times of idle, like leveraging kubectl proxy or kubectl port-forward consider using a low timeout value such as 4 minutes. 另外,使用 TCP keepalive 时,在连接的一端启用它们就足够了。Also, when using TCP keepalives, it's sufficient to enable them on one side of the connection. 例如,若要重置流的空闲计时器,在服务器端启用它们就足够了,没有必要在两端都启动 TCP keepalive。For example, it's sufficient to enable them on the server side only to reset the idle timer of the flow and it's not necessary for both sides to start TCP keepalives. 应用程序层(包括数据库客户端-服务器配置)也存在类似的概念。Similar concepts exist for application layer, including database client-server configurations. 检查服务器端对于特定于应用程序的 keepalive 存在哪些选项。Check the server side for what options exist for application-specific keepalives.

重要

AKS 默认启用“空闲时执行 TCP 重置”,建议保持此配置,并利用它在你的场景中实现可预测性更高的应用程序行为。AKS enables TCP Reset on idle by default and recommends you keep this configuration on and leverage it for more predictable application behavior on your scenarios. 只有在 TCP 连接的状态为“已建立”时才会发送 TCP RST。TCP RST is only sent during TCP connection in ESTABLISHED state. 此处阅读详细信息。Read more about it here.

自定义分配的出站端口和空闲超时的要求Requirements for customizing allocated outbound ports and idle timeout

  • allocatedOutboundPorts 指定的值还必须是 8 的倍数。The value you specify for allocatedOutboundPorts must also be a multiple of 8.

  • 你必须有足够的出站 IP 容量,具体取决于节点 VM 和所需的已分配出站端口的数量。You must have enough outbound IP capacity based on the number of your node VMs and required allocated outbound ports. 若要验证是否有足够的出站 IP 容量,请使用以下公式:To validate you have enough outbound IP capacity, use the following formula:

    outboundIPs * 64,000 > nodeVMs * desiredAllocatedOutboundPortsoutboundIPs * 64,000 > nodeVMs * desiredAllocatedOutboundPorts .

    例如,如果你有 3 个 nodeVM 和 50,000 个 desiredAllocatedOutboundPort ,则至少需要有 3 个 outboundIPFor example, if you have 3 nodeVMs , and 50,000 desiredAllocatedOutboundPorts , you need to have at least 3 outboundIPs . 建议你在所需容量的基础上增加额外的出站 IP 容量。It is recommended that you incorporate additional outbound IP capacity beyond what you need. 此外,在计算出站 IP 容量时,必须考虑群集自动缩放程序和节点池升级的可能性。Additionally, you must account for the cluster autoscaler and the possibility of node pool upgrades when calculating outbound IP capacity. 对于群集自动缩放程序,请查看当前节点计数和最大节点计数,并使用较高的值。For the cluster autoscaler, review the current node count and the maximum node count and use the higher value. 对于升级,请考虑为允许升级的节点池添加一个额外的节点 VM。For upgrading, account for an additional node VM for every node pool that allows upgrading.

  • IdleTimeoutInMinutes 设置为默认值 30 分钟之外的值时,请考虑你的工作负荷多长时间将需要出站连接。When setting IdleTimeoutInMinutes to a different value than the default of 30 minutes, consider how long your workloads will need an outbound connection. 还要考虑在 AKS 外部使用的“标准”SKU 负载平衡器的默认超时值是 4 分钟。Also consider the default timeout value for a Standard SKU load balancer used outside of AKS is 4 minutes. 如果 idletimeoutminutes 值较准确地反映你的具体 AKS 工作负载,则有助于降低由于绑定不再使用的连接而导致的 SNAT 耗尽。An IdleTimeoutInMinutes value that more accurately reflects your specific AKS workload can help decrease SNAT exhaustion caused by tying up connections no longer being used.

警告

更改 AllocatedOutboundPort 和 IdleTimeoutInMinute 的值可能会显著更改负载均衡器的出站规则的行为,因此在不了解弊端和应用程序连接模式的情况下,不应轻易去做,请查看下面的 SNAT 故障排除部分,并在更新这些值之前查看负载均衡器出站规则Azure 中的出站连接,以充分了解更改的影响 。Altering the values for AllocatedOutboundPorts and IdleTimeoutInMinutes may significantly change the behavior of the outbound rule for your load balancer and should not be done lightly, without understanding the tradeoffs and your application's connection patterns, check the SNAT Troubleshooting section below and review the Load Balancer outbound rules and outbound connections in Azure before updating these values to fully understand the impact of your changes.

将入站流量限制为特定 IP 范围Restrict inbound traffic to specific IP ranges

以下清单使用 loadBalancerSourceRanges 来指定允许其发送入站外部流量的新 IP 范围:The following manifest uses loadBalancerSourceRanges to specify a new IP range for inbound external traffic:

apiVersion: v1
kind: Service
metadata:
  name: azure-vote-front
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: azure-vote-front
  loadBalancerSourceRanges:
  - MY_EXTERNAL_IP_RANGE

备注

入站、外部流量从负载均衡器流向 AKS 群集的虚拟网络。Inbound, external traffic flows from the load balancer to the virtual network for your AKS cluster. 该虚拟网络有一个网络安全组 (NSG),该安全组允许来自负载均衡器的所有入站流量。The virtual network has a Network Security Group (NSG) which allows all inbound traffic from the load balancer. 此 NSG 使用 LoadBalancer 类型的服务标记允许来自负载均衡器的流量。This NSG uses a service tag of type LoadBalancer to allow traffic from the load balancer.

维护入站连接上的客户端 IPMaintain the client's IP on inbound connections

默认情况下,在 Kubernetes 和 AKS 中,LoadBalancer 类型的服务在连接到 Pod 时不会持久保留客户端的 IP 地址。By default, a service of type LoadBalancer in Kubernetes and in AKS won't persist the client's IP address on the connection to the pod. 传递到 Pod 的数据包上的源 IP 将是节点的专用 IP。The source IP on the packet that's delivered to the pod will be the private IP of the node. 若要保留客户端的 IP 地址,必须在服务定义中将 service.spec.externalTrafficPolicy 设置为 localTo maintain the client's IP address, you must set service.spec.externalTrafficPolicy to local in the service definition. 以下清单显示了一个示例:The following manifest shows an example:

apiVersion: v1
kind: Service
metadata:
  name: azure-vote-front
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  ports:
  - port: 80
  selector:
    app: azure-vote-front

通过 Kubernetes 注释进行其他自定义Additional customizations via Kubernetes Annotations

下面是类型 LoadBalancer 的 Kubernetes 服务支持的注释列表,这些注释仅适用于入站流:Below is a list of annotations supported for Kubernetes services with type LoadBalancer, these annotations only apply to INBOUND flows:

AnnotationAnnotation ValueValue 说明Description
service.beta.kubernetes.io/azure-load-balancer-internal truefalsetrue or false 指定负载均衡器是否应为“内部”。Specify whether the load balancer should be internal. 如果未设置,则默认为“公共”。It's defaulting to public if not set.
service.beta.kubernetes.io/azure-load-balancer-internal-subnet 子网的名称Name of the subnet 指定内部负载均衡器应绑定到的子网。Specify which subnet the internal load balancer should be bound to. 如果未设置,则默认为云配置文件中配置的子网。It's defaulting to the subnet configured in cloud config file if not set.
service.beta.kubernetes.io/azure-dns-label-name 公共 IP 上的 DNS 标签的名称Name of the DNS label on Public IPs 指定公共服务的 DNS 标签的名称。Specify the DNS label name for the public service. 如果设置为空字符串,则不会使用公共 IP 中的 DNS 条目。If it is set to empty string, the DNS entry in the Public IP will not be used.
service.beta.kubernetes.io/azure-shared-securityrule truefalsetrue or false 指定应使用可能与其他服务共享的 Azure 安全规则公开服务,交易规则的特定性,以增加可公开的服务数量。Specify that the service should be exposed using an Azure security rule that may be shared with another service, trading specificity of rules for an increase in the number of services that can be exposed. 此注释依赖于网络安全组的 Azure 扩充式安全规则功能。This annotation relies on the Azure Augmented Security Rules feature of Network Security groups.
service.beta.kubernetes.io/azure-load-balancer-resource-group 资源组的名称Name of the resource group 指定与群集基础结构(节点资源组)不在同一资源组中的负载均衡器公共 IP 的资源组。Specify the resource group of load balancer public IPs that aren't in the same resource group as the cluster infrastructure (node resource group).
service.beta.kubernetes.io/azure-allowed-service-tags 允许的服务标记列表List of allowed service tags 指定以逗号隔开的允许服务标记的列表。Specify a list of allowed service tags separated by comma.
service.beta.kubernetes.io/azure-load-balancer-tcp-idle-timeout TCP 空闲超时(以分钟为单位)TCP idle timeouts in minutes 指定 TCP 连接空闲超时在负载均衡器上发生的时间(以分钟为单位)。Specify the time, in minutes, for TCP connection idle timeouts to occur on the load balancer. 默认值和最小值为 4。Default and minimum value is 4. 最大值为 30。Maximum value is 30. 必须为整数。Must be an integer.
service.beta.kubernetes.io/azure-load-balancer-disable-tcp-reset true 为 SLB 禁用 enableTcpResetDisable enableTcpReset for SLB

SNAT 疑难解答Troubleshooting SNAT

如果知道正在启动与同一目标 IP 地址和端口的多个出站 TCP 或 UDP 连接,观察失败的出站连接,或者支持人员通知已耗尽 SNAT 端口(PAT 使用的预先分配的临时端口),则有几个常见缓解选项可供选择。If you know that you're starting many outbound TCP or UDP connections to the same destination IP address and port, and you observe failing outbound connections or are advised by support that you're exhausting SNAT ports (preallocated ephemeral ports used by PAT), you have several general mitigation options. 查看这些选项,确定可用且最适合自己的方案的选项。Review these options and decide what is available and best for your scenario. 一个或多个选项可能有助于管理此方案。It's possible that one or more can help manage this scenario. 有关详细信息,请查看出站连接故障排除指南For detailed information, review the Outbound Connections Troubleshooting Guide.

通常,SNAT 耗尽的根本原因是建立和管理出站连接的方式出现了对立模式,或者可配置的计时器已更改,不再使用默认值。Frequently the root cause of SNAT exhaustion is an anti-pattern for how outbound connectivity is established, managed, or configurable timers changed from their default values. 请认真阅读本部分。Review this section carefully.

步骤Steps

  1. 检查连接是否长时间处于空闲状态,以及是否依靠默认的空闲超时释放该端口。Check if your connections remain idle for a long time and rely on the default idle timeout for releasing that port. 如果是,可能需要为你的场景减少 30 分钟的默认超时。If so the default timeout of 30 min might need to be reduced for your scenario.
  2. 调查应用程序如何建立出站连接(例如,通过代码评审或数据包捕获进行调查)。Investigate how your application is creating outbound connectivity (for example, code review or packet capture).
  3. 确定此活动是否为预期行为,或者应用程序是否行为异常。Determine if this activity is expected behavior or whether the application is misbehaving. 使用 Azure Monitor 中的指标日志来证实发现结果。Use metrics and logs in Azure Monitor to substantiate your findings. 例如,对“SNAT 连接”指标使用“失败”类别。Use "Failed" category for SNAT Connections metric for example.
  4. 评估是否遵循适当的模式Evaluate if appropriate patterns are followed.
  5. 评估更多出站 IP 地址 + 更多分配的出站端口是否应可以缓解 SNAT 端口耗尽的问题。Evaluate if SNAT port exhaustion should be mitigated with additional Outbound IP addresses + additional Allocated Outbound Ports .

设计模式Design patterns

始终尽量利用连接重用和连接池。Always take advantage of connection reuse and connection pooling whenever possible. 这些模式可以避免资源耗尽问题,并使行为可预测。These patterns will avoid resource exhaustion problems and result in predictable behavior. 在许多开发库和框架中,都可以找到这些模式的根源。Primitives for these patterns can be found in many development libraries and frameworks.

  • 原子请求(每个连接一个请求)是一个通常并不良好的设计选项。Atomic requests (one request per connection) are generally not a good design choice. 这种对立模式会限制缩放,降低性能并降低可靠性。Such anti-pattern limits scale, reduces performance, and decreases reliability. 应该重复使用 HTTP/S 连接来减少连接数和关联的 SNAT 端口数。Instead, reuse HTTP/S connections to reduce the numbers of connections and associated SNAT ports. 由于使用 TLS 时可以减少握手次数、系统开销以及加密操作的开销,因此应用规模与性能都会提高。The application scale will increase and performance improve because of reduced handshakes, overhead, and cryptographic operation cost when using TLS.
  • 如果你使用的是群集/自定义 DNS,或者 coreDNS 上的自定义上游服务器,请记住,当客户端不缓存 DNS 解析器结果时,DNS 可以在卷上引入许多单独的流。If you're using out of cluster/custom DNS, or custom upstream servers on coreDNS have in mind that DNS can introduce many individual flows at volume when the client isn't caching the DNS resolvers result. 请确保首先自定义 coreDNS 而不是使用自定义 DNS 服务器,并定义合适的缓存值。Make sure to customize coreDNS first instead of using custom DNS servers, and define a good caching value.
  • UDP 流(例如 DNS 查找)根据空闲超时持续时间分配 SNAT 端口。UDP flows (for example DNS lookups) allocate SNAT ports for the duration of the idle timeout. 空闲超时越长,SNAT 端口上的压力越大。The longer the idle timeout, the higher the pressure on SNAT ports. 使用较短的空闲超时(例如 4 分钟)。Use short idle timeout (for example 4 minutes). 使用连接池来调整连接卷。Use connection pools to shape your connection volume.
  • 切勿以静默方式丢弃 TCP 流,且不要依赖 TCP 计时器来清理流。Never silently abandon a TCP flow and rely on TCP timers to clean up flow. 如果不允许 TCP 显式关闭连接,中间系统和终结点上将保持已分配状态,使 SNAT 端口不可用于其他连接。If you don't let TCP explicitly close the connection, state remains allocated at intermediate systems and endpoints and makes SNAT ports unavailable for other connections. 此模式可能会触发应用程序故障和 SNAT 耗尽。This pattern can trigger application failures and SNAT exhaustion.
  • 在对造成的影响了解不深的情况下,请不要更改与 OS 级别的 TCP 关闭相关的计时器值。Don't change OS-level TCP close related timer values without expert knowledge of impact. 当某个连接的终结点不符合预期时,尽管 TCP 堆栈会恢复,但应用程序的性能可能会受负面影响。While the TCP stack will recover, your application performance can be negatively affected when the endpoints of a connection have mismatched expectations. 希望更改计时器往往意味着底层设计出现了问题。Wishing to change timers is usually a sign of an underlying design problem. 查看以下建议:Review following recommendations.

以上示例将更新规则,以便仅允许来自 MY_EXTERNAL_IP_RANGE 范围的入站外部流量。The above example updates the rule to only allow inbound external traffic from the MY_EXTERNAL_IP_RANGE range. 如果将 MY_EXTERNAL_IP_RANGE 替换为内部子网 IP 地址,则流量仅限于群集内部 IP。If you replace MY_EXTERNAL_IP_RANGE with the internal subnet IP address, traffic is restricted to cluster internal IPs only. 这样,来自 Kubernetes 群集外部的客户端不能访问负载均衡器。This will not allow clients from outside of your Kubernetes cluster to access the load balancer.

从基本 SKU 负载均衡器转移到标准 SKUMoving from a basic SKU load balancer to standard SKU

如果现有的群集包含基本 SKU 负载均衡器,则在进行迁移以使用包含标准 SKU 负载均衡器的群集时,需要注意一些重要的行为差异。If you have an existing cluster with the Basic SKU Load Balancer, there are important behavioral differences to note when migrating to use a cluster with the Standard SKU Load Balancer.

例如,假设只能在创建群集时定义群集的 load-balancer-sku 类型,则通过蓝/绿部署迁移群集是常用的做法。For example, making blue/green deployments to migrate clusters is a common practice given the load-balancer-sku type of a cluster can only be defined at cluster create time. 但是,基本 SKU 负载均衡器使用基本 SKU IP 地址,而这些地址与标准 SKU 负载均衡器不兼容,因为这些负载均衡器需要标准 SKU IP 地址 。However, Basic SKU Load Balancers use Basic SKU IP Addresses, which aren't compatible with Standard SKU Load Balancers as they require Standard SKU IP Addresses. 在迁移群集以升级负载均衡器 SKU 时,需要提供一个具有兼容 IP 地址 SKU 的新 IP 地址。When migrating clusters to upgrade Load Balancer SKUs, a new IP address with a compatible IP Address SKU will be required.

有关如何迁移群集的更多注意事项,请访问有关迁移注意事项的文档,以查看迁移时要考虑的重要主题列表。For more considerations on how to migrate clusters, visit our documentation on migration considerations to view a list of important topics to consider when migrating. 以下限制也是在 AKS 中使用标准 SKU 负载均衡器时要注意的重要行为差异。The below limitations are also important behavioral differences to note when using Standard SKU Load Balancers in AKS.

限制Limitations

创建和管理支持标准 SKU 负载均衡器的 AKS 群集时存在以下限制:The following limitations apply when you create and manage AKS clusters that support a load balancer with the Standard SKU:

  • 至少需要指定一个公共 IP 或 IP 前缀来允许 AKS 群集的出口流量。At least one public IP or IP prefix is required for allowing egress traffic from the AKS cluster. 此外,需要使用公共 IP 或 IP 前缀来保持控制平面与代理节点之间的连接,以及保持与旧版 AKS 的兼容性。The public IP or IP prefix is also required to maintain connectivity between the control plane and agent nodes and to maintain compatibility with previous versions of AKS. 可以使用以下选项指定标准 SKU 负载均衡器的公共 IP 或 IP 前缀:You have the following options for specifying public IPs or IP prefixes with a Standard SKU load balancer:
    • 提供自己的公共 IP。Provide your own public IPs.
    • 提供自己的公共 IP 前缀。Provide your own public IP prefixes.
    • 指定最大为 100 的数字,以允许 AKS 群集在其所在的同一个资源组(名称通常以 MC_ 开头)中创建多个标准 SKU 公共 IP。Specify a number up to 100 to allow the AKS cluster to create that many Standard SKU public IPs in the same resource group created as the AKS cluster, which is usually named with MC_ at the beginning. AKS 会将公共 IP 分配到标准 SKU 负载均衡器。AKS assigns the public IP to the Standard SKU load balancer. 默认情况下,如果未指定公共 IP、公共 IP 前缀或 IP 数目,系统会在 AKS 群集所在的同一个资源组中自动创建一个公共 IP。By default, one public IP will automatically be created in the same resource group as the AKS cluster, if no public IP, public IP prefix, or number of IPs is specified. 此外,必须允许公共地址,并避免创建任何会阻止创建 IP 的 Azure Policy。You also must allow public addresses and avoid creating any Azure Policy that bans IP creation.
  • 由 AKS 创建的公共 IP 不能重复使用为自定义自带的公共 IP 地址。A public IP created by AKS cannot be reused as a custom bring your own public IP address. 所有自定义 IP 地址必须由用户创建和管理。All custom IP addresses must be created and managed by the user.
  • 只能在创建 AKS 群集时定义负载均衡器 SKU。Defining the load balancer SKU can only be done when you create an AKS cluster. 创建 AKS 群集后,无法更改负载均衡器 SKU。You can't change the load balancer SKU after an AKS cluster has been created.
  • 在一个群集中只能使用一种类型的负载均衡器 SKU(基本或标准)。You can only use one type of load balancer SKU (Basic or Standard) in a single cluster.
  • 标准 SKU 负载均衡器仅支持标准 SKU IP 地址。 Standard SKU Load Balancers only support Standard SKU IP Addresses.

后续步骤Next steps

Kubernetes 服务文档中详细了解 Kubernetes 服务。Learn more about Kubernetes services at the Kubernetes services documentation.

如需详细了解使用入站流量的内部负载均衡器,请参阅 AKS 内部负载均衡器文档Learn more about using Internal Load Balancer for Inbound traffic at the AKS Internal Load Balancer documentation.