排查出站连接故障Troubleshooting outbound connections failures

本文旨在为 Azure 负载均衡器的出站连接可能出现的常见问题提供解决方法。This article is intended that provide resolutions for common problems can occur with outbound connections from an Azure Load Balancer. 客户遇到的大多数出站连接问题是由于 SNAT 端口耗尽和连接超时导致数据包丢失。Most problems with outbound connectivity that customers experience are due to SNAT port exhaustion and connection timeouts leading to dropped packets. 本文提供了用于缓解上述每个问题的步骤。This article provides steps for mitigating each of these issues.

应对 SNAT (PAT) 端口耗尽问题Managing SNAT (PAT) port exhaustion

用于 PAT临时端口是可用尽的资源,如无公共 IP 地址的独立 VM无公共 IP 地址的负载均衡 VM 中所述。Ephemeral ports used for PAT are an exhaustible resource, as described in Standalone VM without a Public IP address and Load-balanced VM without a Public IP address. 可以根据指南来监视临时端口的使用情况,并将其与当前分配进行比较,以确定 SNAT 耗尽的风险或确认 SNAT 是否耗尽。You can monitor your usage of ephemeral ports and compare with your current allocation to determine the risk of or to confirm SNAT exhaustion using this guide.

如果知道正在启动与同一目标 IP 地址和端口的多个出站 TCP 或 UDP 连接,观察失败的出站连接,或者支持人员通知已耗尽 SNAT 端口(PAT 使用的预先分配临时端口),则有几个常见缓解选项可供选择。If you know that you're initiating many outbound TCP or UDP connections to the same destination IP address and port, and you observe failing outbound connections or are advised by support that you're exhausting SNAT ports (preallocated ephemeral ports used by PAT), you have several general mitigation options. 查看这些选项,确定可用且最适合自己的方案的选项。Review these options and decide what is available and best for your scenario. 一个或多个选项可能有助于管理此方案。It's possible that one or more can help manage this scenario.

如果难以理解出站连接行为,可以使用 IP 堆栈统计 (netstat)。If you are having trouble understanding the outbound connection behavior, you can use IP stack statistics (netstat). 或者使用数据包捕获来观察连接行为。Or it can be helpful to observe connection behaviors by using packet captures. 可以在实例的来宾 OS 中执行这些数据包捕获,或使用网络观察程序来捕获数据包You can perform these packet captures in the guest OS of your instance or use Network Watcher for packet capture.

手动分配 SNAT 端口以最大限度地增加每个 VM 的 SNAT 端口数Manually allocate SNAT ports to maximize SNAT ports per VM

按照预分配端口中的定义,负载均衡器会根据后端的 VM 数量自动分配端口。As defined in preallocated ports, the load balancer will automatically allocate ports based on the number of VMs in the backend. 默认情况下,此操作保守地进行,以确保可伸缩性。By default, this is done conservatively to ensure scalability. 如果已知后端会有的最大 VM 数,则可以在每个出站规则中手动分配 SNAT 端口。If you know the maximum number of VMs you will have in the backend, you can manually allocate SNAT ports in each outbound rule. 例如,如果你知道最多有 10 个VM,则可以为每个 VM 分配 6,400 个 SNAT 端口,而不是默认的 1024 个。For example, if you know you will have a maximum of 10 VMs you can allocate 6,400 SNAT ports per VM rather than the default 1,024.

修改应用程序以重复使用连接Modify the application to reuse connections

在应用程序中重复使用连接,可以降低对用于 SNAT 的临时端口的需求。You can reduce demand for ephemeral ports that are used for SNAT by reusing connections in your application. 连接重用与 HTTP/1.1 之类的协议尤其相关,在这些协议中,默认设置是重复使用连接。Connection reuse is especially relevant for protocols like HTTP/1.1, where connection reuse is the default. 其他使用 HTTP 作为其传输(如 REST)的协议也可以因此受益。And other protocols that use HTTP as their transport (for example, REST) can benefit in turn.

对每个请求来说,重复使用总是优于单独的原子 TCP 连接。Reuse is always better than individual, atomic TCP connections for each request. 重复使用可以带来更高的性能和非常高效的 TCP 事务。Reuse results in more performant, very efficient TCP transactions.

修改应用程序以使用连接池Modify the application to use connection pooling

可以在应用程序中使用连接池方案,其中在一组固定的连接(在可能的情况下,重复使用每一个)上,内部分布请求。You can employ a connection pooling scheme in your application, where requests are internally distributed across a fixed set of connections (each reusing where possible). 此方案会限制正在使用的临时端口的数量,并创建更加可预测的环境。This scheme constrains the number of ephemeral ports in use and creates a more predictable environment. 此方案还可以通过在操作答复上单个连接阻塞时允许多个同时操作,增加请求的吞吐量。This scheme can also increase the throughput of requests by allowing multiple simultaneous operations when a single connection is blocking on the reply of an operation.

连接池可能已经存在于正在使用的框架中,以开发应用程序或应用程序的配置设置。Connection pooling might already exist within the framework that you're using to develop your application or the configuration settings for your application. 可将连接池与连接重复使用相结合。You can combine connection pooling with connection reuse. 多个请求使用指向相同目标 IP 地址和端口的可预测固定数量的端口。Your multiple requests then consume a fixed, predictable number of ports to the same destination IP address and port. 请求还可以从 TCP 事务的高效使用中受益,从而减少延迟和资源利用。The requests also benefit from efficient use of TCP transactions reducing latency and resource utilization. UDP 事务也能从中受益,因为管理 UDP 流数又能避免耗尽情况和管理 SNAT 端口利用率。UDP transactions can also benefit, because managing the number of UDP flows can in turn avoid exhaust conditions and manage the SNAT port utilization.

修改应用程序以使用主动性较低的重试逻辑Modify the application to use less aggressive retry logic

当用于 PAT预先分配临时端口耗尽或应用程序故障发生时,无衰减或回退逻辑的积极重试或暴力重试会使耗尽状况再次发生或一直持续。When preallocated ephemeral ports used for PAT are exhausted or application failures occur, aggressive or brute force retries without decay and backoff logic cause exhaustion to occur or persist. 使用主动性较低的重试逻辑,可以降低对临时端口的需求。You can reduce demand for ephemeral ports by using a less aggressive retry logic.

临时端口有 4 分钟的空闲超时(不可调整)。Ephemeral ports have a 4-minute idle timeout (not adjustable). 如果重试太过积极,则消耗没有机会进行自行清除。If the retries are too aggressive, the exhaustion has no opportunity to clear up on its own. 因此,应用程序停用事务的方式和频率对于设计至关重要。Therefore, considering how--and how often--your application retries transactions is a critical part of the design.

向每个 VM 分配公共 IPAssign a Public IP to each VM

分配公共 IP 地址会将方案更改为 VM 的公共 IPAssigning a Public IP address changes your scenario to Public IP to a VM. 用于各 VM 的公共 IP 的所有临时端口都可供 VM 使用。All ephemeral ports of the public IP that are used for each VM are available to the VM. (与以下方案相反:公共 IP 的临时端口与同相应后端池关联的 VM 的所有临时端口共享)。需要作出一些权衡,比如公共 IP 地址的额外成本和将大量个人 IP 地址列入允许列表所产生的潜在影响。(As opposed to scenarios where ephemeral ports of a public IP are shared with all the VMs associated with the respective backend pool.) There are trade-offs to consider, such as the additional cost of public IP addresses and the potential impact of whitelisting a large number of individual IP addresses.

备注

此选项不适用于 Web 辅助角色。This option is not available for web worker roles.

使用多个前端Use multiple frontends

使用公共标准负载均衡器时,可为出站连接分配多个前端 IP 地址,并将可用的 SNAT 端口数倍增When using public Standard Load Balancer, you assign multiple frontend IP addresses for outbound connections and multiply the number of SNAT ports available. 创建前端 IP 配置、规则和后端池,以触发将 SNAT 编程到前端公共 IP。Create a frontend IP configuration, rule, and backend pool to trigger the programming of SNAT to the public IP of the frontend. 该规则不需要运行,并且运行状况探测不需要成功。The rule does not need to function and a health probe does not need to succeed. 如果对入站连接(而不仅仅是出站连接)也使用多个前端,应使用自定义的运行状况探测来确保可靠性。If you do use multiple frontends for inbound as well (rather than just for outbound), you should use custom health probes well to ensure reliability.

备注

在大多数情况下,SNAT 端口耗尽是设计不当的征兆。In most cases, exhaustion of SNAT ports is a sign of bad design. 在使用更多前端来添加 SNAT 端口之前,请确保已了解端口耗尽的原因。Make sure you understand why you are exhausting ports before using more frontends to add SNAT ports. 否则可能会忽视问题,导致以后出现故障。You may be masking a problem which can lead to failure later.

横向扩展Scale out

预分配端口是根据后端池大小分配的并且分组到各个层中,这样,当某些端口必须重新分配以适应后端池更大的下一个层时,可以最大限度地减少中断。Preallocated ports are assigned based on the backend pool size and grouped into tiers to minimize disruption when some of the ports have to be reallocated to accommodate the next larger backend pool size tier. 有可能可以通过将后端池扩展到给定层的最大大小来提高给定前端的 SNAT 端口利用率。You may have an option to increase the SNAT port utilization for a given frontend by scaling your backend pool to the maximum size for a given tier. 请记住,应用程序需要分配默认端口才能有效横向扩展而不会有耗尽 SNAT 的风险。Keeping in mind the default port allocation is required for the application to scale out efficiently without risk SNAT exhaustion.

例如,后端池中的两个虚拟机对于每个 IP 配置将有 1024 个 SNAT 端口可用,整个部署总共有 2048 个 SNAT 端口。For example, two virtual machines in the backend pool would have 1024 SNAT ports available per IP configuration, allowing a total of 2048 SNAT ports for the deployment. 如果部署已增加到 50 台虚拟机,虽然每台虚拟机的预分配端口数保持不变,但整个部署可以使用 51,200 (50 x 1024) 个 SNAT 端口。If the deployment were to be increased to 50 virtual machines, even though the number of preallocated ports remains constant per virtual machine, a total of 51,200 (50 x 1024) SNAT ports can be used by the deployment. 如果希望横向扩展你的部署,请检查每层的预分配端口的数量,确保将横向扩展规划为各自层的最大容量。If you wish to scale out your deployment, check the number of preallocated ports per tier to make sure you shape your scale-out to the maximum for the respective tier. 在上述示例中,如果选择了横向扩展到 51 个而非 50 个实例,则你将提升到下一个层,最终,每台 VM 的 SNAT 端口数以及端口总数会更少。In the preceding example, if you had chosen to scale out to 51 instead of 50 instances, you would progress to the next tier and end up with fewer SNAT ports per VM as well as in total.

如果横向扩展到后端池较大的下一层,并且需要重新分配已分配端口,则部分出站连接可能会超时。If you scale out to the next larger backend pool size tier, there is potential for some of your outbound connections to time out if allocated ports have to be reallocated. 如果仅使用部分 SNAT 端口,则在后端池较大的下一层中横向扩展无意义。If you are only using some of your SNAT ports, scaling out across the next larger backend pool size is inconsequential. 每次移动到后端池的下一层时,半数现有端口将重新分配。Half the existing ports will be reallocated each time you move to the next backend pool tier. 如果不希望发生此行为,则需要将部署规划为层大小。If you don't want this to take place, you need to shape your deployment to the tier size. 或者确保应用程序可以根据需要进行检测和重试。Or make sure your application can detect and retry as necessary. TCP keepalive 可以帮助检测 SNAT 端口何时由于被重新分配而不再工作。TCP keepalives can assist in detect when SNAT ports no longer function due to being reallocated.

保持 keepalive 重置出站空闲超时Use keepalives to reset the outbound idle timeout

出站连接有 4 分钟的空闲超时。Outbound connections have a 4-minute idle timeout. 此超时可通过出站规则进行调整。This timeout is adjustable via Outbound rules. 还可以使用传输(例如,TCP Keepalives)或应用程序层 Keepalives 刷新空闲流,并在必要时重置此空闲超时。You can also use transport (for example, TCP keepalives) or application-layer keepalives to refresh an idle flow and reset this idle timeout if necessary.

使用 TCP keepalive 时,在连接的一端启用它们就足够了。When using TCP keepalives, it is sufficient to enable them on one side of the connection. 例如,若要重置流的空闲计时器,在服务器端启用它们就足够了,没有必要在两端都启动 TCP keepalive。For example, it is sufficient to enable them on the server side only to reset the idle timer of the flow and it is not necessary for both sides to initiated TCP keepalives. 应用程序层(包括数据库客户端-服务器配置)也存在类似的概念。Similar concepts exist for application layer, including database client-server configurations. 检查服务器端对于特定于应用程序的 keepalive 存在哪些选项。Check the server side for what options exist for application-specific keepalives.