控制 Azure Kubernetes 服务 (AKS) 中群集节点的出口流量Control egress traffic for cluster nodes in Azure Kubernetes Service (AKS)

本文提供了从 Azure Kubernetes 服务 (AKS) 保护出站流量的必要详细信息。This article provides the necessary details that allow you to secure outbound traffic from your Azure Kubernetes Service (AKS). 它包含基本 AKS 部署的群集要求,以及可选加载项和功能的其他要求。It contains the cluster requirements for a base AKS deployment, and additional requirements for optional addons and features. 结尾处有一个示例,介绍如何使用 Azure 防火墙来配置这些要求An example will be provided at the end on how to configure these requirements with Azure Firewall. 但是,可以将此信息应用于任何出站限制方法或设备。However, you can apply this information to any outbound restriction method or appliance.

背景Background

AKS 群集部署在虚拟网络上。AKS clusters are deployed on a virtual network. 此网络可以是托管的(由 AKS 创建),或者是自定义的(由用户预先配置)。This network can be managed (created by AKS) or custom (pre-configured by the user beforehand). 在这两种情况下,群集都对该虚拟网络外部的服务具有出站依赖项(该服务没有入站依赖项)。In either case, the cluster has outbound dependencies on services outside of that virtual network (the service has no inbound dependencies).

为了便于管理和操作,AKS 群集中的节点需要访问特定的端口和完全限定的域名 (FQDN)。For management and operational purposes, nodes in an AKS cluster need to access certain ports and fully qualified domain names (FQDNs). 节点与 API 服务器进行通信,或者下载并安装核心 Kubernetes 群集组件和节点安全更新都需要这些终结点。These endpoints are required for the nodes to communicate with the API server, or to download and install core Kubernetes cluster components and node security updates. 例如,群集需要从 Microsoft 容器注册表 (MCR) 请求基础系统容器映像。For example, the cluster needs to pull base system container images from Microsoft Container Registry (MCR).

AKS 出站依赖项几乎完全是使用 FQDN 定义的,不附带任何静态地址。The AKS outbound dependencies are almost entirely defined with FQDNs, which don't have static addresses behind them. 缺少静态地址意味着无法使用网络安全组锁定来自 AKS 群集的出站流量。The lack of static addresses means that Network Security Groups can't be used to lock down the outbound traffic from an AKS cluster.

默认情况下,AKS 群集具有不受限制的出站(出口)Internet 访问权限。By default, AKS clusters have unrestricted outbound (egress) internet access. 此级别的网络访问权限允许运行的节点和服务根据需要访问外部资源。This level of network access allows nodes and services you run to access external resources as needed. 如果希望限制出口流量,则必须限制可访问的端口和地址数量,才能维护正常的群集维护任务。If you wish to restrict egress traffic, a limited number of ports and addresses must be accessible to maintain healthy cluster maintenance tasks. 保护出站地址的最简单解决方案在于使用可基于域名控制出站流量的防火墙设备。The simplest solution to securing outbound addresses lies in use of a firewall device that can control outbound traffic based on domain names. 例如,Azure 防火墙可以根据目标的 FQDN 限制出站 HTTP 和 HTTPS 流量。Azure Firewall, for example, can restrict outbound HTTP and HTTPS traffic based on the FQDN of the destination. 还可配置首选的防火墙和安全规则,以允许所需的端口和地址。You can also configure your preferred firewall and security rules to allow these required ports and addresses.

重要

本文档仅介绍如何锁定离开 AKS 子网的流量。This document covers only how to lock down the traffic leaving the AKS subnet. 默认情况下,AKS 没有入口需求。AKS has no ingress requirements by default. 不支持使用网络安全组 (NSG) 和防火墙阻止内部子网流量。Blocking internal subnet traffic using network security groups (NSGs) and firewalls is not supported. 若要控制和阻止群集内的流量,请使用网络策略To control and block the traffic within the cluster, use Network Policies.

AKS 群集所需的出站网络规则和 FQDNRequired outbound network rules and FQDNs for AKS clusters

以下网络和 FQDN/应用程序规则为 AKS 群集所必需,若要配置 Azure 防火墙以外的解决方案,可以使用它们。The following network and FQDN/application rules are required for an AKS cluster, you can use them if you wish to configure a solution other than Azure Firewall.

  • IP 地址依赖项适用于非 HTTP/S 流量(TCP 和 UDP 流量)IP Address dependencies are for non-HTTP/S traffic (both TCP and UDP traffic)
  • 可将 FQDN HTTP/HTTPS 终结点放在防火墙设备中。FQDN HTTP/HTTPS endpoints can be placed in your firewall device.
  • 通配符 HTTP/HTTPS 终结点是可以根据许多限定符随 AKS 群集一起变化的依赖项。Wildcard HTTP/HTTPS endpoints are dependencies that can vary with your AKS cluster based on a number of qualifiers.
  • AKS 使用准入控制器将 FQDN 作为环境变量注入 kube-system 和 gatekeeper-system下的所有部署,确保节点和 API 服务器之间的所有系统通信使用 API 服务器 FQDN 而不是 API 服务器 IP。AKS uses an admission controller to inject the FQDN as an environment variable to all deployments under kube-system and gatekeeper-system, that ensures all system communication between nodes and API server uses the API server FQDN and not the API server IP.
  • 如果有需要与 API 服务器通信的应用或解决方案,则必须添加其他网络规则,以允许与 API 服务器 IP 的端口 443 进行 TCP 通信。If you have an app or solution that needs to talk to the API server, you must add an additional network rule to allow TCP communication to port 443 of your API server's IP.
  • 在极少数情况下,如果存在维护操作,则 API 服务器 IP 可能更改。On rare occasions, if there's a maintenance operation your API server IP might change. 始终会提前传达可以更改 API 服务器 IP 的计划内维护操作。Planned maintenance operations that can change the API server IP are always communicated in advance.

Azure 全球的必需网络规则Azure Global required network rules

必需的网络规则和 IP 地址依赖项如下:The required network rules and IP address dependencies are:

目标终结点Destination Endpoint 协议Protocol 端口Port 用途Use
*:1194
OrOr
ServiceTag - AzureCloud.<Region>:1194ServiceTag - AzureCloud.<Region>:1194
OrOr
区域 CIDR - RegionCIDRs:1194Regional CIDRs - RegionCIDRs:1194
OrOr
APIServerIP:1194 (only known after cluster creation)APIServerIP:1194 (only known after cluster creation)
UDPUDP 11941194 用于节点与控制平面之间的隧道安全通信。For tunneled secure communication between the nodes and the control plane.
*:9000
OrOr
ServiceTag - AzureCloud.<Region>:9000ServiceTag - AzureCloud.<Region>:9000
OrOr
区域 CIDR - RegionCIDRs:9000Regional CIDRs - RegionCIDRs:9000
OrOr
APIServerIP:9000 (only known after cluster creation)APIServerIP:9000 (only known after cluster creation)
TCPTCP 90009000 用于节点与控制平面之间的隧道安全通信。For tunneled secure communication between the nodes and the control plane.
*:123ntp.ubuntu.com:123(如果使用 Azure 防火墙网络规则) *:123 or ntp.ubuntu.com:123 (if using Azure Firewall network rules) UDPUDP 123123 在 Linux 节点上进行网络时间协议 (NTP) 时间同步时需要。Required for Network Time Protocol (NTP) time synchronization on Linux nodes.
CustomDNSIP:53 (if using custom DNS servers)CustomDNSIP:53 (if using custom DNS servers) UDPUDP 5353 如果使用的是自定义 DNS 服务器,必须确保群集节点可以访问这些服务器。If you're using custom DNS servers, you must ensure they're accessible by the cluster nodes.
APIServerIP:443 (if running pods/deployments that access the API Server)APIServerIP:443 (if running pods/deployments that access the API Server) TCPTCP 443443 运行访问 API 服务器的 Pod/部署时需要,这些 Pod/部署将使用 API IP。Required if running pods/deployments that access the API Server, those pods/deployments would use the API IP.

Azure 全球的必需 FQDN/应用程序规则Azure Global required FQDN / application rules

必须具有以下 FQDN/应用程序规则:The following FQDN / application rules are required:

目标 FQDNDestination FQDN 端口Port 用途Use
*.hcp.<location>.azmk8s.io HTTPS:443 Node <-> API 服务器通信时需要。Required for Node <-> API server communication. 将 <location> 替换为部署 AKS 群集的区域。Replace <location> with the region where your AKS cluster is deployed.
mcr.microsoft.com HTTPS:443 访问 Microsoft 容器注册表 (MCR) 中的映像时需要。Required to access images in Microsoft Container Registry (MCR). 此注册表包含第一方映像/图表(例如 coreDNS 等)。This registry contains first-party images/charts (for example, coreDNS, etc.). 这些映像是正确创建和正常运行群集所必需的,包括缩放和升级操作。These images are required for the correct creation and functioning of the cluster, including scale and upgrade operations.
*.cdn.mscr.io HTTPS:443 对于 Azure 内容分发网络 (CDN) 支持的 MCR 存储是必需的。Required for MCR storage backed by the Azure Content Delivery Network (CDN).
*.data.mcr.microsoft.com HTTPS:443 对于 Azure 内容分发网络 (CDN) 支持的 MCR 存储是必需的。Required for MCR storage backed by the Azure content delivery network (CDN).
management.azure.com HTTPS:443 对于针对 Azure API 的 Kubernetes 操作是必需的。Required for Kubernetes operations against the Azure API.
login.microsoftonline.com HTTPS:443 对于 Azure Active Directory 身份验证是必需的。Required for Azure Active Directory authentication.
packages.microsoft.com HTTPS:443 Microsoft 包存储库使用此地址缓存 apt-get 操作。This address is the Microsoft packages repository used for cached apt-get operations. 示例包包括 Moby、PowerShell 和 Azure CLI。Example packages include Moby, PowerShell, and Azure CLI.
acs-mirror.azureedge.net HTTPS:443 此地址用于下载和安装所需二进制文件(如 kubenet 和 Azure CNI)所需的存储库。This address is for the repository required to download and install required binaries like kubenet and Azure CNI.

Azure 中国世纪互联的必需网络规则Azure China 21Vianet required network rules

必需的网络规则和 IP 地址依赖项如下:The required network rules and IP address dependencies are:

目标终结点Destination Endpoint 协议Protocol 端口Port 用途Use
*:1194
OrOr
ServiceTag - AzureCloud.Region:1194ServiceTag - AzureCloud.Region:1194
OrOr
区域 CIDR - RegionCIDRs:1194Regional CIDRs - RegionCIDRs:1194
OrOr
APIServerIP:1194 (only known after cluster creation)APIServerIP:1194 (only known after cluster creation)
UDPUDP 11941194 用于节点与控制平面之间的隧道安全通信。For tunneled secure communication between the nodes and the control plane.
*:9000
OrOr
ServiceTag - AzureCloud.<Region>:9000ServiceTag - AzureCloud.<Region>:9000
OrOr
区域 CIDR - RegionCIDRs:9000Regional CIDRs - RegionCIDRs:9000
OrOr
APIServerIP:9000 (only known after cluster creation)APIServerIP:9000 (only known after cluster creation)
TCPTCP 90009000 用于节点与控制平面之间的隧道安全通信。For tunneled secure communication between the nodes and the control plane.
*:22
OrOr
ServiceTag - AzureCloud.<Region>:22ServiceTag - AzureCloud.<Region>:22
OrOr
区域 CIDR - RegionCIDRs:22Regional CIDRs - RegionCIDRs:22
OrOr
APIServerIP:22 (only known after cluster creation)APIServerIP:22 (only known after cluster creation)
TCPTCP 2222 用于节点与控制平面之间的隧道安全通信。For tunneled secure communication between the nodes and the control plane.
*:123ntp.ubuntu.com:123(如果使用 Azure 防火墙网络规则) *:123 or ntp.ubuntu.com:123 (if using Azure Firewall network rules) UDPUDP 123123 在 Linux 节点上进行网络时间协议 (NTP) 时间同步时需要。Required for Network Time Protocol (NTP) time synchronization on Linux nodes.
CustomDNSIP:53 (if using custom DNS servers)CustomDNSIP:53 (if using custom DNS servers) UDPUDP 5353 如果使用的是自定义 DNS 服务器,必须确保群集节点可以访问这些服务器。If you're using custom DNS servers, you must ensure they're accessible by the cluster nodes.
APIServerIP:443 (if running pods/deployments that access the API Server)APIServerIP:443 (if running pods/deployments that access the API Server) TCPTCP 443443 运行访问 API 服务器的 Pod/部署时需要,这些 Pod/部署将使用 API IP。Required if running pods/deployments that access the API Server, those pod/deployments would use the API IP.

Azure 中国世纪互联的必需 FQDN/应用程序规则Azure China 21Vianet required FQDN / application rules

必须具有以下 FQDN/应用程序规则:The following FQDN / application rules are required:

目标 FQDNDestination FQDN 端口Port 用途Use
*.hcp.<location>.cx.prod.service.azk8s.cn HTTPS:443 Node <-> API 服务器通信时需要。Required for Node <-> API server communication. 将 <location> 替换为部署 AKS 群集的区域。Replace <location> with the region where your AKS cluster is deployed.
*.tun.<location>.cx.prod.service.azk8s.cn HTTPS:443 Node <-> API 服务器通信时需要。Required for Node <-> API server communication. 将 <location> 替换为部署 AKS 群集的区域。Replace <location> with the region where your AKS cluster is deployed.
mcr.microsoft.com HTTPS:443 访问 Microsoft 容器注册表 (MCR) 中的映像时需要。Required to access images in Microsoft Container Registry (MCR). 此注册表包含第一方映像/图表(例如 coreDNS 等)。This registry contains first-party images/charts (for example, coreDNS, etc.). 这些映像是正确创建和正常运行群集所必需的,包括缩放和升级操作。These images are required for the correct creation and functioning of the cluster, including scale and upgrade operations.
*.cdn.mscr.io HTTPS:443 对于 Azure 内容分发网络 (CDN) 支持的 MCR 存储是必需的。Required for MCR storage backed by the Azure Content Delivery Network (CDN).
.data.mcr.microsoft.com HTTPS:443 对于 Azure 内容分发网络 (CDN) 支持的 MCR 存储是必需的。Required for MCR storage backed by the Azure Content Delivery Network (CDN).
management.chinacloudapi.cn HTTPS:443 对于针对 Azure API 的 Kubernetes 操作是必需的。Required for Kubernetes operations against the Azure API.
login.chinacloudapi.cn HTTPS:443 对于 Azure Active Directory 身份验证是必需的。Required for Azure Active Directory authentication.
packages.microsoft.com HTTPS:443 Microsoft 包存储库使用此地址缓存 apt-get 操作。This address is the Microsoft packages repository used for cached apt-get operations. 示例包包括 Moby、PowerShell 和 Azure CLI。Example packages include Moby, PowerShell, and Azure CLI.
*.azk8s.cn HTTPS:443 此地址用于下载和安装所需二进制文件(如 kubenet 和 Azure CNI)所需的存储库。This address is for the repository required to download and install required binaries like kubenet and Azure CNI.

Azure 美国政府的必需网络规则Azure US Government required network rules

必需的网络规则和 IP 地址依赖项如下:The required network rules and IP address dependencies are:

目标终结点Destination Endpoint 协议Protocol 端口Port 用途Use
*:1194
OrOr
ServiceTag - AzureCloud.<Region>:1194ServiceTag - AzureCloud.<Region>:1194
OrOr
区域 CIDR - RegionCIDRs:1194Regional CIDRs - RegionCIDRs:1194
OrOr
APIServerIP:1194 (only known after cluster creation)APIServerIP:1194 (only known after cluster creation)
UDPUDP 11941194 用于节点与控制平面之间的隧道安全通信。For tunneled secure communication between the nodes and the control plane.
*:9000
OrOr
ServiceTag - AzureCloud.<Region>:9000ServiceTag - AzureCloud.<Region>:9000
OrOr
区域 CIDR - RegionCIDRs:9000Regional CIDRs - RegionCIDRs:9000
OrOr
APIServerIP:9000 (only known after cluster creation)APIServerIP:9000 (only known after cluster creation)
TCPTCP 90009000 用于节点与控制平面之间的隧道安全通信。For tunneled secure communication between the nodes and the control plane.
*:123ntp.ubuntu.com:123(如果使用 Azure 防火墙网络规则) *:123 or ntp.ubuntu.com:123 (if using Azure Firewall network rules) UDPUDP 123123 在 Linux 节点上进行网络时间协议 (NTP) 时间同步时需要。Required for Network Time Protocol (NTP) time synchronization on Linux nodes.
CustomDNSIP:53 (if using custom DNS servers)CustomDNSIP:53 (if using custom DNS servers) UDPUDP 5353 如果使用的是自定义 DNS 服务器,必须确保群集节点可以访问这些服务器。If you're using custom DNS servers, you must ensure they're accessible by the cluster nodes.
APIServerIP:443 (if running pods/deployments that access the API Server)APIServerIP:443 (if running pods/deployments that access the API Server) TCPTCP 443443 运行访问 API 服务器的 Pod/部署时需要,这些 Pod/部署将使用 API IP。Required if running pods/deployments that access the API Server, those pods/deployments would use the API IP.

Azure 美国政府的必需 FQDN/应用程序规则Azure US Government required FQDN / application rules

必须具有以下 FQDN/应用程序规则:The following FQDN / application rules are required:

目标 FQDNDestination FQDN 端口Port 用途Use
*.hcp.<location>.cx.aks.containerservice.azure.us HTTPS:443 Node <-> API 服务器通信时需要。Required for Node <-> API server communication. 将 <location> 替换为部署 AKS 群集的区域。Replace <location> with the region where your AKS cluster is deployed.
mcr.microsoft.com HTTPS:443 访问 Microsoft 容器注册表 (MCR) 中的映像时需要。Required to access images in Microsoft Container Registry (MCR). 此注册表包含第一方映像/图表(例如 coreDNS 等)。This registry contains first-party images/charts (for example, coreDNS, etc.). 这些映像是正确创建和正常运行群集所必需的,包括缩放和升级操作。These images are required for the correct creation and functioning of the cluster, including scale and upgrade operations.
*.cdn.mscr.io HTTPS:443 对于 Azure 内容分发网络 (CDN) 支持的 MCR 存储是必需的。Required for MCR storage backed by the Azure Content Delivery Network (CDN).
*.data.mcr.microsoft.com HTTPS:443 对于 Azure 内容分发网络 (CDN) 支持的 MCR 存储是必需的。Required for MCR storage backed by the Azure content delivery network (CDN).
management.usgovcloudapi.net HTTPS:443 对于针对 Azure API 的 Kubernetes 操作是必需的。Required for Kubernetes operations against the Azure API.
login.microsoftonline.us HTTPS:443 对于 Azure Active Directory 身份验证是必需的。Required for Azure Active Directory authentication.
packages.microsoft.com HTTPS:443 Microsoft 包存储库使用此地址缓存 apt-get 操作。This address is the Microsoft packages repository used for cached apt-get operations. 示例包包括 Moby、PowerShell 和 Azure CLI。Example packages include Moby, PowerShell, and Azure CLI.
acs-mirror.azureedge.net HTTPS:443 此地址用于安装所需二进制文件(如 kubenet 和 Azure CNI)所需的存储库。This address is for the repository required to install required binaries like kubenet and Azure CNI.

以下 FQDN/应用程序规则是可选的,但建议为 AKS 群集使用:The following FQDN / application rules are optional but recommended for AKS clusters:

目标 FQDNDestination FQDN 端口Port 用途Use
security.ubuntu.comazure.archive.ubuntu.comchangelogs.ubuntu.comsecurity.ubuntu.com, azure.archive.ubuntu.com, changelogs.ubuntu.com HTTP:80 此地址允许 Linux 群集节点下载必需的安全修补程序和更新。This address lets the Linux cluster nodes download the required security patches and updates.

如果选择阻止/不允许这些 FQDN,则仅当进行节点映像升级群集升级时,节点才会接收 OS 更新。If you choose to block/not allow these FQDNs, the nodes will only receive OS updates when you do a node image upgrade or cluster upgrade.

启用 GPU 的 AKS 群集GPU enabled AKS clusters

必需的 FQDN/应用程序规则Required FQDN / application rules

启用 GPU 的 AKS 群集需要以下 FQDN/应用程序规则:The following FQDN / application rules are required for AKS clusters that have GPU enabled:

目标 FQDNDestination FQDN 端口Port 用途Use
nvidia.github.io HTTPS:443 此地址用于在基于 GPU 的节点上进行正确的驱动程序安装和操作。This address is used for correct driver installation and operation on GPU-based nodes.
us.download.nvidia.com HTTPS:443 此地址用于在基于 GPU 的节点上进行正确的驱动程序安装和操作。This address is used for correct driver installation and operation on GPU-based nodes.
apt.dockerproject.org HTTPS:443 此地址用于在基于 GPU 的节点上进行正确的驱动程序安装和操作。This address is used for correct driver installation and operation on GPU-based nodes.

基于 Windows Server 的节点池Windows Server based node pools

必需的 FQDN/应用程序规则Required FQDN / application rules

使用基于 Windows Server 的节点池需要以下 FQDN/应用程序规则:The following FQDN / application rules are required for using Windows Server based node pools:

目标 FQDNDestination FQDN 端口Port 用途Use
onegetcdn.azureedge.net, go.microsoft.com HTTPS:443 安装与 windows 相关的二进制文件To install windows-related binaries
*.mp.microsoft.com, www.msftconnecttest.com, ctldl.windowsupdate.com HTTP:80 安装与 windows 相关的二进制文件To install windows-related binaries

AKS 加载项和集成AKS addons and integrations

用于容器的 Azure MonitorAzure Monitor for containers

有两个选项可以提供对用于容器的 Azure Monitor 的访问,你可以允许 Azure Monitor ServiceTag或者提供对必需 FQDN/应用程序规则的访问权限。There are two options to provide access to Azure Monitor for containers, you may allow the Azure Monitor ServiceTag or provide access to the required FQDN/Application Rules.

必需的网络规则Required network rules

必须具有以下 FQDN/应用程序规则:The following FQDN / application rules are required:

目标终结点Destination Endpoint 协议Protocol 端口Port 用途Use
ServiceTag - AzureMonitor:443ServiceTag - AzureMonitor:443 TCPTCP 443443 此终结点用于将指标数据和日志发送到 Azure Monitor 和 Log Analytics。This endpoint is used to send metrics data and logs to Azure Monitor and Log Analytics.

必需的 FQDN/应用程序规则Required FQDN / application rules

启用了用于容器的 Azure Monitor 的 AKS 群集需要以下 FQDN/应用程序规则:The following FQDN / application rules are required for AKS clusters that have the Azure Monitor for containers enabled:

FQDNFQDN 端口Port 用途Use
dc.services.visualstudio.comdc.services.visualstudio.com HTTPS:443 此终结点适用于使用 Azure Monitor 的指标和监视遥测。This endpoint is used for metrics and monitoring telemetry using Azure Monitor.
*.ods.opinsights.azure.com*.ods.opinsights.azure.com HTTPS:443 Azure Monitor 使用此终结点来引入日志分析数据。This endpoint is used by Azure Monitor for ingesting log analytics data.
*.oms.opinsights.azure.com*.oms.opinsights.azure.com HTTPS:443 此终结点由 omsagent 使用,用于对日志分析服务进行身份验证。This endpoint is used by omsagent, which is used to authenticate the log analytics service.
*.monitoring.azure.com*.monitoring.azure.com HTTPS:443 此终结点用于将指标数据发送到 Azure Monitor。This endpoint is used to send metrics data to Azure Monitor.

Azure Dev SpacesAzure Dev Spaces

更新防火墙或安全配置,以允许去到/来自以下所有 FQDN 和 [Azure Dev Spaces 基础结构服务][dev-spaces-service-tags] 的网络流量。Update your firewall or security configuration to allow network traffic to and from the all of the below FQDNs and [Azure Dev Spaces infrastructure services][dev-spaces-service-tags].

必需的网络规则Required network rules

目标终结点Destination Endpoint 协议Protocol 端口Port 用途Use
ServiceTag - AzureDevSpacesServiceTag - AzureDevSpaces TCPTCP 443443 此终结点用于将指标数据和日志发送到 Azure Monitor 和 Log Analytics。This endpoint is used to send metrics data and logs to Azure Monitor and Log Analytics.

必需的 FQDN/应用程序规则Required FQDN / application rules

启用了 Azure Dev Spaces 的 AKS 群集需要以下 FQDN/应用程序规则:The following FQDN / application rules are required for AKS clusters that have the Azure Dev Spaces enabled:

FQDNFQDN 端口Port 用途Use
cloudflare.docker.com HTTPS:443 此地址用于请求 linux alpine 和其他 Azure Dev Spaces 映像This address is used to pull linux alpine and other Azure Dev Spaces images
gcr.azk8s.cn HTTPS:443 此地址用于请求 helm/tiller 映像This address is used to pull helm/tiller images
storage.googleapis.com HTTPS:443 此地址用于请求 helm/tiller 映像This address is used to pull helm/tiller images

Azure Policy(预览)Azure Policy (preview)

注意

下面的某些功能处于预览阶段。Some of the features below are in preview. 随着此功能向公共预览版和未来发布阶段发展,本文中的建议可能会有所不同。The suggestions in this article are subject to change as the feature moves to public preview and future release stages.

必需的 FQDN/应用程序规则Required FQDN / application rules

启用了 Azure Policy 的 AKS 群集需要以下 FQDN/应用程序规则。The following FQDN / application rules are required for AKS clusters that have the Azure Policy enabled.

FQDNFQDN 端口Port 用途Use
gov-prod-policy-data.trafficmanager.net HTTPS:443 此地址用于正确操作 Azure Policy。This address is used for correct operation of Azure Policy. (AKS 中目前为预览版)(currently in preview in AKS)
raw.githubusercontent.com HTTPS:443 此地址用于从 GitHub 请求内置策略,以确保正确操作 Azure Policy。This address is used to pull the built-in policies from GitHub to ensure correct operation of Azure Policy. (AKS 中目前为预览版)(currently in preview in AKS)
dc.services.visualstudio.com HTTPS:443 Azure Policy 加载项,用于向应用程序见解终结点发送遥测数据。Azure Policy add-on that sends telemetry data to applications insights endpoint.

使用 Azure 防火墙限制出口流量Restrict egress traffic using Azure firewall

Azure 防火墙提供 Azure Kubernetes 服务 (AzureKubernetesService) FQDN 标记以简化此配置。Azure Firewall provides an Azure Kubernetes Service (AzureKubernetesService) FQDN Tag to simplify this configuration.

备注

FQDN 标记包含上面列出的所有 FQDN,并自动保持最新。The FQDN tag contains all the FQDNs listed above and is kept automatically up to date.

对于生产方案,建议在 Azure 防火墙上至少具有 20 个前端 IP,以避免出现 SNAT 端口耗尽问题。We recommend having a minimum of 20 Frontend IPs on the Azure Firewall for production scenarios to avoid incurring in SNAT port exhaustion issues.

下面是部署的示例体系结构:Below is an example architecture of the deployment:

锁定的拓扑

  • 强制公共入口流量流经防火墙筛选器Public Ingress is forced to flow through firewall filters
    • AKS 代理节点隔离在专用子网中。AKS agent nodes are isolated in a dedicated subnet.
    • Azure 防火墙部署在自己的子网中。Azure Firewall is deployed in its own subnet.
    • DNAT 规则将 FW 公共 IP 转换为 LB 前端 IP。A DNAT rule translates the FW public IP into the LB frontend IP.
  • 出站请求从代理节点启动并使用用户定义的路由发送到 Azure 防火墙内部 IPOutbound requests start from agent nodes to the Azure Firewall internal IP using a user-defined route
    • 来自 AKS 代理节点的请求遵循 AKS 群集所部署到的子网中已放置的 UDR。Requests from AKS agent nodes follow a UDR that has been placed on the subnet the AKS cluster was deployed into.
    • Azure 防火墙通过公共 IP 前端将流量传出虚拟网络Azure Firewall egresses out of the virtual network from a public IP frontend
    • 对公共 Internet 或其他 Azure 服务的访问流量会流入和流出防火墙前端 IP 地址Access to the public internet or other Azure services flows to and from the firewall frontend IP address
    • 根据需要,API 服务器授权的 IP 范围(包括防火墙公共前端 IP 地址)保护对 AKS 控制平面的访问。Optionally, access to the AKS control plane is protected by API server Authorized IP ranges, which includes the firewall public frontend IP address.
  • 内部流量Internal Traffic

以下步骤使用 Azure 防火墙的 AzureKubernetesService FQDN 标记限制来自 AKS 群集的出站流量,并提供如何通过防火墙配置公用入站流量的示例。The below steps make use of Azure Firewall's AzureKubernetesService FQDN tag to restrict the outbound traffic from the AKS cluster and provide an example how to configure public inbound traffic via the firewall.

通过环境变量设置配置Set configuration via environment variables

定义创建资源时要使用的一组环境变量。Define a set of environment variables to be used in resource creations.

PREFIX="aks-egress"
RG="${PREFIX}-rg"
LOC="chinaeast2"
PLUGIN=azure
AKSNAME="${PREFIX}"
VNET_NAME="${PREFIX}-vnet"
AKSSUBNET_NAME="aks-subnet"
# DO NOT CHANGE FWSUBNET_NAME - This is currently a requirement for Azure Firewall.
FWSUBNET_NAME="AzureFirewallSubnet"
FWNAME="${PREFIX}-fw"
FWPUBLICIP_NAME="${PREFIX}-fwpublicip"
FWIPCONFIG_NAME="${PREFIX}-fwconfig"
FWROUTE_TABLE_NAME="${PREFIX}-fwrt"
FWROUTE_NAME="${PREFIX}-fwrn"
FWROUTE_NAME_INTERNET="${PREFIX}-fwinternet"

创建包含多个子网的虚拟网络Create a virtual network with multiple subnets

预配包含两个单独子网的虚拟网络,其中一个子网用于群集,一个子网用于防火墙。Provision a virtual network with two separate subnets, one for the cluster, one for the firewall. 还可以选择为内部服务入口创建一个。Optionally you could also create one for internal service ingress.

空网络拓扑

创建一个资源组来存放所有资源。Create a resource group to hold all of the resources.

# Create Resource Group

az group create --name $RG --location $LOC

创建具有两个子网的虚拟网络来托管 AKS 群集和 Azure 防火墙。Create a virtual network with two subnets to host the AKS cluster and the Azure Firewall. 每个虚拟网络都具有自己的子网。Each will have their own subnet. 让我们从 AKS 网络开始。Let's start with the AKS network.

# Dedicated virtual network with AKS subnet

az network vnet create \
    --resource-group $RG \
    --name $VNET_NAME \
    --address-prefixes 10.42.0.0/16 \
    --subnet-name $AKSSUBNET_NAME \
    --subnet-prefix 10.42.1.0/24

# Dedicated subnet for Azure Firewall (Firewall name cannot be changed)

az network vnet subnet create \
    --resource-group $RG \
    --vnet-name $VNET_NAME \
    --name $FWSUBNET_NAME \
    --address-prefix 10.42.2.0/24

使用 UDR 来创建和设置 Azure 防火墙Create and set up an Azure Firewall with a UDR

必须配置 Azure 防火墙入站和出站规则。Azure Firewall inbound and outbound rules must be configured. 防火墙的主要用途是使组织能够针对传入和传出 AKS 群集的流量配置精细的规则。The main purpose of the firewall is to enable organizations to configure granular ingress and egress traffic rules into and out of the AKS Cluster.

防火墙和 UDR

重要

如果群集或应用程序创建众多定向到相同目标或目标子集的出站连接,则可能需要更多的防火墙前端 IP 来避免用尽每个前端 IP 的端口。If your cluster or application creates a large number of outbound connections directed to the same or small subset of destinations, you might require more firewall frontend IPs to avoid maxing out the ports per frontend IP. 有关如何创建具有多个 IP 的 Azure 防火墙的详细信息,请参阅此处For more information on how to create an Azure firewall with multiple IPs, see here

创建将用作 Azure 防火墙前端地址的标准 SKU 公共 IP 资源。Create a standard SKU public IP resource that will be used as the Azure Firewall frontend address.

az network public-ip create -g $RG -n $FWPUBLICIP_NAME -l $LOC --sku "Standard"

注册预览版 CLI 扩展以创建 Azure 防火墙。Register the preview cli-extension to create an Azure Firewall.

# Install Azure Firewall preview CLI extension

az extension add --name azure-firewall

# Deploy Azure Firewall

az network firewall create -g $RG -n $FWNAME -l $LOC --enable-dns-proxy true

现在,可将前面创建的 IP 地址分配到防火墙前端。The IP address created earlier can now be assigned to the firewall frontend.

备注

设置 Azure 防火墙的公共 IP 地址可能需要几分钟时间。Set up of the public IP address to the Azure Firewall may take a few minutes. 若要对网络规则使用 FQDN,需要启用 DNS 代理。如果启用,防火墙将侦听端口 53,并将 DNS 请求转发到上面指定的 DNS 服务器。To leverage FQDN on network rules we need DNS proxy enabled, when enabled the firewall will listen on port 53 and will forward DNS requests to the DNS server specified above. 这将允许防火墙自动转换该 FQDN。This will allow the firewall to translate that FQDN automatically.

# Configure Firewall IP Config

az network firewall ip-config create -g $RG -f $FWNAME -n $FWIPCONFIG_NAME --public-ip-address $FWPUBLICIP_NAME --vnet-name $VNET_NAME

当前面的命令成功时,保存防火墙前端 IP 地址以便稍后进行配置。When the previous command has succeeded, save the firewall frontend IP address for configuration later.

# Capture Firewall IP Address for Later Use

FWPUBLIC_IP=$(az network public-ip show -g $RG -n $FWPUBLICIP_NAME --query "ipAddress" -o tsv)
FWPRIVATE_IP=$(az network firewall show -g $RG -n $FWNAME --query "ipConfigurations[0].privateIpAddress" -o tsv)

备注

如果通过授权 IP 地址范围安全访问 AKS API 服务器,需要将防火墙公共 IP 添加到授权的 IP 范围。If you use secure access to the AKS API server with authorized IP address ranges, you need to add the firewall public IP into the authorized IP range.

创建包含 Azure 防火墙跃点的 UDRCreate a UDR with a hop to Azure Firewall

Azure 自动在 Azure 子网、虚拟网络与本地网络之间路由流量。Azure automatically routes traffic between Azure subnets, virtual networks, and on-premises networks. 若要更改 Azure 的任何默认路由,可以创建一个路由表。If you want to change any of Azure's default routing, you do so by creating a route table.

创建一个要与给定子网关联的空路由表。Create an empty route table to be associated with a given subnet. 该路由表将下一跃点定义为前面创建的 Azure 防火墙。The route table will define the next hop as the Azure Firewall created above. 每个子网可以有一个与之关联的路由表,也可以没有。Each subnet can have zero or one route table associated to it.

# Create UDR and add a route for Azure Firewall

az network route-table create -g $RG -l -$LOC --name $FWROUTE_TABLE_NAME
az network route-table route create -g $RG --name $FWROUTE_NAME --route-table-name $FWROUTE_TABLE_NAME --address-prefix 0.0.0.0/0 --next-hop-type VirtualAppliance --next-hop-ip-address $FWPRIVATE_IP --subscription $SUBID
az network route-table route create -g $RG --name $FWROUTE_NAME_INTERNET --route-table-name $FWROUTE_TABLE_NAME --address-prefix $FWPUBLIC_IP/32 --next-hop-type Internet

请参阅虚拟网络路由表文档,了解如何替代 Azure 的默认系统路由或者在子网的路由表中添加更多路由。See virtual network route table documentation about how you can override Azure's default system routes or add additional routes to a subnet's route table.

添加防火墙规则Adding firewall rules

下面是可用于在防火墙上配置的三个网络规则,可能需要根据部署来调整这些规则。Below are three network rules you can use to configure on your firewall, you may need to adapt these rules based on your deployment. 第一个规则允许通过 TCP 访问端口 9000。The first rule allows access to port 9000 via TCP. 第二个规则允许通过 UDP 访问端口 1194 和 123(如果部署到 Azure 中国世纪互联,可能需要更多)。The second rule allows access to port 1194 and 123 via UDP (if you're deploying to Azure China 21Vianet, you might require more). 这两个规则仅允许目标为当前所用 Azure 区域 CIDR 的流量,本例中为中国东部 2。Both these rules will only allow traffic destined to the Azure Region CIDR that we're using, in this case China East 2. 最后添加第三个网络规则,通过 UDP 打开端口 123 转到 ntp.ubuntu.com FQDN(将 FQDN 添加为网络规则是 Azure 防火墙的一项特定功能,使用自己的选项时需要对其进行调整)。Finally, we'll add a third network rule opening port 123 to ntp.ubuntu.com FQDN via UDP (adding an FQDN as a network rule is one of the specific features of Azure Firewall, and you'll need to adapt it when using your own options).

设置网络规则后,还将使用 AzureKubernetesService 添加应用程序规则,该规则涵盖可通过 TCP 端口 443 和端口 80 访问的所有必需 FQDN。After setting the network rules, we'll also add an application rule using the AzureKubernetesService that covers all needed FQDNs accessible through TCP port 443 and port 80.

# Add FW Network Rules

az network firewall network-rule create -g $RG -f $FWNAME --collection-name 'aksfwnr' -n 'apiudp' --protocols 'UDP' --source-addresses '*' --destination-addresses "AzureCloud.$LOC" --destination-ports 1194 --action allow --priority 100
az network firewall network-rule create -g $RG -f $FWNAME --collection-name 'aksfwnr' -n 'apitcp' --protocols 'TCP' --source-addresses '*' --destination-addresses "AzureCloud.$LOC" --destination-ports 9000
az network firewall network-rule create -g $RG -f $FWNAME --collection-name 'aksfwnr' -n 'time' --protocols 'UDP' --source-addresses '*' --destination-fqdns 'ntp.ubuntu.com' --destination-ports 123

# Add FW Application Rules

az network firewall application-rule create -g $RG -f $FWNAME --collection-name 'aksfwar' -n 'fqdn' --source-addresses '*' --protocols 'http=80' 'https=443' --fqdn-tags "AzureKubernetesService" --action allow --priority 100

请参阅 Azure 防火墙文档来详细了解 Azure 防火墙服务。See Azure Firewall documentation to learn more about the Azure Firewall service.

将路由表关联到 AKSAssociate the route table to AKS

若要将群集与防火墙相关联,群集子网的专用子网必须引用前面创建的路由表。To associate the cluster with the firewall, the dedicated subnet for the cluster's subnet must reference the route table created above. 可以通过向包含群集和防火墙的虚拟网络发出更新群集子网路由表的命令来执行关联。Association can be done by issuing a command to the virtual network holding both the cluster and firewall to update the route table of the cluster's subnet.

# Associate route table with next hop to Firewall to the AKS subnet

az network vnet subnet update -g $RG --vnet-name $VNET_NAME --name $AKSSUBNET_NAME --route-table $FWROUTE_TABLE_NAME

将出站类型为 UDR 的 AKS 部署到现有网络Deploy AKS with outbound type of UDR to the existing network

现在,可将 AKS 群集部署到现有的虚拟网络。Now an AKS cluster can be deployed into the existing virtual network. 还将使用出站类型userDefinedRouting,此功能确保通过防火墙强制执行任何出站流量,并且不存在其他传出路径(默认情况下,可以使用负载均衡器出站类型)。We'll also use outbound type userDefinedRouting, this feature ensures any outbound traffic will be forced through the firewall and no other egress paths will exist (by default the Load Balancer outbound type could be used).

aks-deploy

创建有权在现有虚拟网络中进行预配的服务主体Create a service principal with access to provision inside the existing virtual network

AKS 使用服务主体来创建群集资源。A service principal is used by AKS to create cluster resources. 创建时传递的服务主体用于创建底层 AKS 资源,例如 AKS 使用的存储资源、IP 和负载均衡器(还可以改为使用托管标识)。The service principal that is passed at create time is used to create underlying AKS resources such as Storage resources, IPs, and Load Balancers used by AKS (you may also use a managed identity instead). 如果未授予以下适当的权限,则无法预配 AKS 群集。If not granted the appropriate permissions below, you won't be able to provision the AKS Cluster.

# Create SP and Assign Permission to Virtual Network

az ad sp create-for-rbac -n "${PREFIX}sp" --skip-assignment

现在,请将下面的 APPIDPASSWORD 替换为前一命令输出自动生成的服务主体 appid 和服务主体密码。Now replace the APPID and PASSWORD below with the service principal appid and service principal password autogenerated by the previous command output. 将引用 VNET 资源 ID 来向服务主体授予权限,使 AKS 能够将资源部署到其中。We'll reference the VNET resource ID to grant the permissions to the service principal so AKS can deploy resources into it.

APPID="<SERVICE_PRINCIPAL_APPID_GOES_HERE>"
PASSWORD="<SERVICEPRINCIPAL_PASSWORD_GOES_HERE>"
VNETID=$(az network vnet show -g $RG --name $VNET_NAME --query id -o tsv)

# Assign SP Permission to VNET

az role assignment create --assignee $APPID --scope $VNETID --role "Network Contributor"

可以在此处查看所需的详细权限。You can check the detailed permissions that are required here.

备注

如果使用的是 kubenet 网络插件,则需要为预先创建的路由表提供 AKS 服务主体或托管标识权限,因为 kubenet 需要路由表来添加必要传递规则。If you're using the kubenet network plugin, you'll need to give the AKS service principal or managed identity permissions to the pre-created route table, since kubenet requires a route table to add neccesary routing rules.

RTID=$(az network route-table show -g $RG -n $FWROUTE_TABLE_NAME --query id -o tsv)
az role assignment create --assignee $APPID --scope $RTID --role "Network Contributor"

部署 AKSDeploy AKS

最后,可将 AKS 群集部署到专用于群集的现有子网中。Finally, the AKS cluster can be deployed into the existing subnet we've dedicated for the cluster. 要部署到的目标子网是使用环境变量 ($SUBNETID) 定义的。The target subnet to be deployed into is defined with the environment variable, $SUBNETID. 在前面的步骤中,我们未定义 $SUBNETID 变量。We didn't define the $SUBNETID variable in the previous steps. 若要设置子网 ID 的值,可使用以下命令:To set the value for the subnet ID, you can use the following command:

SUBNETID=$(az network vnet subnet show -g $RG --vnet-name $VNET_NAME --name $AKSSUBNET_NAME --query id -o tsv)

使用子网中已存在的 UDR 来定义出站类型。You'll define the outbound type to use the UDR that already exists on the subnet. 此配置使 AKS 跳过负载均衡器的设置和 IP 预配。This configuration will enable AKS to skip the setup and IP provisioning for the load balancer.

重要

有关出站类型 UDR(包括限制)的详细信息,请参阅流出量出站类型 UDRFor more information on outbound type UDR including limitations, see egress outbound type UDR.

提示

可以添加 API 服务器已授权 IP 范围 AKS 功能,以便限制 API 服务器仅访问防火墙的公共终结点。The AKS feature for API server authorized IP ranges can be added to limit API server access to only the firewall's public endpoint. 已授权 IP 范围功能在图中表示为可选。The authorized IP ranges feature is denoted in the diagram as optional. 启用已授权 IP 范围功能来限制 API 服务器访问权限时,开发人员工具必须使用防火墙虚拟网络中的 Jumpbox,或者必须将所有开发人员终结点添加到已授权 IP 范围。When enabling the authorized IP range feature to limit API server access, your developer tools must use a jumpbox from the firewall's virtual network or you must add all developer endpoints to the authorized IP range.

az aks create -g $RG -n $AKSNAME -l $LOC \
  --node-count 3 --generate-ssh-keys \
  --network-plugin $PLUGIN \
  --outbound-type userDefinedRouting \
  --service-cidr 10.41.0.0/16 \
  --dns-service-ip 10.41.0.10 \
  --docker-bridge-address 172.17.0.1/16 \
  --vnet-subnet-id $SUBNETID \
  --service-principal $APPID \
  --client-secret $PASSWORD \
  --api-server-authorized-ip-ranges $FWPUBLIC_IP

使开发人员能够访问 API 服务器Enable developer access to the API server

如果在上一步中为群集使用了已授权 IP 范围,则必须将开发人员工具 IP 地址添加到 AKS 群集的已批准 IP 范围列表,以便从该处访问 API 服务器。If you used authorized IP ranges for the cluster on the previous step, you must add your developer tooling IP addresses to the AKS cluster list of approved IP ranges in order to access the API server from there. 另一种做法是在防火墙虚拟网络中的单独子网内,使用所需的工具配置 Jumpbox。Another option is to configure a jumpbox with the needed tooling inside a separate subnet in the Firewall's virtual network.

使用以下命令将另一个 IP 地址添加到已批准范围Add another IP address to the approved ranges with the following command

# Retrieve your IP address
CURRENT_IP=$(dig @resolver1.opendns.com ANY myip.opendns.com +short)

# Add to AKS approved list
az aks update -g $RG -n $AKSNAME --api-server-authorized-ip-ranges $CURRENT_IP/32

使用 [az aks get-credentials][az-aks-get-credentials] 命令将 kubectl 配置为连接到新建的 Kubernetes 群集。Use the [az aks get-credentials][az-aks-get-credentials] command to configure kubectl to connect to your newly created Kubernetes cluster.

az aks get-credentials -g $RG -n $AKSNAME

部署公共服务Deploy a public service

现在可以开始公开服务并将应用程序部署到此群集。You can now start exposing services and deploying applications to this cluster. 此示例将公开公共服务,但也可以选择通过内部负载均衡器公开内部服务。In this example, we'll expose a public service, but you may also choose to expose an internal service via internal load balancer.

公共服务 DNAT

通过将以下 yaml 复制为名为 example.yaml 的文件来部署 Azure 投票应用程序。Deploy the Azure voting app application by copying the yaml below to a file named example.yaml.

# voting-storage-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: voting-storage
spec:
  replicas: 1
  selector:
    matchLabels:
      app: voting-storage
  template:
    metadata:
      labels:
        app: voting-storage
    spec:
      containers:
      - name: voting-storage
        image: mcr.microsoft.com/aks/samples/voting/storage:2.0
        args: ["--ignore-db-dir=lost+found"]
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 250m
            memory: 256Mi
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: voting-storage-secret
              key: MYSQL_ROOT_PASSWORD
        - name: MYSQL_USER
          valueFrom:
            secretKeyRef:
              name: voting-storage-secret
              key: MYSQL_USER
        - name: MYSQL_PASSWORD
          valueFrom:
            secretKeyRef:
              name: voting-storage-secret
              key: MYSQL_PASSWORD
        - name: MYSQL_DATABASE
          valueFrom:
            secretKeyRef:
              name: voting-storage-secret
              key: MYSQL_DATABASE
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pv-claim
---
# voting-storage-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: voting-storage-secret
type: Opaque
data:
  MYSQL_USER: ZGJ1c2Vy
  MYSQL_PASSWORD: UGFzc3dvcmQxMg==
  MYSQL_DATABASE: YXp1cmV2b3Rl
  MYSQL_ROOT_PASSWORD: UGFzc3dvcmQxMg==
---
# voting-storage-pv-claim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
# voting-storage-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: voting-storage
  labels: 
    app: voting-storage
spec:
  ports:
  - port: 3306
    name: mysql
  selector:
    app: voting-storage
---
# voting-app-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: voting-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: voting-app
  template:
    metadata:
      labels:
        app: voting-app
    spec:
      containers:
      - name: voting-app
        image: mcr.microsoft.com/aks/samples/voting/app:2.0
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          name: http
        env:
        - name: MYSQL_HOST
          value: "voting-storage"
        - name: MYSQL_USER
          valueFrom:
            secretKeyRef:
              name: voting-storage-secret
              key: MYSQL_USER
        - name: MYSQL_PASSWORD
          valueFrom:
            secretKeyRef:
              name: voting-storage-secret
              key: MYSQL_PASSWORD
        - name: MYSQL_DATABASE
          valueFrom:
            secretKeyRef:
              name: voting-storage-secret
              key: MYSQL_DATABASE
        - name: ANALYTICS_HOST
          value: "voting-analytics"
---
# voting-app-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: voting-app
  labels: 
    app: voting-app
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 8080
    name: http
  selector:
    app: voting-app
---
# voting-analytics-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: voting-analytics
spec:
  replicas: 1
  selector:
    matchLabels:
      app: voting-analytics
      version: "2.0"
  template:
    metadata:
      labels:
        app: voting-analytics
        version: "2.0"
    spec:
      containers:
      - name: voting-analytics
        image: mcr.microsoft.com/aks/samples/voting/analytics:2.0
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          name: http
        env:
        - name: MYSQL_HOST
          value: "voting-storage"
        - name: MYSQL_USER
          valueFrom:
            secretKeyRef:
              name: voting-storage-secret
              key: MYSQL_USER
        - name: MYSQL_PASSWORD
          valueFrom:
            secretKeyRef:
              name: voting-storage-secret
              key: MYSQL_PASSWORD
        - name: MYSQL_DATABASE
          valueFrom:
            secretKeyRef:
              name: voting-storage-secret
              key: MYSQL_DATABASE
---
# voting-analytics-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: voting-analytics
  labels: 
    app: voting-analytics
spec:
  ports:
  - port: 8080
    name: http
  selector:
    app: voting-analytics

运行以下命令来部署服务:Deploy the service by running:

kubectl apply -f example.yaml

将 DNAT 规则添加到 Azure 防火墙Add a DNAT rule to Azure Firewall

重要

使用 Azure 防火墙限制出口流量并创建用户定义的路由 (UDR) 来强制所有出口流量时,请确保在防火墙中创建适当的 DNAT 规则,以正确允许入口流量。When you use Azure Firewall to restrict egress traffic and create a user-defined route (UDR) to force all egress traffic, make sure you create an appropriate DNAT rule in Firewall to correctly allow ingress traffic. 结合使用 Azure 防火墙和 UDR 时,会因为路由不对称而中断入口设置。Using Azure Firewall with a UDR breaks the ingress setup due to asymmetric routing. (如果 AKS 子网具有指向防火墙专用 IP 地址的默认路由,但你使用的是公共负载均衡器 - 类型为 LoadBalancer 的入口或 Kubernetes 服务,则会出现此问题)。(The issue occurs if the AKS subnet has a default route that goes to the firewall's private IP address, but you're using a public load balancer - ingress or Kubernetes service of type: LoadBalancer). 在这种情况下,将通过负载均衡器的公共 IP 地址接收传入的负载均衡器流量,但返回路径将通过防火墙的专用 IP 地址。In this case, the incoming load balancer traffic is received via its public IP address, but the return path goes through the firewall's private IP address. 由于防火墙是有状态的,并且无法识别已建立的会话,因此会丢弃返回的数据包。Because the firewall is stateful, it drops the returning packet because the firewall isn't aware of an established session. 若要了解如何将 Azure 防火墙与入口或服务负载均衡器集成,请参阅将 Azure 防火墙与 Azure 标准负载均衡器集成To learn how to integrate Azure Firewall with your ingress or service load balancer, see Integrate Azure Firewall with Azure Standard Load Balancer.

若要配置入站连接,必须将一个 DNAT 规则写入到 Azure 防火墙。To configure inbound connectivity, a DNAT rule must be written to the Azure Firewall. 为了测试与群集的连接,为防火墙前端公共 IP 地址定义了规则,以便路由到内部服务公开的内部 IP。To test connectivity to your cluster, a rule is defined for the firewall frontend public IP address to route to the internal IP exposed by the internal service.

可以自定义目标地址,因为它是防火墙上要访问的端口。The destination address can be customized as it's the port on the firewall to be accessed. 转换的地址必须是内部负载均衡器的 IP 地址。The translated address must be the IP address of the internal load balancer. 转换的端口必须是 Kubernetes 服务的已公开端口。The translated port must be the exposed port for your Kubernetes service.

需要指定分配给 Kubernetes 服务所创建的负载均衡器的内部 IP 地址。You'll need to specify the internal IP address assigned to the load balancer created by the Kubernetes service. 运行以下命令来检索该地址:Retrieve the address by running:

kubectl get services

所需的 IP 地址将在“EXTERNAL-IP”列中列出,如下所示。The IP address needed will be listed in the EXTERNAL-IP column, similar to the following.

NAME               TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes         ClusterIP      10.41.0.1       <none>        443/TCP        10h
voting-analytics   ClusterIP      10.41.88.129    <none>        8080/TCP       9m
voting-app         LoadBalancer   10.41.185.82    20.39.18.6    80:32718/TCP   9m
voting-storage     ClusterIP      10.41.221.201   <none>        3306/TCP       9m

运行以下内容来获取服务 IP:Get the service IP by running:

SERVICE_IP=$(k get svc voting-app -o jsonpath='{.status.loadBalancer.ingress[*].ip}')

运行以下内容来添加 NAT 规则:Add the NAT rule by running:

az network firewall nat-rule create --collection-name exampleset --destination-addresses $FWPUBLIC_IP --destination-ports 80 --firewall-name $FWNAME --name inboundrule --protocols Any --resource-group $RG --source-addresses '*' --translated-port 80 --action Dnat --priority 100 --translated-address $SERVICE_IP

验证连接Validate connectivity

在浏览器中导航到 Azure 防火墙前端 IP 地址来验证连接。Navigate to the Azure Firewall frontend IP address in a browser to validate connectivity.

应看到 AKS 投票应用程序。You should see the AKS voting app. 此示例中,防火墙公共 IP 为 52.253.228.132In this example, the Firewall public IP was 52.253.228.132.

aks-vote

清理资源Clean up resources

若要清理 Azure 资源,请删除 AKS 资源组。To clean up Azure resources, delete the AKS resource group.

az group delete -g $RG

后续步骤Next steps

通过学习这篇文章,了解了要限制群集的出口流量时可以使用的端口和地址。In this article, you learned what ports and addresses to allow if you want to restrict egress traffic for the cluster. 还知道了如何使用 Azure 防火墙保护出站流量。You also saw how to secure your outbound traffic using Azure Firewall.

如果需要,可以将以上步骤推广到将流量转发到首选出口解决方案,根据出站类型userDefinedRoute文档进行操作。If needed, you can generalize the steps above to forward the traffic to your preferred egress solution, following the Outbound Type userDefinedRoute documentation.

如果想要限制 Pod 在彼此之间如何通信以及群集内的东-西流量限制,请参阅在 AKS 中使用网络策略保护 Pod 之间的流量If you want to restrict how pods communicate between themselves and East-West traffic restrictions within cluster see Secure traffic between pods using network policies in AKS.