Azure Kubernetes 服务 (AKS) 中的网络连接和安全的最佳做法Best practices for network connectivity and security in Azure Kubernetes Service (AKS)

在创建和管理 Azure Kubernetes 服务 (AKS) 中的群集时,可以为节点和应用程序提供网络连接。As you create and manage clusters in Azure Kubernetes Service (AKS), you provide network connectivity for your nodes and applications. 这些网络资源包括 IP 地址范围、负载均衡器和入口控制器。These network resources include IP address ranges, load balancers, and ingress controllers. 若要保持高质量的应用程序服务,则需要计划和配置这些资源。To maintain a high quality of service for your applications, you need to plan for and then configure these resources.

这篇最佳做法文章重点介绍群集运算符的网络连接和安全。This best practices article focuses on network connectivity and security for cluster operators. 在本文中,学习如何:In this article, you learn how to:

  • 比较 AKS 中的 kubenet 和 Azure CNI 网络模式Compare the kubenet and Azure CNI network modes in AKS
  • 计划所需的 IP 地址和连接Plan for required IP addressing and connectivity
  • 使用负载均衡器、入口控制器或 Web 应用程序防火墙 (WAF) 分配流量Distribute traffic using load balancers, ingress controllers, or a web application firewall (WAF)
  • 安全地连接到群集节点Securely connect to cluster nodes

选择合适的网络模型Choose the appropriate network model

最佳做法指南 - 为了与现有的虚拟网络或本地网络集成,请在 AKS 中使用 Azure CNI 网络。Best practice guidance - For integration with existing virtual networks or on-premises networks, use Azure CNI networking in AKS. 利用此网络模型,还可以更大程度地将企业环境中的资源和控制相分离。This network model also allows greater separation of resources and controls in an enterprise environment.

虚拟网络为 AKS 节点和客户提供了用于访问应用程序的基本链接。Virtual networks provide the basic connectivity for AKS nodes and customers to access your applications. 将 AKS 群集部署到虚拟网络有两种不同的方法:There are two different ways to deploy AKS clusters into virtual networks:

  • Kubenet 网络 - Azure 在部署群集时管理虚拟网络资源,并使用 kubenet Kubernetes 插件。Kubenet networking - Azure manages the virtual network resources as the cluster is deployed and uses the kubenet Kubernetes plugin.
  • Azure CNI 网络 - 部署到虚拟网络,并使用 Azure 容器网络接口 (CNI) Kubernetes 插件。Azure CNI networking - Deploys into a virtual network, and uses the Azure Container Networking Interface (CNI) Kubernetes plugin. Pod 接收可以路由到其他网络服务或本地资源的各个 Ip。Pods receive individual IPs that can route to other network services or on-premises resources.

容器网络接口 (CNI) 是与供应商无关的协议,允许容器运行时将向网络提供程序发出请求。The Container Networking Interface (CNI) is a vendor-neutral protocol that lets the container runtime make requests to a network provider. Azure CNI 将 IP 地址分配给 Pod 和节点,并在接到现有的 Azure 虚拟网络时提供 IP 地址管理 (IPAM) 功能。The Azure CNI assigns IP addresses to pods and nodes, and provides IP address management (IPAM) features as you connect to existing Azure virtual networks. 每个节点和 Pod 资源接收 Azure 虚拟网络中的 IP 地址,与其他资源或服务通信不需要其他路由。Each node and pod resource receives an IP address in the Azure virtual network, and no additional routing is needed to communicate with other resources or services.

显示两个节点的示意图,其中的网桥将每个节点连接到单个 Azure VNet

对于生产部署,kubenet 和 Azure CNI 都是有效选项。For production deployments, both kubenet and Azure CNI are valid options.

用于生产的 Azure CNI 网络的一个明显优势是网络模型允许将资源的控制和管理分离。A notable benefit of Azure CNI networking for production is the network model allows for separation of control and management of resources. 从安全角度看,通常希望不同的团队来管理和保护这些资源。From a security perspective, you often want different teams to manage and secure those resources. 使用 Azure CNI 网络,可以通过分配到每个 Pod 的 IP 地址直接连接到现有 Azure 资源、本地资源或其他服务。Azure CNI networking lets you connect to existing Azure resources, on-premises resources, or other services directly via IP addresses assigned to each pod.

使用 Azure CNI 网络时,虚拟网络资源位于 AKS 群集外单独存在的资源组中。When you use Azure CNI networking, the virtual network resource is in a separate resource group to the AKS cluster. 委托 AKS 服务主体的权限以访问和管理这些资源。Delegate permissions for the AKS service principal to access and manage these resources. AKS 群集使用的服务主体在虚拟网络中的子网上必须至少具有网络参与者权限。The service principal used by the AKS cluster must have at least Network Contributor permissions on the subnet within your virtual network. 如果希望定义自定义角色而不是使用内置的网络参与者角色,则需要以下权限:If you wish to define a custom role instead of using the built-in Network Contributor role, the following permissions are required:

  • Microsoft.Network/virtualNetworks/subnets/join/action
  • Microsoft.Network/virtualNetworks/subnets/read

有关 AKS 服务主体委托的详细信息,请参阅委托对其他 Azure 资源的访问权限For more information about AKS service principal delegation, see Delegate access to other Azure resources. 你还可以使用系统分配的托管标识来获得权限,而非使用服务主体。Instead of a service principal, you can also use the system assigned managed identity for permissions. 有关详细信息,请参阅使用托管标识For more information, see Use managed identities.

每个节点和 Pod 在接收自己的 IP 地址时,请规划 AKS 子网的地址范围。As each node and pod receive its own IP address, plan out the address ranges for the AKS subnets. 子网必须大到足以为每个部署的节点、Pod 和网络资源提供 IP 地址。The subnet must be large enough to provide IP addresses for every node, pods, and network resources that you deploy. 每个 AKS 群集必须位于自己的子网中。Each AKS cluster must be placed in its own subnet. 要允许连接到 Azure 中的本地网络或对等互连网络,请勿使用与现有网络资源重叠的 IP 地址范围。To allow connectivity to on-premises or peered networks in Azure, don't use IP address ranges that overlap with existing network resources. 每个节点使用 kubenet 和 Azure CNI 网络运行的 Pod 数量存在默认限制。There are default limits to the number of pods that each node runs with both kubenet and Azure CNI networking. 若要处理横向扩展事件或群集升级,还需要其他可以在分配的子网中使用的 IP 地址。To handle scale out events or cluster upgrades, you also need additional IP addresses available for use in the assigned subnet. 如果使用 Windows Server 容器,此额外的地址空间尤其重要,因为这些节点池需要升级才能应用最新的安全修补程序。This additional address space is especially important if you use Windows Server containers, as those node pools require an upgrade to apply the latest security patches. 若要详细了解 Windows Server 节点,请参阅升级 AKS 中的节点池For more information on Windows Server nodes, see Upgrade a node pool in AKS.

若要计算所需的 IP 地址,请参阅在 AKS 中配置 Azure CNI 网络To calculate the IP address required, see Configure Azure CNI networking in AKS.

Kubenet 网络Kubenet networking

尽管 kubenet 不需要在部署群集之前设置虚拟网络,但也有一些缺点:Although kubenet doesn't require you to set up the virtual networks before the cluster is deployed, there are disadvantages:

  • 节点和 Pod 位于不同的 IP 子网中。Nodes and pods are placed on different IP subnets. 用户定义的路由 (UDR) 和 IP 转发用于 Pod 和节点之间的路由流量。User Defined Routing (UDR) and IP forwarding is used to route traffic between pods and nodes. 这个额外的路由可能会降低网络性能。This additional routing may reduce network performance.
  • 连接到现有本地网络或与其他 Azure 虚拟网络对等互连可能很复杂。Connections to existing on-premises networks or peering to other Azure virtual networks can be complex.

Kubenet 适用于小型开发或测试工作负荷,因为无需从 AKS 群集单独创建虚拟网络和子网。Kubenet is suitable for small development or test workloads, as you don't have to create the virtual network and subnets separately from the AKS cluster. 流量较低或者将工作负荷直接迁移到容器中的简单网站,也可以受益于使用 kubenet 网络部署的 AKS 群集的简单性。Simple websites with low traffic, or to lift and shift workloads into containers, can also benefit from the simplicity of AKS clusters deployed with kubenet networking. 对于大多数生产部署,应计划和使用 Azure CNI 网络。For most production deployments, you should plan for and use Azure CNI networking. 还可以使用 kubenet 配置自己的 IP 地址范围和虚拟网络You can also configure your own IP address ranges and virtual networks using kubenet.

分配入口流量Distribute ingress traffic

最佳实践指南 - 要将 HTTP 或 HTTPS 流量分配到应用程序,请使用入口资源和控制器。Best practice guidance - To distribute HTTP or HTTPS traffic to your applications, use ingress resources and controllers. 入口控制器通过常规 Azure 负载均衡器提供附加功能,可作为本机 Kubernetes 资源进行管理。Ingress controllers provide additional features over a regular Azure load balancer, and can be managed as native Kubernetes resources.

Azure 负载均衡器可以将客户流量分配到 AKS 群集中的各个应用程序,但是对这些流量的了解有限。An Azure load balancer can distribute customer traffic to applications in your AKS cluster, but it's limited in what it understands about that traffic. 负载均衡器资源在第 4 层工作,并根据协议或端口分配流量。A load balancer resource works at layer 4, and distributes traffic based on protocol or ports. 大多数使用 HTTP 或 HTTPS 的 Web 应用程序应使用在第 7 层工作的 Kubernetes 入口资源和控制器。Most web applications that use HTTP or HTTPS should use Kubernetes ingress resources and controllers, which work at layer 7. 入口可以根据应用程序的 URL 分配流量并处理 TLS/SSL 终端。Ingress can distribute traffic based on the URL of the application and handle TLS/SSL termination. 此功能还可以减少公开和映射的 IP 地址数。This ability also reduces the number of IP addresses you expose and map. 使用负载平衡器,每个应用程序通常需要分配一个公共 IP 地址并映射到 AKS 群集中的服务。With a load balancer, each application typically needs a public IP address assigned and mapped to the service in the AKS cluster. 使用入口资源,单个 IP 地址可以将流量分配给多个应用程序。With an ingress resource, a single IP address can distribute traffic to multiple applications.

显示两个节点的示意图,其中的网桥将每个节点连接到单个 Azure VNet

入口有两个组件:There are two components for ingress:

  • 入口资源,和An ingress resource, and
  • 入口控制器An ingress controller

入口资源是 kind: Ingress 的 YAML 清单,它定义了将流量路由到 AKS 群集中运行的服务的主机、证书和规则。The ingress resource is a YAML manifest of kind: Ingress that defines the host, certificates, and rules to route traffic to services that run in your AKS cluster. 以下示例 YAML 清单会将 myapp.com 的流量分配到 blogservice 或 storeservice 两个服务中的一个 。The following example YAML manifest would distribute traffic for myapp.com to one of two services, blogservice or storeservice. 客户根据他们访问的 URL,被定向到一个或另一个服务。The customer is directed to one service or the other based on the URL they access.

kind: Ingress
metadata:
 name: myapp-ingress
   annotations: kubernetes.io/ingress.class: "PublicIngress"
spec:
 tls:
 - hosts:
   - myapp.com
   secretName: myapp-secret
 rules:
   - host: myapp.com
     http:
      paths:
      - path: /blog
        backend:
         serviceName: blogservice
         servicePort: 80
      - path: /store
        backend:
         serviceName: storeservice
         servicePort: 80

入口控制器是在 AKS 节点上运行的守护程序并监视传入请求。An ingress controller is a daemon that runs on an AKS node and watches for incoming requests. 然后根据入口资源中定义的规则分配流量。Traffic is then distributed based on the rules defined in the ingress resource. 最佳常见的入口控制器基于 NGINXThe most common ingress controller is based on NGINX. AKS 不会限制你使用特定控制器,因此可以使用其他控制器,例如 ContourHAProxyTraefikAKS doesn't restrict you to a specific controller, so you can use other controllers such as Contour, HAProxy, or Traefik.

必须在 Linux 节点上计划入口控制器。Ingress controllers must be scheduled on a Linux node. Windows Server 节点不应运行入口控制器。Windows Server nodes shouldn't run the ingress controller. 在 YAML 清单或 Helm 图表部署中使用节点选择器来指示资源应在基于 Linux 的节点上运行。Use a node selector in your YAML manifest or Helm chart deployment to indicate that the resource should run on a Linux-based node. 有关详细信息,请参阅使用节点选择器控制在 AKS 中计划 Pod 的位置For more information, see Use node selectors to control where pods are scheduled in AKS.

入口有许多方案,包括以下操作指南:There are many scenarios for ingress, including the following how-to guides:

使用 Web 应用程序防火墙 (WAF) 保护流量Secure traffic with a web application firewall (WAF)

最佳做法指南 - 要扫描传入流量是否存在潜在攻击,请使用 Web 应用程序防火墙 (WAF),例如 Barracuda WAF for Azure 或 Azure 应用程序网关。Best practice guidance - To scan incoming traffic for potential attacks, use a web application firewall (WAF) such as Barracuda WAF for Azure or Azure Application Gateway. 这些更高级的网络资源还可以路由 HTTP 和 HTTPS 连接或基本 TLS 终止之外的流量。These more advanced network resources can also route traffic beyond just HTTP and HTTPS connections or basic TLS termination.

将流量分配到服务和应用程序的入口控制器通常是 AKS 群集中的 Kubernetes 资源。An ingress controller that distributes traffic to services and applications is typically a Kubernetes resource in your AKS cluster. 控制器作为守护程序在 AKS 节点上运行,并使用一些节点资源(例如 CPU、内存和网络带宽)。The controller runs as a daemon on an AKS node, and consumes some of the node's resources such as CPU, memory, and network bandwidth. 在较大的环境中,通常需要将部分流量路由或 TLS 终端卸载到 AKS 群集之外的网络资源。In larger environments, you often want to offload some of this traffic routing or TLS termination to a network resource outside of the AKS cluster. 还需要扫描传入流量是否存在潜在攻击。You also want to scan incoming traffic for potential attacks.

显示两个节点的示意图,其中的网桥将每个节点连接到单个 Azure VNet

Web 应用程序防火墙 (WAF) 通过筛选传入流量提供额外的安全层。A web application firewall (WAF) provides an additional layer of security by filtering the incoming traffic. 开放式 Web 应用程序安全项目 (OWASP) 提供了一套规则来监视跨网站脚本或 cookie 中毒之类的攻击。The Open Web Application Security Project (OWASP) provides a set of rules to watch for attacks like cross site scripting or cookie poisoning. Azure 应用程序网关(目前在 AKS 中处于预览状态)是一种 WAF,可在流量到达 AKS 群集和应用程序之前与 AKS 群集集成以提供这些安全功能。Azure Application Gateway (currently in preview in AKS) is a WAF that can integrate with AKS clusters to provide these security features, before the traffic reaches your AKS cluster and applications. 其他第三方解决方案也可以执行这些功能,因此可以在给定的产品中继续使用现有的资源和专业知识。Other third-party solutions also perform these functions, so you can continue to use existing investments or expertise in a given product.

负载均衡器或入口资源继续在 AKS 群集中运行以进一步优化流量分配。Load balancer or ingress resources continue to run in your AKS cluster to further refine the traffic distribution. 通过资源定义,可以将应用程序网关可以作为入口控制器进行集中管理。App Gateway can be centrally managed as an ingress controller with a resource definition. 首先,创建应用程序网关入口控制器To get started, create an Application Gateway Ingress controller.

使用网络策略控制流量流Control traffic flow with network policies

最佳做法指南 - 使用网络策略允许或拒绝到 Pod 的流量。Best practice guidance - Use network policies to allow or deny traffic to pods. 默认情况下,将允许群集中 Pod 之间的所有流量。By default, all traffic is allowed between pods within a cluster. 为了提高安全性,请定义对 Pod 通信进行限制的规则。For improved security, define rules that limit pod communication.

网络策略是一项 Kubernetes 功能,可用于控制 Pod 之间的流量流。Network policy is a Kubernetes feature that lets you control the traffic flow between pods. 可选择基于分配的标签、命名空间或流量端口等设置来允许或拒绝流量。You can choose to allow or deny traffic based on settings such as assigned labels, namespace, or traffic port. 使用网络策略提供了一种云本机方式来控制流量流。The use of network policies gives a cloud-native way to control the flow of traffic. 因为 Pod 是在 AKS 群集中动态创建的,则可以动态应用所需的网络策略。As pods are dynamically created in an AKS cluster, the required network policies can be automatically applied. 不要使用 Azure 网络安全组来控制 Pod 到 Pod 流量,请使用网络策略。Don't use Azure network security groups to control pod-to-pod traffic, use network policies.

若要使用网络策略,必须在创建 AKS 群集时启用此功能。To use network policy, the feature must be enabled when you create an AKS cluster. 无法在现有 AKS 群集上启用网络策略。You can't enable network policy on an existing AKS cluster. 请提前进行规划以确保在群集上启用网络策略并根据需要使用它们。Plan ahead to make sure that you enable network policy on clusters and can use them as needed. 应该只对 AKS 中基于 Linux 的节点和 Pod 使用网络策略。Network policy should only be used for Linux-based nodes and pods in AKS.

网络策略是使用 YAML 清单作为 Kubernetes 资源创建的。A network policy is created as a Kubernetes resource using a YAML manifest. 策略应用于所定义的 Pod,然后,入口或出口规则定义流量可以如何流动。The policies are applied to defined pods, then ingress or egress rules define how the traffic can flow. 以下示例将向应用了 app: backend 标签的 Pod 应用网络策略。The following example applies a network policy to pods with the app: backend label applied to them. 然后,入口规则仅允许来自具有 app: frontend 标签的 Pod 的流量:The ingress rule then only allows traffic from pods with the app: frontend label:

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: backend-policy
spec:
  podSelector:
    matchLabels:
      app: backend
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend

若要开始使用策略,请参阅在 Azure Kubernetes 服务 (AKS) 中使用网络策略保护 Pod 之间的流量To get started with policies, see Secure traffic between pods using network policies in Azure Kubernetes Service (AKS).

通过堡垒主机安全地连接到节点Securely connect to nodes through a bastion host

最佳做法指南 - 不公开到 AKS 节点的远程连接。Best practice guidance - Don't expose remote connectivity to your AKS nodes. 在管理虚拟网络中创建堡垒主机或跳转盒。Create a bastion host, or jump box, in a management virtual network. 使用堡垒主机将流量安全地路由到 AKS 群集以远程管理任务。Use the bastion host to securely route traffic into your AKS cluster to remote management tasks.

AKS 中的大多数操作都可以使用 Azure 管理工具或通过 Kubernetes API 服务器来完成。Most operations in AKS can be completed using the Azure management tools or through the Kubernetes API server. AKS 节点不会连接到公共 Internet,并且仅在专用网络上可用。AKS nodes aren't connected to the public internet, and are only available on a private network. 要连接到节点并执行维护或排查问题,请通过堡垒主机或跳转盒路由连接。To connect to nodes and perform maintenance or troubleshoot issues, route your connections through a bastion host, or jump box. 此主机应位于与 AKS 群集虚拟网络安全对等互连的单独的管理虚拟网络中。This host should be in a separate management virtual network that is securely peered to the AKS cluster virtual network.

显示两个节点的示意图,其中的网桥将每个节点连接到单个 Azure VNet

堡垒主机的管理网络也应受到保护。The management network for the bastion host should be secured, too. 使用 Azure ExpressRouteVPN 网关连接到本地网络,并使用网络安全组控制访问。Use an Azure ExpressRoute or VPN gateway to connect to an on-premises network, and control access using network security groups.

后续步骤Next steps

本文重点介绍网络连接性和安全性。This article focused on network connectivity and security. 有关 Kubernetes 中的网络基础知识的详细信息,请参阅 Azure Kubernetes 服务 (AKS) 中应用程序的网络概念For more information about network basics in Kubernetes, see Network concepts for applications in Azure Kubernetes Service (AKS)