Vulnerability management for Azure Kubernetes Service (AKS)
Vulnerability management involves detecting, assessing, mitigating, and reporting on any security vulnerabilities that exist in an organization’s systems and software. Vulnerability management is a shared responsibility between you and Microsoft.
This article describes how Azure manages security vulnerabilities and security updates (also referred to as patches), for Azure Kubernetes Service (AKS) clusters.
How vulnerabilities are discovered
Azure identifies and patches vulnerabilities and missing security updates for the following components:
AKS Container Images
Ubuntu operating system 18.04 and 22.04 worker nodes: Canonical provides Azure with OS builds that have all available security updates applied.
Windows Server 2022 OS worker nodes: The Windows Server operating system is patched on the second Tuesday of every month. SLAs should be the same as per their support contract and severity.
Azure Linux OS Nodes: Azure Linux provides AKS with OS builds that have all available security updates applied.
AKS Container Images
While the Cloud Native Computing Foundation (CNCF) owns and maintains most of the code AKS runs, Azure takes responsibility for building the open-source packages we deploy on AKS. That responsibility includes having complete ownership of the build, scan, sign, validate, and hotfix process, as well as control over the binaries in container images. Having responsibility for building the open-source packages deployed on AKS enables us to establish a software supply chain over the binary, and to patch the software as needed.
Azure is active in the broader Kubernetes ecosystem to help build the future of cloud-native compute in the wider CNCF community. This work not only ensures the quality of every Kubernetes release for the world, but also enables AKS quickly get new Kubernetes releases out into production for several years. In some cases, ahead of other cloud providers by several months. Azure collaborates with other industry partners in the Kubernetes security organization. For example, the Security Response Committee (SRC) receives, prioritizes, and patches embargoed security vulnerabilities before they're announced to the public. This commitment ensures Kubernetes is secure for everyone, and enables AKS to patch and respond to vulnerabilities faster to keep our customers safe. In addition to Kubernetes, Azure has signed up to receive pre-release notifications for software vulnerabilities for products such as Envoy, container runtimes, and many other open-source projects.
Azure scans container images using static analysis to discover vulnerabilities and missing updates in Kubernetes and Azure-managed containers. If fixes are available, the scanner automatically begins the update and release process.
In addition to automated scanning, Azure discovers and updates vulnerabilities unknown to scanners in the following ways:
Azure performs its own audits, penetration testing, and vulnerability discovery across all AKS platforms. Specialized teams inside Azure and trusted third-party security vendors conduct their own attack research.
Azure actively engages with the security research community through multiple vulnerability reward programs. A dedicated Azure Bounty program provides significant bounties for the best cloud vulnerability found each year.
Azure collaborates with other industry and open source software partners who share vulnerabilities, security research, and updates before the public release of the vulnerability. The goal of this collaboration is to update large pieces of Internet infrastructure before the vulnerability is announced to the public. In some cases, Azure contributes vulnerabilities found to this community.
Azure's security collaboration happens on many levels. Sometimes it occurs formally through programs where organizations sign up to receive pre-release notifications about software vulnerabilities for products such as Kubernetes and Docker. Collaboration also happens informally due to our engagement with many open source projects such as the Linux kernel, container runtimes, virtualization technology, and others.
Worker Nodes
Linux nodes
The nightly canonical OS security updates are turned off by default in AKS. In order to enable them explicitly, use the unmanaged [channel][aks-node-image-upgrade].
If you are using the unmanaged [channel][aks-node-image-upgrade], then nightly canonical security updates are applied to the OS on the node. The node image used to create nodes for your cluster remains unchanged. If a new Linux node is added to your cluster, the original image is used to create the node. This new node receives all the security and kernel updates available during the automatic assessment performed every night, but remains unpatched until all checks and restarts are complete. You can use node image upgrade to check for and update node images used by your cluster. For more information on node image upgrade, see [Azure Kubernetes Service (AKS) node image upgrade][aks-node-image-upgrade].
For AKS clusters using a [channel][aks-node-image-upgrade] other than unmanaged, the unattended upgrade process is disabled.
Windows Server nodes
For Windows Server nodes, Windows Update doesn't automatically run and apply the latest updates. Schedule Windows Server node pool upgrades in your AKS cluster around the regular Windows Update release cycle and your own update management process. This upgrade process creates nodes that run the latest Windows Server image and patches, then removes the older nodes. For more information on this process, see [Upgrade a node pool in AKS][upgrade-node-pool-in-aks].
How vulnerabilities are classified
Azure makes large investments in security hardening the entire stack, including the OS, container, Kubernetes, and network layers, in addition to setting good defaults and providing security-hardened configurations and managed components. Combined, these efforts help to reduce the impact and likelihood of vulnerabilities.
The AKS team classifies vulnerabilities according to the Kubernetes vulnerability scoring system. Classifications consider many factors including AKS configuration and security hardening. As a result of this approach, and the investments AKS make in security, AKS vulnerability classifications might differ from other classification sources.
The following table describes vulnerability severity categories:
| Severity | Description |
|---|---|
| Critical | A vulnerability easily exploitable in all clusters by an unauthenticated remote attacker that leads to full system compromise. |
| High | A vulnerability easily exploitable for many clusters that leads to loss of confidentiality, integrity, or availability. |
| Medium | A vulnerability exploitable for some clusters where loss of confidentiality, integrity, or availability is limited by common configurations, difficulty of the exploit itself, required access, or user interaction. |
| Low | All other vulnerabilities. Exploitation is unlikely or consequences of exploitation are limited. |
How vulnerabilities are updated
AKS patches Common Vulnerabilities and Exposures (CVEs) that have a vendor fix every week. Any CVEs without a fix are waiting on a vendor fix before they can be remediated. The fixed container images are cached in the next corresponding virtual hard disk (VHD) build, which also contains the updated Ubuntu/Azure Linux/Windows patched CVEs. As long as you're running the updated VHD, you shouldn't be running any container image CVEs with a vendor fix that is over 30 days old.
For the OS-based vulnerabilities in the VHD, AKS also relies on node image vhd updates by default, so any security updates will come with weekly node image releases. Unattended upgrades is disabled unless you switch to unmanaged which is not recommended as its release is global.
Update release timelines
Azure's goal is to mitigate detected vulnerabilities within a time period appropriate for the risks they represent. The [Azure FedRAMP High][microsoft-azure-fedramp-high] Provisional Authorization to Operate (P-ATO) includes AKS in audit scope and has been authorized. FedRAMP Continuous Monitoring Strategy Guide and the FedRAMP Low, Moderate, and High Security Control baselines requires remediation of known vulnerabilities within a specific time period according to their severity level. As specified in FedRAMP RA-5d.
How vulnerabilities and updates are communicated
In general, Azure doesn't broadly communicate the release of new patch versions for AKS. However, Azure constantly monitors and validates available CVE patches to support them in AKS in a timely manner. If a critical patch is found or user action is required, Azure posts and updates CVE issue details on GitHub.
Security Reporting
You can report a security issue to the Microsoft Security Response Center (MSRC), by creating a vulnerability report.
If you prefer to submit a report without logging in to the tool, send email to secure@microsoft.com. If possible, encrypt your message with our PGP key by downloading it from the Microsoft Security Response Center PGP Key page.
You should receive a response within 24 hours. If for some reason you don't, follow up with an email to ensure we received your original message. For more information, go to the Microsoft Security Response Center.
Include the following requested information (as much as you can provide) to help us better understand the nature and scope of the possible issue:
- Type of issue (for example, buffer overflow, SQL injection, cross-site scripting, etc.)
- Full paths of source file(s) related to the manifestation of the issue
- The location of the affected source code (tag/branch/commit or direct URL)
- Any special configuration required to reproduce the issue
- Step-by-step instructions to reproduce the issue
- Proof-of-concept or exploit code (if possible)
- Impact of the issue, including how an attacker might exploit the issue
This information helps us triage your reported security issue quicker.
If you're reporting for a bug bounty, more complete reports can contribute to a higher bounty award. For more information about our active programs, see Microsoft Bug Bounty Program.
Policy
Azure follows the principle of Coordinated Vulnerability Disclosure.
Next steps
See the overview about Upgrading Azure Kubernetes Service clusters and node pools.
[apply-security-kernel-updates-to-aks-nodes]: node-updates-kured.md [aks-node-image-upgrade]: node-image-upgrade.md [upgrade-node-pool-in-aks]: manage-node-pools.md#upgrade-a-single-node-pool [aks-node-image-upgrade]: auto-upgrade-node-image.md