Safely update Kubernetes and node images across multiple clusters

Applies to: ✔️ Fleet Manager ✔️ Fleet Manager with hub cluster

Platform admins managing large number of clusters often have problems with staging the updates of multiple clusters (for example, upgrading node OS image or Kubernetes versions) in a safe and predictable way. To address this challenge, Azure Kubernetes Fleet Manager allows you to orchestrate updates across multiple clusters using update runs.

Update runs consist of stages, groups, and strategies. You can apply update runs manually for one-time updates or automatically for ongoing regular updates by using auto-upgrade profiles. All update runs, both manual and automated, honor cluster maintenance windows.

Understanding Update Runs

An update run represents an update being applied to a collection of AKS clusters. It consists of the update goal and sequence. The update goal describes the desired updates. For example, upgrading to a specific Kubernetes version or applying a consistent node image across all clusters.

To achieve the best results when using update runs, it's important to understand the following concepts.

• Update Strategy: describes a reusable update sequence consisting of stages and groups of clusters. A cluster appears in a group in a stage based on the update group or member labels it's assigned. For more information, see understanding update strategies.

  • Update Stage: an Update Strategy is divided into Update Stages, which are applied sequentially. For example, test environment clusters are in the first Update Stage, while production environment clusters go in a second Update Stage. An Update Stage contains one or more Update Groups. You can use additional controls such as maximum concurrency, wait times, and approval gates for more control over Update Stage execution.

  • Update Group: Each Update Stage contains one or more Update Groups, which select clusters to update. Assign member clusters to Update Groups by using either the Update Group property of the cluster or the in-preview label-based matching by using member labels. Update Groups in an Update Stage update in parallel.

  Note

  The maximum number of Update Groups in each Update Stage is 50.

  • Further flow controls: more controls are available to provide flexibility over how fast a fleet of clusters can be updated:

    • Maximum Concurrency (preview): use the Maximum Concurrency configuration to change how many clusters are upgraded in parallel. You can configure this behavior at both the Stage and Group level.

    • Maximum allowed failures (preview): use the Maximum allowed failures configuration to control how many member cluster upgrade failures are tolerated before the update run is stopped. You can configure this behavior at both the Stage and Group level.

    • Gates: Pause the update run until their condition is met.

      • Approval gates (preview): Can be configured before or after each stage or group. Approvals pause the update run, allowing either you or automations that you've set up to check that it's OK to proceed. After you or your automation grants approval, the update run will continue.

      • Scheduled start gates (preview): Can be configured before each stage or group. Scheduled start gates pause the update run until a specified day and time. The gate completes automatically when the scheduled time is reached, or you can manually complete it to proceed at any time.

• Auto-upgrade Profile: automatically create and start an update run when new Kubernetes or node image versions are made available by AKS. For further information, see understanding auto-upgrade profiles.

Update Run options

Update runs can apply three types of upgrades:

• Upgrade Kubernetes versions for the control plane and the nodes. This upgrade includes upgrading the node image.
• Upgrade Kubernetes versions for only the control plane of the clusters.
• Upgrade only the node images.

You can specify the target Kubernetes version to upgrade to, but you can't select the target node image versions. The system automatically selects the target node image versions based on your preferences:

• Latest: Use the latest node images available in the Azure region of each cluster when the upgrade of that cluster starts. As a result, different image versions could be used across the fleet, depending on which Azure region a cluster is in and when its upgrade actually starts.
• Consistent: When the update run starts, pick image versions that are currently available in all Azure regions where the clusters in this run are located. As such, consistent image versions are used across all clusters.

Choose Latest to use fresher image versions and minimize security risks. Choose Consistent to improve reliability by using and verifying those images in clusters in earlier stages before using them in later clusters.

Update Run states

To understand the lifecycle of an update run, you need to know each status, the actions you can take, and how the status is calculated.

Pending status

• Update Run: if the current stage is in Pending state.
• Update Stage: if all Update Groups in the stage are Pending or not started, or if it has a Pending gate.
• Update Group: if all clusters in the group are Pending or not started, or if it has a Pending gate. When a cluster moves to Pending, the update run attempts to upgrade the next cluster in the group. If all members are Pending, the group moves to Pending. The update run waits for all groups in a stage to complete before moving on to the next stage.
• Member cluster: for any of the following reasons, which you can view in the message field.
  • Maintenance window isn't open. Message indicates next opening time.
  • Target Kubernetes or node image version isn't yet available in the cluster's Azure region. Message links to the AKS release tracker to check release status.

Skipped status

• Update Run: the system detected that all stages were Skipped.
• Update Stage: a user marked the stage, or all groups in the stage as Skipped.
• Update Group: a user marked the group, or all clusters in the group as Skipped.
• Member cluster: for any of the following reasons, which you can view in the message field.
  • User explicitly skipped the cluster, group, or stage.
  • Cluster is already at the target Kubernetes version (if update run mode is Full or ControlPlaneOnly) and all node pools are at the target node image version.
  • When consistent node image is selected and it's not possible to find the target image version for one of the node pools. This situation can occur when a new node pool with a new Virtual Machine (VM) SKU is added after an update run is started.

Failed status

The failed status cascades from clusters as shown. A summary error message displays the cause of the failed cluster upgrade.

• Update Run: at least one cluster in the current group failed.
• Update Stage: at least one cluster in a group in the stage failed.
• Update Group: at least one cluster in the group failed.
• Member cluster: the upgrade failed and the cluster status is set as Failed.

When you configure Maximum allowed failures, the failure threshold affects the status of the update group, update stage, and update run. It doesn't affect an individual member cluster's status. If a member cluster upgrade fails, its status is still set to Failed.

• If the failure count for the group or stage exceeds the configured maxAllowedFailures threshold, the group or stage is marked as Failed with a summary error message, and the update run stops progressing. If no maxAllowedFailures is configured (or it's set to 0), a single failure stops the entire run.
• If the failure count is within the maxAllowedFailures threshold, the update run continues upgrading subsequent members. For more information, see Maximum allowed failures.

Completed status

A Completed status means the update run reached a terminal lifecycle state. When you use Maximum allowed failures, Completed means the configured failure threshold wasn't exceeded when Fleet Manager decided whether to continue scheduling work. It doesn't guarantee a minimum success rate or healthy outcomes. Always inspect FailureCount, member states, and failure messages.

Planned maintenance windows

Update runs honor planned maintenance windows that you set at the AKS cluster level.

AKS clusters support two distinct maintenance windows - one for Kubernetes (control plane) upgrades and one for node image upgrades. Maintenance windows define periods when updates can be applied to a cluster, but aren't an update trigger.

Fleet Manager update runs honors AKS maintenance windows as follows:

Update run prioritizes upgrading clusters based on planned maintenance in the following order:

1. Cluster with an open ongoing maintenance window.
2. Cluster with maintenance window opening in the next four hours.
3. Cluster with no maintenance window.
4. Cluster with a closed maintenance window.

Understanding Auto-upgrade Profiles

Use Auto-upgrade Profiles to automatically trigger Update Runs when new Kubernetes or node image versions are available for AKS.

In an Auto-upgrade Profile, you configure:

• a Channel (Rapid, Stable, TargetKubernetesVersion, NodeImage) which determines the type of update applied to the clusters.
• an UpdateStrategy that configures the sequence in which the clusters are upgraded. If you don't supply a strategy, clusters update one by one sequentially.
• the NodeImageSelectionType (Latest, Consistent) to specify how the node image is selected when upgrading the Kubernetes version.

Rapid channel

The Rapid channel is always the most recent AKS-supported Kubernetes minor release. Cluster minor versions automatically change when AKS releases a new Kubernetes minor version.

Examples:

• The latest supported minor version is 1.30. Any patch release in the 1.30 minor range is considered for Rapid channel updates.
• A new minor Kubernetes version of 1.31 is published. 1.30 shifts to the Stable channel. Any cluster previously receiving updates from 1.30 is updated to the most recent patch for 1.31 that's now the Rapid channel.

Stable channel

The Stable channel is always the minor version before the Rapid channel. Sometimes, people refer to Stable as "N-1", where "N" is the latest (Rapid channel) supported Kubernetes minor version. Cluster minor versions automatically change when AKS releases a new Kubernetes minor version.

Examples:

• The latest supported minor Kubernetes version is 1.30. Any patch releases in the 1.29 minor range would be considered for Stable channel updates.
• A new minor Kubernetes version of 1.31 is published. The Stable channel considers any patch release in the 1.30 minor range for updates. Any cluster previously receiving updates from 1.29 is updated to the most recent patch for 1.30.

TargetKubernetesVersion channel

The TargetKubernetesVersion channel gives you control over when to move your clusters to the next Kubernetes minor version. You must specify the target Kubernetes version in the format "{major}.{minor}" (for example, "1.33"). Fleet Manager automatically upgrades clusters to the latest patch release of the specified target Kubernetes version when the patch is available. Fleet Manager doesn't upgrade to the next minor version until the auto-upgrade profile's target Kubernetes version is updated.

Examples:

• You create an auto upgrade profile by using the TargetKubernetesVersion channel and specify a target Kubernetes version of "1.30". A new patch version 1.30.5 is published. An update run is automatically created with the target of 1.30.5.
• You create an auto-upgrade profile by using the TargetKubernetesVersion channel, specify a target Kubernetes version of "1.29", and enable LongTermSupport (LTS) in the auto-upgrade profile. The latest community supported minor version is "1.33". A new patch version 1.29.5 is published. An update run is automatically created with the target of 1.29.5. If the generated update run includes clusters without LTS enabled, it fails.

Minor version skipping behavior

Auto-upgrade doesn't move clusters between minor Kubernetes versions when there's more than one minor Kubernetes version difference (for example: 1.28 to 1.30). When administrators have a diverse set of Kubernetes versions, first use one or more update run to bring clusters into a set of consistently versioned releases so that configured Stable or Rapid channel updates ensure consistency is maintained in future.

NodeImage channel

Member cluster nodes are updated with a newly patched VHD containing security fixes and bug fixes on a weekly cadence. The update to the new VHD is disruptive, following maintenance windows and surge settings. No extra VHD cost is incurred when choosing this option. Node image upgrades support patch versions that are deprecated, so long as the minor Kubernetes version is still supported. Node images are AKS-tested, fully managed, and applied with safe deployment practices.

Nodes on different operating systems are updated in accordance with the node image versions aligned to those operating systems.

Example:

• A cluster has nodes with a NodeImage of AKSWindows-2022-containerd of version 20348.2582.240716. A new NodeImage version 20348.2582.240916 is released and the cluster nodes are automatically upgraded to version 20348.2582.240916.

Understanding node image upgrades and snapshots

When a cluster has agent pools that were created from a node pool snapshot, the outcome of the node image upgrade depends on the Fleet Manager Update Run node image selection.

Understanding Update Strategies

Administrators can control the order in which clusters are updated by building reusable Update Strategies using series of Update Stages and Groups. They can configure when approvals and pauses should occur within those stages and groups. The entire configuration can be saved as an Update Strategy which can be managed independently of Update Runs or Auto-upgrade Profiles, allowing strategies to be reused as required.

Group clusters using member labels (preview)

Member labels can be used to group clusters and configure your update sequence with Kubernetes-style label selectors. This way, you can assign multiple labels to member clusters and use them for different strategies instead of being restricted to a single update group per cluster. You can group clusters in your strategy with their member labels by configuring memberSelector on your strategy at two levels:

• Stage level: Selects clusters for the entire stage. When no groups are defined, all matching clusters form a single implicit group. When groups are also defined, the stage-level selector acts as a pre-filter before group-level matching is applied.
• Group level: Selects clusters for a specific group within a stage, enabling parallel subsets with different concurrency limits.

memberSelector uses a string-based label selector parsed using standard Kubernetes label selector syntax. Supported operators are: =, ==, !=, in, notin, exists, and !exists.

For instructions on assigning member labels and using memberSelector in a strategy, see Create an update strategy using member selectors.

Maximum concurrency (preview)

Maximum concurrency is an optional setting on your update strategy that controls how many clusters can upgrade concurrently. You can set Maximum concurrency at two levels:

• Stage level: Defines the maximum number of clusters that can upgrade at the same time across all groups in a stage. Acts as a global ceiling for the stage.
• Group level: Defines the maximum number of clusters that can upgrade concurrently within a specific group.

Maximum Concurrency accepts two value forms:

• Fixed integer: For example, "3" limits concurrency to exactly three clusters.
• Percentage: For example, "25%" limits concurrency to a percentage of clusters. For stage-level settings, the percentage is calculated from all clusters in the stage. For group-level settings, the percentage is calculated from the clusters in that group. Percentages are calculated at runtime, rounded down, and enforced with a minimum resolved value of 1.

Concurrency control suggestions

If you want to upgrade with safety (less speed, but less likely to end with multiple broken clusters): set maximum concurrency to a smaller value. If you want to upgrade with speed (more speed, but more likely end with multiple broken clusters): set maximum concurrency to a larger value.

How stage and group limits interact

The stage-level Maximum concurrency always acts as the overall ceiling. Even if individual groups allow higher concurrency, the stage limit takes precedence. Group-level concurrency might be lower than configured due to the stage-level limit, group size, or member-specific conditions.

Example 1: Fixed limits

Result: Up to four clusters total, with a maximum of two per group.

Example 2: Stage limit throttles groups

Result: Only two clusters total upgrade at the same time because the stage limit takes precedence.

Example 3: Percentage-based rollout

A stage has 20 clusters across two groups: Group A (eight clusters) and Group B (12 clusters).

Result: Up to five concurrent upgrades total, distributed across groups according to their individual limits.

Maximum allowed failures (preview)

Maximum allowed failures is an optional update strategy setting that controls how Fleet Manager responds to failures during multicluster updates.

By default, updates follow a fail-fast model - a single cluster failure stops further updates. When you configure maximum allowed failures, the update run becomes failure-tolerant, and updates continue across clusters until the specified failure threshold is reached.

This setting provides a deliberate balance between early error detection and maintaining deployment momentum. Set maximum allowed failures at two levels:

• Stage level: Defines the maximum number of member upgrade failures tolerated across all groups in a stage before the stage is marked as failed.
• Group level: Defines the maximum number of member upgrade failures tolerated within a specific group before the group is marked as failed.

Maximum allowed failures accepts two value forms:

• Fixed integer: For example, "3" allows up to three member cluster failures before the group or stage is marked as failed.
• Percentage: For example, "25%" allows failures up to 25% of the member clusters. For stage-level settings, the percentage is calculated from all clusters in the stage. For group-level settings, the percentage is calculated from the clusters in that group. Percentages are resolved when the update run is created, using ceiling rounding, for example, 25% of 5 clusters resolves to 2.

Fleet Manager evaluates maxAllowedFailures only against the count of failed member updates. It doesn't evaluate success rate, and it doesn't require any minimum number of successful members. When this setting is present, Completed means the configured failure threshold wasn't exceeded at the time Fleet Manager made its scheduling decisions. Completed doesn't mean the rollout was healthy or successful.

Example: Completed with 0% success

Suppose a group has four members and group.maxAllowedFailures is set to "4". If all four member updates fail, the group can still be marked Completed. This outcome is expected and intentional, not a bug, because four failures equals the configured tolerance and therefore doesn't exceed the threshold. In other words, a group can be Completed even when 100% of its members failed.

Only use such a threshold when that behavior matches your rollout expectations.

Choose threshold values carefully

• Absolute values are easy to understand, but they can produce counterintuitive results in small groups. For example, allowing "2" failures in a two-member group means the group can finish Completed even if neither member succeeded.
• Percentage values scale better across different group sizes, so they're recommended for most users.
• Avoid setting maxAllowedFailures equal to the total number of members unless you intentionally want effectively "never fail because of member failures" behavior. If that behavior is intentional, 100% usually scales more clearly than a fixed number.
• Be extra careful with small groups, where a single failure can represent a large percentage of the rollout.
• Reevaluate the threshold as your fleet grows. A value that feels safe for five clusters might be too strict or too permissive for 500 clusters.

Understand FailureCount

The FailureCount fields are reporting metrics. They count failed member updates. They aren't ratios or percentages, and they aren't the same as the configured enforcement threshold.

• UpdateRun.FailureCount: Total failed member updates across all stages and all groups in the run.
• Stage.FailureCount: Total failed member updates across all groups within that stage.
• Group.FailureCount: Total failed member updates within that group.

Always review these counts together with member-level statuses, conditions, and failure messages before you decide whether the rollout outcome is acceptable.

Why FailureCount can exceed maxAllowedFailures

maxAllowedFailures controls Fleet Manager's decision-making about whether to continue scheduling new work. It isn't a hard upper bound on the final reported number of failures.

If you allow parallel member updates by using maxConcurrency, several members can fail at nearly the same time before Fleet Manager observes that the threshold is exceeded and stops scheduling more work. As a result, it's possible and expected for FailureCount to be greater than the configured maxAllowedFailures value.

How stage and group failure limits interact

Group-level and stage-level maxAllowedFailures are evaluated independently:

• Each group tracks its own failure count against its own threshold.
• Those same group failures also roll up into the stage's FailureCount.
• If either threshold is exceeded, Fleet Manager stops scheduling new work for that segment.
• Already-started member updates might still finish, which can increase the final reported FailureCount after the stop decision is made.

The stage-level maxAllowedFailures acts as the overall ceiling for failure tolerance in a stage. Even if individual groups allow higher failure counts, the stage limit takes precedence. If a group's failure count exceeds its own maxAllowedFailures, that group is marked as failed regardless of the stage-level setting.

Example 1: Fixed failure limits

Result: Group A tolerates up to two failures and Group B tolerates up to three. If total failures across the stage exceed five, the stage fails.

Example 2: Percentage-based tolerance

A stage has 19 clusters across two groups: Group A with 7 clusters and Group B with 12 clusters.

Recommended and nonrecommended uses

Use higher or percentage-based thresholds when you're upgrading large fleets, you expect some transient failures, or rollout progress is more important than strict fail-fast behavior.

Use 0 or very small values for safety-critical rollouts, tightly controlled production stages, or any situation where you must stop and inspect after the first failure.

Next steps

• How-to: Upgrade multiple clusters using Azure Kubernetes Fleet Manager update runs.
• How-to: Automatically upgrade multiple clusters using Azure Kubernetes Fleet Manager.
• How-to: Monitor update runs for Azure Kubernetes Fleet Manager.
• Multi-cluster updates FAQs.