Configure and use cross-cluster networking for Azure Kubernetes Fleet Manager (preview)

Applies to: ✔️ Fleet Manager ✔️ Fleet Manager with hub cluster

Cross-cluster networking for Azure Kubernetes Fleet Manager provides delivers managed Cilium multi-cluster that extends Cilium's eBPF-based networking and observability across multiple clusters.

Fleet Manager supports creation of multiple cross-cluster networking profiles with each supporting up to 255 member clusters. Clusters in a profile participate in a managed federated network with access to global services and service discovery and observability via Hubble.

Platform engineers label member clusters that should participate in a single cross-cluster network, specifying the label in a selector when defining the network profile. To allow for bulk membership management and avoid unintended interruptions, the cross-cluster network is only updated when explicitly requested by an authorized user.

This article provides instructions on how to configure a cross-cluster networking profile, add member clusters and how to expose global services.

Prerequisites and limitations

• A cross-cluster network can have up to 255 member clusters.
• Fleet Manager member clusters can only participate in a single cross-cluster network at any time.
• Clusters must run Kubernetes v1.32 or above and have Advanced Container Networking Services (ACNS) with Cilium enabled.
• Clusters must be connected to a single flat network (virtual network or multiple peered networks).
• Overlay networking with tunnels isn't supported.
• Self-managed Cilium multi-cluster can't be deployed at the same time.
• ACNS sets the Cilium version and enabled features which are read-only and can't be modified by users.

Before you begin

If you don't have an Azure trail subscription, create a trial subscription before you begin.

Create two member AKS clusters mbr-aks-member-1 and mbr-aks-member-2 with Advanced Container Networking Services (ACNS) enabled.

Set the following environment variables:

export GROUP=<resource-group>
export FLEET=<fleet-name>
export MEMBER_CLUSTER_1=mbr-aks-member-1
export MEMBER_CLUSTER_2=mbr-aks-member-2
export TRAFFIC_MANAGER_GROUP=rg-flt-tm-demo
export NETWORK_NAME=demo-network
export NETWORK_PROFILE_NAME=fccnp-demo-01

Label the member clusters

Label both member clusters so they're configured to be included in the cross-cluster network.

az fleet member update \
    --fleet-name ${FLEET} \
    --resource-group ${GROUP} \
    --name ${MEMBER_CLUSTER_1} \
    --labels "mesh=${NETWORK_NAME}"

az fleet member update \
    --fleet-name ${FLEET} \
    --resource-group ${GROUP} \
    --name ${MEMBER_CLUSTER_2} \
    --labels "mesh=${NETWORK_NAME}"

Create a cross-cluster network profile

Create a cross-cluster network profile using the az fleet clustermeshprofile create command.

az fleet clustermeshprofile create \
    --fleet-name ${FLEET} \
    --resource-group ${GROUP} \
    --name ${NETWORK_PROFILE_NAME} \
    --member-label-selector "mesh=${NETWORK_NAME}"

While a network profile is created as an Azure Resource, no Cilium multi-cluster configuration is yet applied on clusters. The following steps still need to be performed to apply and connect the cross-cluster network.

Validate the selected clusters

Validate which clusters will be included in the cross-cluster network by supplying the whatif parameter to the az fleet clustermeshprofile apply command.

az fleet clustermeshprofile apply \
    --what-if \
    --fleet-name ${FLEET} \
    --resource-group ${GROUP} \
    --name ${NETWORK_PROFILE_NAME} \
    --output table

As we're creating a new cross-cluster network, both clusters have an Add action listed.

ClusterResourceId    	            ETag        Name  		       Action   MeshMembershipState
----------------------------------  ----------  -----------------  -------  -------------------
/subscription/…/…/mbr-aks-member-1  "fd009cd9"  mbr-aks-member-1   Add       -
/subscription/…/…/mbr-aks-member-2  "a400f86e"  mbr-aks-member-2   Add       -

Connect the cross-cluster network

You can now apply the cross-cluster networking changes by omitting the what-if parameter from the az fleet clustermeshprofile apply command.

az fleet clustermeshprofile apply \
    --fleet-name ${FLEET} \
    --resource-group ${GROUP} \
    --name ${NETWORK_PROFILE_NAME} 

The cross-cluster network creation starts and Cilium multi-cluster configuration is applied to selected member clusters. The creation process is asynchronous, with the process duration determined by the number of clusters being updated.

The apply operation runs to completion and can't be interrupted. While it's running, you can monitor the network status of each member:

az fleet clustermeshprofile list-members \
    --fleet-name ${FLEET} \
    --resource-group ${GROUP} \
    --name ${NETWORK_PROFILE_NAME} \
    --query "[].{name: name, state: meshProperties.status.state}" \
    -o table

Name              State
----------------  ----------
mbr-aks-member-1  Connecting
mbr-aks-member-2  Connecting

You can also monitor the status of the overall operation:

az fleet clustermeshprofile show \
    --fleet-name ${FLEET} \
    --resource-group ${GROUP} \
    --name ${NETWORK_PROFILE_NAME} \
    --query "properties.status.state" \
    -o tsv

Confirm cross-cluster network connection via the dataplane

Once the members' states are showing as Connected, the cross-cluster network is established.

You can confirm the successful connection using standard dataplane tools such as the Cilium CLI. First, obtain the Kubernetes access credentials for both member clusters using az aks get-credentials, setting context appropriately.

az aks get-credentials \
    --resource-group $RESOURCE_GROUP \
    --name $AKS_CLUSTER_1 \
    --context cluster1

Then, use the Cilium CLI's status command to see that all member clusters are connected:

cilium clustermesh status --context cluster1

✅ Cluster access information is available:
  - 10.168.0.89:2379
✅ Service "clustermesh-apiserver" of type "LoadBalancer" found
✅ All 2 nodes are connected to all clusters [min:1 / avg:1.0 / max:1]
🔌 Cluster Connections:
- mbr-aks-member-2: 2/2 configured, 2/2 connected

Test load balancing and service discovery

Once the cross-cluster network is created successfully, you can test load balancing out by following the official Cilium multi-cluster example, or using the steps shown next. The steps in this document provide extra guidance on working with AKS clusters and Fleet Manager.

• On mbr-aks-member-1 create a dedicated namespace and annotate it so Services within it are eligible to be shared across the cross-cluster network. Then apply the Deployment and Service resources using the cluster1.yaml manifest.

  kubectl --context=cluster1 create namespace rebel-base-demo
  kubectl --context=cluster1 annotate namespace rebel-base-demo clustermesh.cilium.io/global="true" --overwrite
  kubectl --context=cluster1 -n rebel-base-demo apply -f https://raw.githubusercontent.com/cilium/cilium/refs/heads/main/examples/kubernetes/clustermesh/cluster1.yaml
  kubectl --context=cluster1 -n rebel-base-demo apply -f https://raw.githubusercontent.com/cilium/cilium/refs/heads/main/examples/kubernetes/clustermesh/global-service-example.yaml
  

• On mbr-aks-member-2 apply the equivalent, but this time use the cluster2.yaml manifest.

  kubectl --context=cluster2 create namespace rebel-base-demo
  kubectl --context=cluster2 annotate namespace rebel-base-demo clustermesh.cilium.io/global="true" --overwrite
  kubectl --context=cluster2 -n rebel-base-demo apply -f https://raw.githubusercontent.com/cilium/cilium/refs/heads/main/examples/kubernetes/clustermesh/cluster2.yaml
  kubectl --context=cluster2 -n rebel-base-demo apply -f https://raw.githubusercontent.com/cilium/cilium/refs/heads/main/examples/kubernetes/clustermesh/global-service-example.yaml
  

• Run the following command multiple times on each cluster. Observe the serving cluster changes, demonstrating the request is being served by multiple clusters despite calling the service on only one.

  kubectl --context=cluster1 -n rebel-base-demo exec -ti deployment/x-wing -- curl rebel-base
  

  {"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
  {"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
  {"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
  

• Set the clustermesh.cilium.io/global annotation on the rebel-base-demo Namespace on mbr-aks-member-1 (cluster 1) to "false" so Services in that Namespace are no longer shared across the cross-cluster network from cluster 1.

  kubectl --context=cluster1 annotate namespace rebel-base-demo clustermesh.cilium.io/global="false" --overwrite
  

• Let's validate mbr-aks-member-1 (cluster 1) can reach the rebel-base Service on cluster 1, and mbr-aks-member-2 (cluster 2) can't.

  kubectl --context=cluster1 -n rebel-base-demo exec -ti deployment/x-wing -- curl rebel-base
  

  On cluster 1, we only see local responses because the Namespace is no longer global and remote endpoints aren't imported.

  {"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
  {"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
  {"Galaxy": "Alderaan", "Cluster": "Cluster-1"}
  

  kubectl --context=cluster2 -n rebel-base-demo exec -ti deployment/x-wing -- curl rebel-base
  

  On cluster 2, we only see local responses because the Service on cluster 1 is no longer shared.

  {"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
  {"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
  {"Galaxy": "Alderaan", "Cluster": "Cluster-2"}
  

Updating a cross-cluster network

The process of adding or removing clusters is demonstrated in this guide, but can be summarized as:

1. Modify labels on the Fleet Manager member clusters to be added or removed.
2. Review cross-cluster networking changes by using the whatif parameter with the az fleet clustermeshprofile apply command.
3. Once satisfied with the changes, apply them by running the same command, omitting the whatif parameter.

Reviewing the changes is optional, but recommended, especially for larger cross-cluster networks where any change can take some time to complete.

Next steps

• Overview of Fleet Manager multi-cluster networking.
• Fleet Manager Frequently Asked Questions (FAQs).