Set up multi-cluster layer 4 load balancing across Azure Kubernetes Fleet Manager member clusters (preview)

For applications deployed across multiple clusters, admins often want to route incoming traffic to them across clusters.

You can follow this document to set up layer 4 load balancing for such multi-cluster applications.

Important

Azure Kubernetes Fleet Manager preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. Azure Kubernetes Fleet Manager previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use.

Prerequisites

If you don't have an Azure trail subscription, create a trial subscription before you begin.

You can use the local Azure CLI.

Deploy a workload across member clusters of the Fleet resource

Note

  • The steps in this how-to guide refer to a sample application for demonstration purposes only. You can substitute this workload for any of your own existing Deployment and Service objects.

  • These steps deploy the sample workload from the Fleet cluster to member clusters using Kubernetes configuration propagation. Alternatively, you can choose to deploy these Kubernetes configurations to each member cluster separately, one at a time.

  1. Create a namespace on the fleet cluster:

    KUBECONFIG=fleet kubectl create namespace kuard-demo
    

    Output looks similar to the following example:

    namespace/kuard-demo created
    
  2. Apply the Deployment, Service, ServiceExport objects:

    KUBECONFIG=fleet kubectl apply -f https://raw.githubusercontent.com/Azure/AKS/master/examples/fleet/kuard/kuard-export-service.yaml
    

    The ServiceExport specification in the above file allows you to export a service from member clusters to the Fleet resource. Once successfully exported, the service and all its endpoints are synced to the fleet cluster and can then be used to set up multi-cluster load balancing across these endpoints. The output looks similar to the following example:

    deployment.apps/kuard created
    service/kuard created
    serviceexport.networking.fleet.azure.com/kuard created
    
  3. Create the following ClusterResourcePlacement in a file called crp-2.yaml. Notice we're selecting clusters in the chinanorth3 region:

    apiVersion: placement.kubernetes-fleet.io/v1beta1
    kind: ClusterResourcePlacement
    metadata:
      name: kuard-demo
    spec:
      resourceSelectors:
        - group: ""
          version: v1
          kind: Namespace
          name: kuard-demo
      policy:
        affinity:
          clusterAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              clusterSelectorTerms:
                - labelSelector:
                    matchLabels:
                    fleet.azure.cn/location: chinanorth
    
  4. Apply the ClusterResourcePlacement:

    KUBECONFIG=fleet kubectl apply -f crp-2.yaml
    

    If successful, the output looks similar to the following example:

    clusterresourceplacement.fleet.azure.com/kuard-demo created
    
  5. Check the status of the ClusterResourcePlacement:

    KUBECONFIG=fleet kubectl get clusterresourceplacements
    

    If successful, the output looks similar to the following example:

    NAME            GEN   SCHEDULED   SCHEDULEDGEN   APPLIED   APPLIEDGEN   AGE
    kuard-demo      1     True        1              True      1            20s
    

Create MultiClusterService to load balance across the service endpoints in multiple member clusters

  1. Check whether the service is successfully exported for the member clusters in eastus region:

    KUBECONFIG=aks-member-1 kubectl get serviceexport kuard --namespace kuard-demo
    

    Output looks similar to the following example:

    NAME    IS-VALID   IS-CONFLICTED   AGE
    kuard   True       False           25s
    
    KUBECONFIG=aks-member-2 kubectl get serviceexport kuard --namespace kuard-demo
    

    Output looks similar to the following example:

    NAME    IS-VALID   IS-CONFLICTED   AGE
    kuard   True       False           55s
    

    You should see that the service is valid for export (IS-VALID field is true) and has no conflicts with other exports (IS-CONFLICT is false).

    Note

    It may take a minute or two for the ServiceExport to be propagated.

  2. Create MultiClusterService on one member to load balance across the service endpoints in these clusters:

    KUBECONFIG=aks-member-1 kubectl apply -f https://raw.githubusercontent.com/Azure/AKS/master/examples/fleet/kuard/kuard-mcs.yaml
    

    Note

    To expose the service via the internal IP instead of public one, add the annotation to the MultiClusterService:

    apiVersion: networking.fleet.azure.com/v1alpha1
    kind: MultiClusterService
    metadata:
      name: kuard
      namespace: kuard-demo
      annotations:
         service.beta.kubernetes.io/azure-load-balancer-internal: "true"
      ...
    

    Output looks similar to the following example:

    multiclusterservice.networking.fleet.azure.com/kuard created
    
  3. Verify the MultiClusterService is valid by running the following command:

    KUBECONFIG=aks-member-1 kubectl get multiclusterservice kuard --namespace kuard-demo
    

    The output should look similar to the following example:

    NAME    SERVICE-IMPORT   EXTERNAL-IP     IS-VALID   AGE
    kuard   kuard            <a.b.c.d>       True       40s
    

    The IS-VALID field should be true in the output. Check out the external load balancer IP address (EXTERNAL-IP) in the output. It may take a while before the import is fully processed and the IP address becomes available.

  4. Run the following command multiple times using the external load balancer IP address:

    curl <a.b.c.d>:8080 | grep addrs 
    

    Notice that the IPs of the pods serving the request is changing and that these pods are from member clusters aks-member-1 and aks-member-2 from the chinanorth3 region. You can verify the pod IPs by running the following commands on the clusters from chinanorth region:

    KUBECONFIG=aks-member-1 kubectl get pods -n kuard-demo -o wide
    
    KUBECONFIG=aks-member-2 kubectl get pods -n kuard-demo -o wide