Propagate reserved resources from an Azure Kubernetes Fleet Manager hub cluster to member clusters

This article provides an overview of how to use envelope objects to propagate reserved Kubernetes resource types from an Azure Kubernetes Fleet Manager hub cluster to member clusters.

Envelope Object with ConfigMap

Currently, we support using a ConfigMap as an envelope object by leveraging a fleet-reserved annotation.

To designate a ConfigMap as an envelope object, ensure that it contains the following annotation:

metadata:
  annotations:
    kubernetes-fleet.io/envelope-configmap: "true"

Example ConfigMap Envelope Object

apiVersion: v1
kind: ConfigMap
metadata:
    name: envelope-configmap
    namespace: app
    annotations:
        kubernetes-fleet.io/envelope-configmap: "true"
data:
    resourceQuota.yaml: |
        apiVersion: v1
        kind: ResourceQuota
        metadata:
            name: mem-cpu-demo
            namespace: app
        spec:
            hard:
                requests.cpu: "1"
                requests.memory: 1Gi
                limits.cpu: "2"
                limits.memory: 2Gi
    webhook.yaml: |
        apiVersion: admissionregistration.k8s.io/v1
        kind: MutatingWebhookConfiguration
        metadata:
            creationTimestamp: null
            labels:
                azure-workload-identity.io/system: "true"
            name: azure-wi-webhook-mutating-webhook-configuration
        webhooks:
        - admissionReviewVersions:
          - v1
          - v1beta1
          clientConfig:
              service:
                  name: azure-wi-webhook-webhook-service
                  namespace: app
                  path: /mutate-v1-pod
          failurePolicy: Fail
          matchPolicy: Equivalent
          name: mutation.azure-workload-identity.io
          rules:
          - apiGroups:
              - ""
              apiVersions:
              - v1
              operations:
              - CREATE
              - UPDATE
              resources:
              - pods
          sideEffects: None

Propagating an Envelope ConfigMap to member clusters

We now apply the example envelope object above on our hub cluster. Then we use a ClusterResourcePlacement object to propagate the resource from hub to a member cluster named kind-cluster-1.

Here's a sample ClusterResourcePlacement (CRP) specification.

spec:
    policy:
        clusterNames:
        - kind-cluster-1
        placementType: PickFixed
    resourceSelectors:
    - group: ""
        kind: Namespace
        name: app
        version: v1
    revisionHistoryLimit: 10
    strategy:
        type: RollingUpdate

Retrieve the status of Envelope ConfigMap placement

Here's a sample status that shows the successful placement of an Enveloped Object.

status:
conditions:
- lastTransitionTime: "2023-11-30T19:54:13Z"
  message: found all the clusters needed as specified by the scheduling policy
  observedGeneration: 2
  reason: SchedulingPolicyFulfilled
  status: "True"
  type: ClusterResourcePlacementScheduled
- lastTransitionTime: "2023-11-30T19:54:18Z"
  message: All 1 cluster(s) are synchronized to the latest resources on the hub
  cluster
  observedGeneration: 2
  reason: SynchronizeSucceeded
  status: "True"
  type: ClusterResourcePlacementSynchronized
- lastTransitionTime: "2023-11-30T19:54:18Z"
  message: Successfully applied resources to 1 member clusters
  observedGeneration: 2
  reason: ApplySucceeded
  status: "True"
  type: ClusterResourcePlacementApplied
  placementStatuses:
- clusterName: kind-cluster-1
  conditions:
    - lastTransitionTime: "2023-11-30T19:54:13Z"
      message: 'Successfully scheduled resources for placement in kind-cluster-1:
      picked by scheduling policy'
      observedGeneration: 2
      reason: ScheduleSucceeded
      status: "True"
      type: ResourceScheduled
    - lastTransitionTime: "2023-11-30T19:54:18Z"
      message: Successfully Synchronized work(s) for placement
      observedGeneration: 2
      reason: WorkSynchronizeSucceeded
      status: "True"
      type: WorkSynchronized
    - lastTransitionTime: "2023-11-30T19:54:18Z"
      message: Successfully applied resources
      observedGeneration: 2
      reason: ApplySucceeded
      status: "True"
      type: ResourceApplied
      selectedResources:
- kind: Namespace
  name: app
  version: v1
- kind: ConfigMap
  name: envelope-configmap
  namespace: app
  version: v1

Note

In the selectedResources section, we specifically display the propagated envelope object. Please note that we do not individually list all the resources contained within the envelope object in the status.

Upon inspection of the selectedResources, it indicates that the namespace app and the configmap envelope-configmap have been successfully propagated. Users can further verify the successful propagation of resources mentioned within the envelope-configmap object by ensuring that the failedPlacements section in the placementStatus for kind-cluster-1 doesn't appear in the status.

Here's a sample where the placement has failed. In this example below, within the placementStatus section for kind-cluster-1, the failedPlacements section provides details on resource that failed to apply along with information about the envelope object which contained the resource.

status:
conditions:
- lastTransitionTime: "2023-12-06T00:09:53Z"
  message: found all the clusters needed as specified by the scheduling policy
  observedGeneration: 2
  reason: SchedulingPolicyFulfilled
  status: "True"
  type: ClusterResourcePlacementScheduled
- lastTransitionTime: "2023-12-06T00:09:58Z"
  message: All 1 cluster(s) are synchronized to the latest resources on the hub
  cluster
  observedGeneration: 2
  reason: SynchronizeSucceeded
  status: "True"
  type: ClusterResourcePlacementSynchronized
- lastTransitionTime: "2023-12-06T00:09:58Z"
  message: Failed to apply manifests to 1 clusters, please check the `failedPlacements`
  status
  observedGeneration: 2
  reason: ApplyFailed
  status: "False"
  type: ClusterResourcePlacementApplied
  placementStatuses:
- clusterName: kind-cluster-1
  conditions:
    - lastTransitionTime: "2023-12-06T00:09:53Z"
      message: 'Successfully scheduled resources for placement in kind-cluster-1:
      picked by scheduling policy'
      observedGeneration: 2
      reason: ScheduleSucceeded
      status: "True"
      type: ResourceScheduled
    - lastTransitionTime: "2023-12-06T00:09:58Z"
      message: Successfully Synchronized work(s) for placement
      observedGeneration: 2
      reason: WorkSynchronizeSucceeded
      status: "True"
      type: WorkSynchronized
    - lastTransitionTime: "2023-12-06T00:09:58Z"
      message: Failed to apply manifests, please check the `failedPlacements` status
      observedGeneration: 2
      reason: ApplyFailed
      status: "False"
      type: ResourceApplied
      failedPlacements:
    - condition:
      lastTransitionTime: "2023-12-06T00:09:53Z"
      message: 'Failed to apply manifest: namespaces "app" not found'
      reason: AppliedManifestFailedReason
      status: "False"
      type: Applied
      envelope:
      name: envelop-configmap
      namespace: test-ns
      type: ConfigMap
      kind: ResourceQuota
      name: mem-cpu-demo
      namespace: app
      version: v1
      selectedResources:
- kind: Namespace
  name: test-ns
  version: v1
- kind: ConfigMap
  name: envelop-configmap
  namespace: test-ns
  version: v1