Propagate reserved resources from an Azure Kubernetes Fleet Manager hub cluster to member clusters

This article provides an overview of how to use envelope objects to propagate reserved Kubernetes resource types from an Azure Kubernetes Fleet Manager (Kubernetes Fleet) hub cluster to member clusters.

You can designate a ConfigMap as an envelope object by using the following Kubernetes Fleet reserved annotation:

metadata:
  annotations:
    kubernetes-fleet.io/envelope-configmap: "true"

Here's an example of using ConfigMap as an envelope object:

apiVersion: v1
kind: ConfigMap
metadata:
    name: envelope-configmap
    namespace: app
    annotations:
        kubernetes-fleet.io/envelope-configmap: "true"
data:
    resourceQuota.yaml: |
        apiVersion: v1
        kind: ResourceQuota
        metadata:
            name: mem-cpu-demo
            namespace: app
        spec:
            hard:
                requests.cpu: "1"
                requests.memory: 1Gi
                limits.cpu: "2"
                limits.memory: 2Gi
    webhook.yaml: |
        apiVersion: admissionregistration.k8s.io/v1
        kind: MutatingWebhookConfiguration
        metadata:
            creationTimestamp: null
            labels:
                azure-workload-identity.io/system: "true"
            name: azure-wi-webhook-mutating-webhook-configuration
        webhooks:
        - admissionReviewVersions:
          - v1
          - v1beta1
          clientConfig:
              service:
                  name: azure-wi-webhook-webhook-service
                  namespace: app
                  path: /mutate-v1-pod
          failurePolicy: Fail
          matchPolicy: Equivalent
          name: mutation.azure-workload-identity.io
          rules:
          - apiGroups:
              - ""
              apiVersions:
              - v1
              operations:
              - CREATE
              - UPDATE
              resources:
              - pods
          sideEffects: None

Apply the preceding example envelope object on your hub cluster. Then, use a ClusterResourcePlacement object to propagate the resource from the hub cluster to a member cluster named kind-cluster-1.

Here's a sample ClusterResourcePlacement specification:

spec:
    policy:
        clusterNames:
        - kind-cluster-1
        placementType: PickFixed
    resourceSelectors:
    - group: ""
        kind: Namespace
        name: app
        version: v1
    revisionHistoryLimit: 10
    strategy:
        type: RollingUpdate

Here's a sample status that shows the successful placement of an envelope object:

status:
conditions:
- lastTransitionTime: "2023-11-30T19:54:13Z"
  message: found all the clusters needed as specified by the scheduling policy
  observedGeneration: 2
  reason: SchedulingPolicyFulfilled
  status: "True"
  type: ClusterResourcePlacementScheduled
- lastTransitionTime: "2023-11-30T19:54:18Z"
  message: All 1 cluster(s) are synchronized to the latest resources on the hub
  cluster
  observedGeneration: 2
  reason: SynchronizeSucceeded
  status: "True"
  type: ClusterResourcePlacementSynchronized
- lastTransitionTime: "2023-11-30T19:54:18Z"
  message: Successfully applied resources to 1 member clusters
  observedGeneration: 2
  reason: ApplySucceeded
  status: "True"
  type: ClusterResourcePlacementApplied
  placementStatuses:
- clusterName: kind-cluster-1
  conditions:
    - lastTransitionTime: "2023-11-30T19:54:13Z"
      message: 'Successfully scheduled resources for placement in kind-cluster-1:
      picked by scheduling policy'
      observedGeneration: 2
      reason: ScheduleSucceeded
      status: "True"
      type: ResourceScheduled
    - lastTransitionTime: "2023-11-30T19:54:18Z"
      message: Successfully Synchronized work(s) for placement
      observedGeneration: 2
      reason: WorkSynchronizeSucceeded
      status: "True"
      type: WorkSynchronized
    - lastTransitionTime: "2023-11-30T19:54:18Z"
      message: Successfully applied resources
      observedGeneration: 2
      reason: ApplySucceeded
      status: "True"
      type: ResourceApplied
      selectedResources:
- kind: Namespace
  name: app
  version: v1
- kind: ConfigMap
  name: envelope-configmap
  namespace: app
  version: v1

The selectedResources section indicates that the namespace app and the ConfigMap named envelope-configmap were successfully propagated. You can further verify the successful propagation of resources mentioned within the envelope-configmap object by ensuring that the failedPlacements section in placementStatus for kind-cluster-1 doesn't appear in the status.

Here's an example where the placement failed. In this example, within the placementStatus section for kind-cluster-1, the failedPlacements section provides details on the resource that failed to apply. The failedPlacements section also provides information about the envelope object that contained the resource.

status:
conditions:
- lastTransitionTime: "2023-12-06T00:09:53Z"
  message: found all the clusters needed as specified by the scheduling policy
  observedGeneration: 2
  reason: SchedulingPolicyFulfilled
  status: "True"
  type: ClusterResourcePlacementScheduled
- lastTransitionTime: "2023-12-06T00:09:58Z"
  message: All 1 cluster(s) are synchronized to the latest resources on the hub
  cluster
  observedGeneration: 2
  reason: SynchronizeSucceeded
  status: "True"
  type: ClusterResourcePlacementSynchronized
- lastTransitionTime: "2023-12-06T00:09:58Z"
  message: Failed to apply manifests to 1 clusters, please check the `failedPlacements`
  status
  observedGeneration: 2
  reason: ApplyFailed
  status: "False"
  type: ClusterResourcePlacementApplied
  placementStatuses:
- clusterName: kind-cluster-1
  conditions:
    - lastTransitionTime: "2023-12-06T00:09:53Z"
      message: 'Successfully scheduled resources for placement in kind-cluster-1:
      picked by scheduling policy'
      observedGeneration: 2
      reason: ScheduleSucceeded
      status: "True"
      type: ResourceScheduled
    - lastTransitionTime: "2023-12-06T00:09:58Z"
      message: Successfully Synchronized work(s) for placement
      observedGeneration: 2
      reason: WorkSynchronizeSucceeded
      status: "True"
      type: WorkSynchronized
    - lastTransitionTime: "2023-12-06T00:09:58Z"
      message: Failed to apply manifests, please check the `failedPlacements` status
      observedGeneration: 2
      reason: ApplyFailed
      status: "False"
      type: ResourceApplied
      failedPlacements:
    - condition:
      lastTransitionTime: "2023-12-06T00:09:53Z"
      message: 'Failed to apply manifest: namespaces "app" not found'
      reason: AppliedManifestFailedReason
      status: "False"
      type: Applied
      envelope:
      name: envelop-configmap
      namespace: test-ns
      type: ConfigMap
      kind: ResourceQuota
      name: mem-cpu-demo
      namespace: app
      version: v1
      selectedResources:
- kind: Namespace
  name: test-ns
  version: v1
- kind: ConfigMap
  name: envelop-configmap
  namespace: test-ns
  version: v1