Use deployment safeguards to enforce best practices in Azure Kubernetes Service (AKS) (Preview)

  • Article

This article shows you how to use deployment safeguards to enforce best practices on an Azure Kubernetes Service (AKS) cluster.

Overview

Throughout the development lifecycle, it's common for bugs, issues, and other problems to arise if the initial deployment of your Kubernetes resources includes misconfigurations. To ease the burden of Kubernetes development, Azure Kubernetes Service (AKS) offers deployment safeguards (preview). Deployment safeguards enforce Kubernetes best practices in your AKS cluster through Azure Policy controls.

Deployment safeguards offer two levels of configuration:

  • Warning: Displays warning messages in the code terminal to alert you of any noncompliant cluster configurations but still allows the request to go through.
  • Enforcement: Enforces compliant configurations by denying and mutating deployments if they don't follow best practices.

After you configure deployment safeguards for 'Warning' or 'Enforcement', Deployment safeguards programmatically assess your clusters at creation or update time for compliance. Deployment safeguards also display aggregated compliance information across your workloads at a per resource level via Azure Policy's compliance dashboard in the Azure portal or in your CLI or terminal. Running a noncompliant workload indicates that your cluster isn't following best practices and that workloads on your cluster are at risk of experiencing issues caused by your cluster configuration.

Important

AKS preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. AKS previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use. For more information, see the following support articles:

Prerequisites

Install the aks-preview CLI extension

  1. Install the aks-preview CLI extension using the az extension add command.

    az extension add --name aks-preview

  2. Update the extension to ensure you have the latest version installed using the az extension update command.

    az extension update --name aks-preview

Register the deployment safeguards feature flag

  1. Register the SafeguardsPreview feature flag using the az feature register command.

    az feature register --namespace Microsoft.ContainerService --name SafeguardsPreview

    It takes a few minutes for the status to show Registered.

  2. Verify the registration status using the az feature show command.

    az feature show --namespace Microsoft.ContainerService --name SafeguardsPreview

  3. When the status reflects Registered, refresh the registration of the Microsoft.ContainerService resource provider using the az provider register command.

    az provider register --namespace Microsoft.ContainerService

Deployment safeguards policies

Note

The ReadOnlyRootFilesystem and RootfilesystemInitContainers policies are currently only available on Linux.

The following table lists the policies that become active and the Kubernetes resources they target when you enable deployment safeguards. You can view the currently available deployment safeguards in the Azure portal as an Azure Policy definition or at Azure Policy built-in definitions for Azure Kubernetes Service. The intention behind this collection is to create a common and generic list of best practices applicable to most users and use cases.

Deployment safeguard policy Targeted Kubernetes resource Mutation outcome if available
[Preview]: Cannot Edit Individual Nodes Node N/A
Kubernetes cluster containers CPU and memory resource limits shouldn't exceed the specified limits Pod Sets CPU resource limits to 500m if not set and sets memory limits to 500Mi if no path is present
[Preview]: Must Have Anti Affinity Rules Set Deployment, StatefulSet, ReplicationController, ReplicaSet N/A
[Preview]: No AKS Specific Labels Deployment, StatefulSet, Replicaset N/A
Kubernetes cluster containers should only use allowed images Pod N/A
[Preview]: Reserved System Pool Taints Node Removes the CriticalAddonsOnly taint from a user node pool if not set. AKS uses the CriticalAddonsOnly taint to keep customer pods away from the system pool. This configuration ensures a clear separation between AKS components and customer pods and prevents eviction of customer pods that don't tolerate the CriticalAddonsOnly taint.
Ensure cluster containers have readiness or liveness probes configured Pod N/A
Kubernetes clusters should use Container Storage Interface (CSI) driver StorageClass StorageClass N/A
[Preview]: Kubernetes cluster containers should only pull images when image pull secrets are present Pod N/A
[Preview]: Kubernetes cluster should implement accurate Pod Disruption Budgets Deployment, ReplicaSet, StatefulSet Sets maxUnavailable in PodDisruptionBudget resource to 1.
[Preview]: Kubernetes cluster services should use unique selectors Service N/A
[Preview]: ReadOnlyRootFilesystem in Pod spec is set to true Pod Sets readOnlyRootFilesystem in the Pod spec to true if not set. This configuration prevents containers from writing to the root filesystem.
[Preview]: RootfilesystemInitContainers in Pod spec is set to true Pod Sets rootFilesystemInitContainers in the Pod spec to true if not set.
[Preview]: Kubernetes cluster container images should not include latest image tag Deployment, StatefulSet, ReplicationController, ReplicaSet N/A
[Preview]: Kubernetes cluster container images must include the preStop hook Deployment, StatefulSet, ReplicationController, ReplicaSet N/A

If you want to submit an idea or request for deployment safeguards, open an issue in the AKS GitHub repository and add [deployment safeguards request] to the beginning of the title.

Enable deployment safeguards

Note

If you enabled Azure Policy for the first time to use deployment safeguards, you might need to wait up to 20 minutes for Azure Policy to take effect.

Using the deployment safeguards Enforcement level means you're opting in to deployments being blocked and mutated. Please consider how these policies might work with your AKS cluster before enabling Enforcement.

Enable deployment safeguards on a new cluster

Enable deployment safeguards on a new cluster using the az aks create command with the --safeguards-level and --safeguards-version flags.

If you want to receive noncompliance warnings, set the --safeguards-level to Warning. If you want to deny or mutate all noncompliant deployments, set it to Enforcement. To receive warnings, set the --safeguards-level to "Warning". To deny or mutate all deployments that don't adhere to deployment safeguards, set the --safeguards-level to "Enforcement". To set the deployment safeguards version, use the --safeguards-version flag. Currently, V2.0.0 is the latest version of deployment safeguards.

az aks create \
    --name myAKSCluster \
    --resource-group myResourceGroup \
    --enable-addons azure-policy \
    --safeguards-level Warning \
    --safeguards-version v2.0.0 \
    --generate-ssh-keys

Enable deployment safeguards on an existing cluster

Enable deployment safeguards on an existing cluster that has the Azure Policy add-on enabled using the az aks update command with the --safeguards-level and --safeguards-version flags. If you want to receive noncompliance warnings, set the --safeguards-level to Warning. If you want to deny or mutate all noncompliant deployments, set it to Enforcement.

az aks update --name myAKSCluster --resource-group myResourceGroup --safeguards-level Enforcement --safeguards-version v2.0.0

If you want to update the deployment safeguards level of an existing cluster, use the az aks update command with the --safeguards-level flag set to Warning or Enforcement.

az aks update --name myAKSCluster --resource-group myResourceGroup --safeguards-level Enforcement

Excluding namespaces

You can also exclude certain namespaces from deployment safeguards. When you exclude a namespace, activity in that namespace is unaffected by deployment safeguards warnings or enforcement.

For example, to exclude the namespaces ns1 and ns2, use a comma-separated list of namespaces with the --safeguards-excluded-ns flag, as shown in the following example:

az aks update --name myAKSCluster --resource-group myResourceGroup --safeguards-level Warning --safeguards-version v2.0.0 --safeguards-excluded-ns ns1,ns2

Update your deployment safeguard version

Note

v2.0.0 is the latest version of deployment safeguards.

Update your deployment safeguards version using the az aks update command with the --safeguards-version flag set to the new version. The following example updates an existing cluster to use version 2.0.0:

az aks update --name myAKSCluster --resource-group myResourceGroup --safeguards-version v2.0.0

Verify compliance across clusters

After deploying your Kubernetes manifest, you see warnings or a potential denial message in your CLI or terminal if the cluster isn't compliant with deployment safeguards, as shown in the following examples:

Warning

$ kubectl apply -f pod.yml
Warning: [azurepolicy-k8sazurev2containerenforceprob-0e8a839bcd103e7b96a8] Container <my-container> in your Pod <my-pod> has no <livenessProbe>. Required probes: ["readinessProbe", "livenessProbe"]
Warning: [azurepolicy-k8sazurev2containerenforceprob-0e8a839bcd103e7b96a8] Container <my-container> in your Pod <my-pod> has no <readinessProbe>. Required probes: ["readinessProbe", "livenessProbe"]
Warning: [azurepolicy-k8sazurev1restrictedlabels-67c4210cc58f28acdfdb] Label <{"kubernetes.azure.com"}> is reserved for AKS use only
Warning: [azurepolicy-k8sazurev3containerlimits-a8754961dbd4c1d8b49d] container <my-container> has no resource limits
Warning: [azurepolicy-k8sazurev1containerrestrictedi-bde07e1776cbcc9aa8b8] my-pod in default does not have imagePullSecrets. Unauthenticated image pulls are not recommended.
pod/my-pod created

Enforcement

With deployment safeguard mutations, the Enforcement level mutates your Kubernetes resources when applicable. However, your Kubernetes resources still need to pass all safeguards to deploy successfully. If any safeguard policies fail, your resource is denied and won't be deployed.

$ kubectl apply -f pod.yml
Error from server (Forbidden): error when creating ".\pod.yml": admission webhook "validation.gatekeeper.sh" denied the request: [azurepolicy-k8sazurev2containerenforceprob-0e8a839bcd103e7b96a8] Container <my-container> in your Pod <my-pod> has no <livenessProbe>. Required probes: ["readinessProbe", "livenessProbe"]
[azurepolicy-k8sazurev2containerenforceprob-0e8a839bcd103e7b96a8] Container <my-container> in your Pod <my-pod> has no <readinessProbe>. Required probes: ["readinessProbe", "livenessProbe"]
[azurepolicy-k8sazurev2containerallowedimag-1ff6d14b2f8da22019d7] Container image my-image for container my-container has not been allowed.
[azurepolicy-k8sazurev1restrictedlabels-67c4210cc58f28acdfdb] Label <{"kubernetes.azure.com"}> is reserved for AKS use only
[azurepolicy-k8sazurev1containerrestrictedi-bde07e1776cbcc9aa8b8] my-pod in default does not have imagePullSecrets. Unauthenticated image pulls are not recommended.

If your Kubernetes resources comply with the applicable mutation safeguards and meet all other safeguard requirements, they will be successfully deployed, as shown in the following example:

$ kubectl apply -f pod.yml
pod/my-pod created

Verify compliance across clusters using the Azure Policy dashboard

To verify deployment safeguards have been applied and to check on your cluster's compliance, navigate to the Azure portal page for your cluster and select Policies, then select go to Azure Policy.

From the list of policies and initiatives, select the initiative associated with deployment safeguards. You see a dashboard showing compliance state across your AKS cluster.

Note

To properly assess compliance across your AKS cluster, the Azure Policy initiative must be scoped to your cluster's resource group.

Disable deployment safeguards

Disable deployment safeguards on your cluster using the az aks update command and set the --safeguards-level to Off.

az aks update --name myAKSCluster --resource-group myResourceGroup --safeguards-level Off

--

FAQ

I enabled deployment safeguards with Azure Policy for the first time. Why don't I see any warnings? Why aren't my pods being declined?

Azure Policy can take up to 35 minutes to sync with your cluster after it is enabled for the first time.

I just switched from Warning to Enforcement. Will this take effect immediately?

When switching deployment safeguard levels, you may need to wait up to 15 minutes for the new level to take effect.

Can I create my own mutations?

No. If you have an idea for a safeguard, open an issue in the AKS GitHub repository and add [deployment safeguards request] to the beginning of the title.

Can I pick and choose which mutations I want in Enforcement?

No. Deployment safeguards is all or nothing. Once you turn on Warning or Enforcement, all safeguards will be active.

Why did my deployment resource get admitted even though it wasn't following best practices?

Deployment safeguards enforce best practice standards through Azure Policy controls and has policies that validate against Kubernetes resources. To evaluate and enforce cluster components, Azure Policy extends Gatekeeper. Gatekeeper enforcement also currently operates in a fail-open model. As there's no guarantee that Gatekeeper will respond to our networking call, we make sure that in that case, the validation is skipped so that the deny doesn't block your deployments.

To learn more, see workload validation in Gatekeeper.

Next steps