Custom certificate authority (CA) in Azure Kubernetes Service (AKS) (preview)

Article
11/21/2024

AKS generates and uses the following certificates, Certificate Authorities (CAs), and Service Accounts (SAs):

The AKS API server creates a CA called the Cluster CA.
The API server has a Cluster CA, which signs certificates for one-way communication from the API server to kubelets.
Each kubelet also creates a Certificate Signing Request (CSR), which is signed by the Cluster CA, for communication from the kubelet to the API server.
The API aggregator uses the Cluster CA to issue certificates for communication with other APIs. The API aggregator can also have its own CA for issuing those certificates, but it currently uses the Cluster CA.
Each node uses an SA token, which is signed by the Cluster CA.
The kubectl client has a certificate for communicating with the AKS cluster.

You can also create custom certificate authorities, which allow you to establish trust between your Azure Kubernetes Service (AKS) clusters and workloads, such as private registries, proxies, and firewalls. A Kubernetes secret stores the certificate authority's information, and then it's passed to all nodes in the cluster. This feature is applied per node pool, so you need to enable it on new and existing node pools.

This article shows you how to create custom CAs and apply them to your AKS clusters.

Prerequisites

An Azure subscription. If you don't have an Azure subscription, create a trial subscription.
Azure CLI installed (version 2.43.0 or greater).
A base64 encoded certificate string or a text file with certificate.

Limitations

This feature currently isn't supported for Windows node pools.

Install the `aks-preview` Azure CLI extension

Important

AKS preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. AKS previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use. For more information, see the following support articles:

Install the aks-preview extension using the az extension add command.
```
az extension add --name aks-preview
```
Update to the latest version of the extension using the az extension update command.
```
az extension update --name aks-preview
```

Register the `CustomCATrustPreview` feature flag

Register the CustomCATrustPreview feature flag using the az feature register command.
```
az feature register --namespace "Microsoft.ContainerService" --name "CustomCATrustPreview"
```
It takes a few minutes for the status to show Registered.

Verify the registration status using the az feature show command.

az feature show --namespace "Microsoft.ContainerService" --name "CustomCATrustPreview"

When the status reflects Registered, refresh the registration of the Microsoft.ContainerService resource provider using the az provider register command.
```
az provider register --namespace Microsoft.ContainerService
```

Custom CA installation on AKS node pools

Install CAs on AKS node pools

If your environment requires your custom CAs to be added to node trust store for correct provisioning, you need to pass a text file containing up to 10 blank line separated certificates during az aks create or az aks update operations. Example text file:
```
-----BEGIN CERTIFICATE-----
cert1
-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----
cert2
-----END CERTIFICATE-----
```

Install CAs during node pool creation

Install CAs during node pool creation using the az aks create command and specifying your text file for the --custom-ca-trust-certificates parameter.

az aks create \
    --resource-group <resource-group-name> \
    --name <cluster-name> \
    --node-count 2 \
    --enable-custom-ca-trust \
    --custom-ca-trust-certificates pathToFileWithCAs \
    --generate-ssh-keys

CA rotation for availability during node pool boot up

Update CAs passed to your cluster during boot up using the az aks update command and specifying your text file for the --custom-ca-trust-certificates parameter.
```
az aks update \
    --resource-group <resource-group-name> \
    --name <cluster-name> \
    --custom-ca-trust-certificates pathToFileWithCAs
```
Note

This operation triggers a model update, ensuring new nodes have the newest CAs required for correct provisioning. AKS creates additional nodes, drains existing ones, deletes them, and replaces them with nodes that have the new set of CAs installed.

Install CAs after node pool creation

If your environment can be successfully provisioned without your custom CAs, you can provide the CAs by deploying a secret in the kube-system namespace. This approach allows for certificate rotation without the need for node recreation.

Create a [Kubernetes secret][kubernetes-secrets] YAML manifest with your base64 encoded certificate string in the data field.
```
apiVersion: v1
kind: Secret
metadata: 
    name: custom-ca-trust-secret
    namespace: kube-system
type: Opaque
data:
    ca1.crt: |
      {base64EncodedCertStringHere}
    ca2.crt: |
      {anotherBase64EncodedCertStringHere}
```
Data from this secret is used to update CAs on all nodes. Make sure the secret is named custom-ca-trust-secret and is created in the kube-system namespace. Installing CAs using the secret in the kube-system namespace allows for CA rotation without the need for node recreation. To update or remove a CA, you can edit and apply the YAML manifest. The cluster polls for changes and updates the nodes accordingly. It may take a couple minutes before changes are applied.

Note

containerd restart on the node might be required for the CAs to be picked up properly. If it appears like CAs aren't correctly added to your node's trust store, you can trigger a restart using the following command from node's shell:

systemctl restart containerd

Configure a new AKS cluster to use a custom CA

Configure a new AKS cluster to use a custom CA using the az aks create command with the --enable-custom-ca-trust parameter.

az aks create \
    --resource-group <resource-group-name> \
    --name <cluster-name> \
    --node-count 2 \
    --enable-custom-ca-trust \
    --generate-ssh-keys

Configure a new AKS cluster to use a custom CA with CAs installed before node boots up

Configure a new AKS cluster to use custom CA with CAs installed before the node boots up using the az aks create command with the --enable-custom-ca-trust and --custom-ca-trust-certificates parameters.

az aks create \
    --resource-group <resource-group-name> \
    --name <cluster-name> \
    --node-count 2 \
    --enable-custom-ca-trust \
    --custom-ca-trust-certificates pathToFileWithCAs \
    --generate-ssh-keys

Configure an existing AKS cluster to have custom CAs installed before node boots up

Configure an existing AKS cluster to have your custom CAs added to node's trust store before it boots up using the az aks update command with the --custom-ca-trust-certificates parameter.
```
az aks update \
    --resource-group <resource-group-name> \
    --name <cluster-name> \
    --custom-ca-trust-certificates pathToFileWithCAs
```

Configure a new node pool to use a custom CA

Configure a new node pool to use a custom CA using the az aks nodepool add command with the --enable-custom-ca-trust parameter.
```
az aks nodepool add \
    --cluster-name <cluster-name> \
    --resource-group <resource-group-name> \
    --name <node-pool-name> \
    --enable-custom-ca-trust \
    --os-type Linux
```
If no other node pools with the feature enabled exist, the cluster has to reconcile its settings for the changes to take effect. This operation happens automatically as a part of AKS's reconcile loop. Before the operation, the daemon set and pods don't appear on the cluster. You can trigger an immediate reconcile operation using the az aks update command. The daemon set and pods appear after the update completes.

Configure an existing node pool to use a custom CA

Configure an existing node pool to use a custom CA using the az aks nodepool update command with the --enable-custom-ca-trust parameter.
```
az aks nodepool update \
    --resource-group <resource-group-name> \
    --cluster-name <cluster-name> \
    --name <node-pool-name> \
    --enable-custom-ca-trust
```
If no other node pools with the feature enabled exist, the cluster has to reconcile its settings for the changes to take effect. This operation happens automatically as a part of AKS's reconcile loop. Before the operation, the daemon set and pods don't appear on the cluster. You can trigger an immediate reconcile operation using the az aks update command. The daemon set and pods appear after the update completes.

Troubleshooting

Feature is enabled and secret with CAs is added, but operations are failing with X.509 Certificate Signed by Unknown Authority error

Incorrectly formatted certs passed in the secret

AKS requires certs passed in the user-created secret to be properly formatted and base64 encoded. Make sure the CAs you passed are properly base64 encoded and that files with CAs don't have CRLF line breaks. Certificates passed to --custom-ca-trust-certificates shouldn't be base64 encoded.

containerd hasn't picked up new certs

From the node's shell, run systemctl restart containerd. Once containerd is restarts, the new certs are properly picked up by the container runtime.

Next steps

For more information on AKS security best practices, see Best practices for cluster security and upgrades in Azure Kubernetes Service (AKS).