Connect your Azure identity provider to the Azure Key Vault Secrets Store CSI Driver in Azure Kubernetes Service (AKS)

  • Article

The Secrets Store Container Storage Interface (CSI) Driver on Azure Kubernetes Service (AKS) provides various methods of identity-based access to your Azure Key Vault. This article outlines these methods and best practices for when to use Role-based access control (RBAC) or OpenID Connect (OIDC) security models to access your key vault and AKS cluster.

You can use one of the following access methods:

  • Service Connector with Workload ID
  • Workload ID
  • User-assigned managed identity

Learn how to connect to Azure Key Vault with the Secrets Store CSI Driver in an Azure Kubernetes Service (AKS) cluster using Service Connector. In this article, you complete the following tasks:

  • Create an AKS cluster and an Azure Key Vault.
  • Create a connection between the AKS cluster and the Azure Key Vault with Service Connector.
  • Create a SecretProviderClass CRD and a Pod that consumes the CSI provider to test the connection.
  • Clean up resources.

Important

AKS preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. AKS previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use. For more information, see the following support articles:

Prerequisites

Create Azure resources

  1. Create a resource group using the az group create command.

    az group create \
    --name <resource-group-name> \
    --location <location>

  2. Create an AKS cluster using the az aks create command. The following example creates a single-node AKS cluster with managed identity enabled.

    az aks create \
    --resource-group <resource-group-name> \
    --name <cluster-name> \
    --enable-managed-identity \
    --node-count 1

  3. Connect to the cluster using the az aks get-credentials command.

    az aks get-credentials \
    --resource-group <resource-group-name> \
    --name <cluster-name>

  4. Create an Azure key vault using the az keyvault create command.

    az keyvault create \
    --resource-group <resource-group-name> \  
    --name <key-vault-name> \
    --location <location>

  5. Create a secret in the key vault using the az keyvault secret set command.

    az keyvault secret set \
    --vault-name <key-vault-name> \
    --name <secret-name> \
    --value <secret-value>

Create a service connection in AKS with Service Connector (preview)

You can create a service connection to Azure Key Vault using the Azure portal or Azure CLI.

  1. In the Azure portal, navigate to your AKS cluster resource.

  2. From the service menu, under Settings, select Service Connector (Preview) > Create.

  3. On the Create connection page, configure the following settings in the Basics tab:

    • Kubernetes namespace: Select default.
    • Service type: Select Key Vault and select the checkbox to enable the Azure Key Vault CSI Provider.
    • Connection name: Enter a name for the connection.
    • Subscription: Select the subscription that contains the key vault.
    • Key vault: Select the key vault you created.
    • Client type: Select None.

  4. Select Review + create, and then select Create to create the connection.

Test the connection

Clone the sample repo and deploy manifest files

  1. Clone the sample repository using the git clone command.

    git clone https://github.com/Azure-Samples/serviceconnector-aks-samples.git

  2. Change directories to the Azure Key Vault CSI provider sample.

    cd serviceconnector-aks-samples/azure-keyvault-csi-provider

  3. In the secret_provider_class.yaml file, replace the following placeholders with your Azure Key Vault information:

    • Replace <AZURE_KEYVAULT_NAME> with the name of the key vault you created and connected.
    • Replace <AZURE_KEYVAULT_TENANTID> with the tenant ID of the key vault.
    • Replace <AZURE_KEYVAULT_CLIENTID> with identity client ID of the azureKeyvaultSecretsProvider addon.
    • Replace <KEYVAULT_SECRET_NAME> with the key vault secret you created. For example, ExampleSecret.

  4. Deploy the SecretProviderClass CRD using the kubectl apply command.

    kubectl apply -f secret_provider_class.yaml

  5. Deploy the Pod manifest file using the kubectl apply command.

    The command creates a pod named sc-demo-keyvault-csi in the default namespace of your AKS cluster.

    kubectl apply -f pod.yaml

Verify the connection

  1. Verify the pod was created successfully using the kubectl get command.

    kubectl get pod/sc-demo-keyvault-csi

    After the pod starts, the mounted content at the volume path specified in your deployment YAML is available.

  2. Show the secrets held in the secrets store using the kubectl exec command.

    kubectl exec sc-demo-keyvault-csi -- ls /mnt/secrets-store/

  3. Display a secret using the kubectl exec command.

    This example command shows a test secret named ExampleSecret.

    kubectl exec sc-demo-keyvault-csi -- cat /mnt/secrets-store/ExampleSecret

Prerequisites for CSI Driver

Access with a Microsoft Entra Workload ID

A Microsoft Entra Workload ID is an identity that an application running on a pod uses to authenticate itself against other Azure services, such as workloads in software. The Secret Store CSI Driver integrates with native Kubernetes capabilities to federate with external identity providers.

In this security model, the AKS cluster acts as token issuer. Microsoft Entra ID then uses OIDC to discover public signing keys and verify the authenticity of the service account token before exchanging it for a Microsoft Entra token. For your workload to exchange a service account token projected to its volume for a Microsoft Entra token, you need the Azure Identity client library in the Azure SDK or the Microsoft Authentication Library (MSAL)

Note

  • This authentication method replaces Microsoft Entra pod-managed identity (preview). The open source Microsoft Entra pod-managed identity (preview) in Azure Kubernetes Service has been deprecated as of 10/24/2022.
  • Microsoft Entra Workload ID is supports both Windows and Linux clusters.

Configure workload identity

  1. Set your subscription using the az account set command.

    export SUBSCRIPTION_ID=<subscription id>
export RESOURCE_GROUP=<resource group name>
export UAMI=<name for user assigned identity>
export KEYVAULT_NAME=<existing keyvault name>
export CLUSTER_NAME=<aks cluster name>

az account set --subscription $SUBSCRIPTION_ID

  2. Create a managed identity using the az identity create command.

    Note

    This step assumes you have an existing AKS cluster with workload identity enabled. If you don't have it enabled, see Enable workload identity on an existing AKS cluster to enable it.

    az identity create --name $UAMI --resource-group $RESOURCE_GROUP

export USER_ASSIGNED_CLIENT_ID="$(az identity show --resource-group $RESOURCE_GROUP --name $UAMI --query 'clientId' -o tsv)"
export IDENTITY_TENANT=$(az aks show --name $CLUSTER_NAME --resource-group $RESOURCE_GROUP --query identity.tenantId -o tsv)

  3. Create a role assignment that grants the workload identity permission to access the key vault secrets, access keys, and certificates using the az role assignment create command.

    Important

    • If your key vault is set with --enable-rbac-authorization and you're using key or certificate type, assign the Key Vault Certificate User role to give permissions.
    • If your key vault is set with --enable-rbac-authorization and you're using secret type, assign the Key Vault Secrets User role.
    • If your key vault isn't set with --enable-rbac-authorization, you can use the az keyvault set-policy command with the --key-permissions get, --certificate-permissions get, or --secret-permissions get parameter to create a key vault policy to grant access for keys, certificates, or secrets. For example:
    az keyvault set-policy --name $KEYVAULT_NAME --key-permissions get --object-id $IDENTITY_OBJECT_ID

    export KEYVAULT_SCOPE=$(az keyvault show --name $KEYVAULT_NAME --query id -o tsv)

# Example command for key vault with RBAC enabled using `key` type
az role assignment create --role "Key Vault Certificate User" --assignee $USER_ASSIGNED_CLIENT_ID --scope $KEYVAULT_SCOPE

  4. Get the AKS cluster OIDC Issuer URL using the az aks show command.

    Note

    This step assumes you have an existing AKS cluster with the OIDC Issuer URL enabled. If you don't have it enabled, see Update an AKS cluster with OIDC Issuer to enable it.

    export AKS_OIDC_ISSUER="$(az aks show --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME --query "oidcIssuerProfile.issuerUrl" -o tsv)"
echo $AKS_OIDC_ISSUER

  5. Establish a federated identity credential between the Microsoft Entra application, service account issuer, and subject. Get the object ID of the Microsoft Entra application using the following commands. Make sure to update the values for serviceAccountName and serviceAccountNamespace with the Kubernetes service account name and its namespace.

    export SERVICE_ACCOUNT_NAME="workload-identity-sa"  # sample name; can be changed
export SERVICE_ACCOUNT_NAMESPACE="default" # can be changed to namespace of your workload

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    azure.workload.identity/client-id: ${USER_ASSIGNED_CLIENT_ID}
  name: ${SERVICE_ACCOUNT_NAME}
  namespace: ${SERVICE_ACCOUNT_NAMESPACE}
EOF

  6. Create the federated identity credential between the managed identity, service account issuer, and subject using the az identity federated-credential create command.

    export FEDERATED_IDENTITY_NAME="aksfederatedidentity" # can be changed as needed

az identity federated-credential create --name $FEDERATED_IDENTITY_NAME --identity-name $UAMI --resource-group $RESOURCE_GROUP --issuer ${AKS_OIDC_ISSUER} --subject system:serviceaccount:${SERVICE_ACCOUNT_NAMESPACE}:${SERVICE_ACCOUNT_NAME}

  7. Deploy a SecretProviderClass using the kubectl apply command and the following YAML script.

    cat <<EOF | kubectl apply -f -
# This is a SecretProviderClass example using workload identity to access your key vault
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kvname-wi # needs to be unique per namespace
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    clientID: "${USER_ASSIGNED_CLIENT_ID}" # Setting this to use workload identity
    keyvaultName: ${KEYVAULT_NAME}       # Set to the name of your key vault
    cloudName: "AzureChinaCloud"                         # [OPTIONAL for Azure] if not provided, the Azure environment defaults to AzureChinaCloud
    objects:  |
      array:
        - |
          objectName: secret1             # Set to the name of your secret
          objectType: secret              # object types: secret, key, or cert
          objectVersion: ""               # [OPTIONAL] object versions, default to latest if empty
        - |
          objectName: key1                # Set to the name of your key
          objectType: key
          objectVersion: ""
    tenantId: "${IDENTITY_TENANT}"        # The tenant ID of the key vault
EOF

    Note

    If you use objectAlias instead of objectName, update the YAML script to account for it.

    Note

    In order for the SecretProviderClass to function properly, make sure to populate your Azure Key Vault with secrets, keys, or certificates before referencing them in the objects section.

  8. Deploy a sample pod using the kubectl apply command and the following YAML script.

    cat <<EOF | kubectl apply -f -
# This is a sample pod definition for using SecretProviderClass and workload identity to access your key vault
kind: Pod
apiVersion: v1
metadata:
  name: busybox-secrets-store-inline-wi
  labels:
    azure.workload.identity/use: "true"
spec:
  serviceAccountName: "workload-identity-sa"
  containers:
    - name: busybox
      image: k8sgcr.azk8s.cn/e2e-test-images/busybox:1.29-4
      command:
        - "/bin/sleep"
        - "10000"
      volumeMounts:
      - name: secrets-store01-inline
        mountPath: "/mnt/secrets-store"
        readOnly: true
  volumes:
    - name: secrets-store01-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "azure-kvname-wi"
EOF

Prerequisites for CSI Driver

Access with managed identity

A Microsoft Entra Managed ID is an identity that an administrator uses to authenticate themselves against other Azure services. The managed identity uses RBAC to federate with external identity providers.

In this security model, you can grant access to your cluster's resources to team members or tenants sharing a managed role. The role is checked for scope to access the keyvault and other credentials. When you enabled the Azure Key Vault provider for Secrets Store CSI Driver on your AKS Cluster, it created a user identity.

Configure managed identity

  1. Access your key vault using the az aks show command and the user-assigned managed identity created by the add-on. You should also retrieve the identity's clientId, which you'll use in later steps when creating a SecretProviderClass.

    az aks show --resource-group <resource-group> --name <cluster-name> --query addonProfiles.azureKeyvaultSecretsProvider.identity.objectId -o tsv
az aks show --resource-group <resource-group> --name <cluster-name> --query addonProfiles.azureKeyvaultSecretsProvider.identity.clientId -o tsv

    Alternatively, you can create a new managed identity and assign it to your virtual machine (VM) scale set or to each VM instance in your availability set using the following commands.

    az identity create --resource-group <resource-group> --name <identity-name>
az vmss identity assign --resource-group <resource-group> --name <agent-pool-vmss> --identities <identity-resource-id>
az vm identity assign --resource-group <resource-group> --name <agent-pool-vm> --identities <identity-resource-id>

az identity show --resource-group <resource-group> --name <identity-name> --query 'clientId' -o tsv

  2. Create a role assignment that grants the identity permission to access the key vault secrets, access keys, and certificates using the az role assignment create command.

    Important

    • If your key vault is set with --enable-rbac-authorization and you're using key or certificate type, assign the Key Vault Certificate User role.
    • If your key vault is set with --enable-rbac-authorization and you're using secret type, assign the Key Vault Secrets User role.
    • If your key vault isn't set with --enable-rbac-authorization, you can use the az keyvault set-policy command with the --key-permissions get, --certificate-permissions get, or --secret-permissions get parameter to create a key vault policy to grant access for keys, certificates, or secrets. For example:
    az keyvault set-policy --name $KEYVAULT_NAME --key-permissions get --object-id $IDENTITY_OBJECT_ID

    export IDENTITY_OBJECT_ID="$(az identity show --resource-group <resource-group> --name <identity-name> --query 'principalId' -o tsv)"
export KEYVAULT_SCOPE=$(az keyvault show --name <key-vault-name> --query id -o tsv)

# Example command for key vault with RBAC enabled using `key` type
az role assignment create --role "Key Vault Certificate User" --assignee $USER_ASSIGNED_CLIENT_ID --scope $KEYVAULT_SCOPE

  3. Create a SecretProviderClass using the following YAML. Make sure to use your own values for userAssignedIdentityID, keyvaultName, tenantId, and the objects to retrieve from your key vault.

    # This is a SecretProviderClass example using user-assigned identity to access your key vault
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kvname-user-msi
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "true"          # Set to true for using managed identity
    userAssignedIdentityID: <client-id>   # Set the clientID of the user-assigned managed identity to use
    keyvaultName: <key-vault-name>        # Set to the name of your key vault
    cloudName: "AzureChinaCloud"          # [OPTIONAL for Azure] if not provided, the Azure environment defaults to AzurePublicCloud
    objects:  |
      array:
        - |
          objectName: secret1
          objectType: secret              # object types: secret, key, or cert
          objectVersion: ""               # [OPTIONAL] object versions, default to latest if empty
        - |
          objectName: key1
          objectType: key
          objectVersion: ""
    tenantId: <tenant-id>                 # The tenant ID of the key vault

    Note

    If you use objectAlias instead of objectName, make sure to update the YAML script.

    Note

    In order for the SecretProviderClass to function properly, make sure to populate your Azure Key Vault with secrets, keys, or certificates before referencing them in the objects section.

  4. Apply the SecretProviderClass to your cluster using the kubectl apply command.

    kubectl apply -f secretproviderclass.yaml

  5. Create a pod using the following YAML.

    # This is a sample pod definition for using SecretProviderClass and the user-assigned identity to access your key vault
kind: Pod
apiVersion: v1
metadata:
  name: busybox-secrets-store-inline-user-msi
spec:
  containers:
    - name: busybox
      image: k8sgcr.azk8s.cn/e2e-test-images/busybox:1.29-4
      command:
        - "/bin/sleep"
        - "10000"
      volumeMounts:
      - name: secrets-store01-inline
        mountPath: "/mnt/secrets-store"
        readOnly: true
  volumes:
    - name: secrets-store01-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "azure-kvname-user-msi"

  6. Apply the pod to your cluster using the kubectl apply command.

    kubectl apply -f pod.yaml

Validate Key Vault secrets

After the pod starts, the mounted content at the volume path specified in your deployment YAML is available. Use the following commands to validate your secrets and print a test secret.

  1. Show secrets held in the secrets store using the following command.

    kubectl exec busybox-secrets-store-inline-user-msi -- ls /mnt/secrets-store/

  2. Display a secret in the store using the following command. This example command shows the test secret ExampleSecret.

    kubectl exec busybox-secrets-store-inline-user-msi -- cat /mnt/secrets-store/ExampleSecret

Obtain certificates and keys

The Azure Key Vault design makes sharp distinctions between keys, secrets, and certificates. The certificate features of the Key Vault service are designed to make use of key and secret capabilities. When you create a key vault certificate, it creates an addressable key and secret with the same name. This key allows authentication operations, and the secret allows the retrieval of the certificate value as a secret.

A key vault certificate also contains public x509 certificate metadata. The key vault stores both the public and private components of your certificate in a secret. You can obtain each individual component by specifying the objectType in SecretProviderClass. The following table shows which objects map to the various resources associated with your certificate:

Object Return value Returns entire certificate chain
key The public key, in Privacy Enhanced Mail (PEM) format. N/A
cert The certificate, in PEM format. No
secret The private key and certificate, in PEM format. Yes

Disable the addon on existing clusters

Note

Before you disable the add-on, ensure that no SecretProviderClass is in use. Trying to disable the add-on while a SecretProviderClass exists results in an error.

  • Disable the Azure Key Vault provider for Secrets Store CSI Driver capability in an existing cluster using the az aks disable-addons command with the azure-keyvault-secrets-provider add-on.

    az aks disable-addons --addons azure-keyvault-secrets-provider --resource-group myResourceGroup --name myAKSCluster

Note

When you disable the add-on, existing workloads should have no issues or see any updates in the mounted secrets. If the pod restarts or a new pod is created as part of scale-up event, the pod fails to start because the driver is no longer running.

Next steps

In this article, you learned how to create and provide an identity to access your Azure Key Vault. If you want to configure extra configuration options or perform troubleshooting, continue to the next article.

If you want to configure extra configuration options or perform troubleshooting, see Configuration options and troubleshooting resources for Azure Key Vault provider with Secrets Store CSI Driver in AKS.