Use the Azure Disk Container Storage Interface (CSI) driver in Azure Kubernetes Service (AKS)

The Azure Disks Container Storage Interface (CSI) driver is a CSI specification-compliant driver used by Azure Kubernetes Service (AKS) to manage the lifecycle of Azure Disk.

The CSI is a standard for exposing arbitrary block and file storage systems to containerized workloads on Kubernetes. By adopting and using CSI, AKS now can write, deploy, and iterate plug-ins to expose new or improve existing storage systems in Kubernetes. Using CSI drivers in AKS avoids having to touch the core Kubernetes code and wait for its release cycles.

To create an AKS cluster with CSI driver support, see Enable CSI driver on AKS. This article describes how to use the Azure Disk CSI driver version 1.

Azure Disk CSI driver features

In addition to in-tree driver features, Azure Disk CSI driver supports the following features:

• Performance improvements during concurrent disk attach and detach

  • In-tree drivers attach or detach disks in serial, while CSI drivers attach or detach disks in batch. There's significant improvement when there are multiple disks attaching to one node.

• Snapshot

• Volume clone

• Resize disk PV without downtime

Use CSI persistent volumes with Azure Disks

A persistent volume (PV) represents a piece of storage that's provisioned for use with Kubernetes pods. A PV can be used by one or many pods and can be dynamically or statically provisioned. This article shows you how to dynamically create PVs with Azure disk for use by a single pod in an AKS cluster. For static provisioning, see Create a static volume with Azure Disks.

For more information on Kubernetes volumes, see Storage options for applications in AKS.

Dynamically create Azure Disks PVs by using the built-in storage classes

A storage class is used to define how a unit of storage is dynamically created with a persistent volume. For more information on Kubernetes storage classes, see Kubernetes storage classes.

When you use the Azure Disk CSI driver on AKS, there are two more built-in StorageClasses that use the Azure Disk CSI storage driver. The other CSI storage classes are created with the cluster alongside the in-tree default storage classes.

• managed-csi: Uses Azure Standard SSD locally redundant storage (LRS) to create a managed disk.
• managed-csi-premium: Uses Azure Premium LRS to create a managed disk.

The reclaim policy in both storage classes ensures that the underlying Azure Disks are deleted when the respective PV is deleted. The storage classes also configure the PVs to be expandable. You just need to edit the persistent volume claim (PVC) with the new size.

To use these storage classes, create a PVC and respective pod that references and uses them. A PVC is used to automatically provision storage based on a storage class. A PVC can use one of the pre-created storage classes or a user-defined storage class to create an Azure-managed disk for the desired SKU and size. When you create a pod definition, the PVC is specified to request the desired storage.

Create an example pod and respective PVC by running the kubectl apply command:

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/master/deploy/example/pvc-azuredisk-csi.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/master/deploy/example/nginx-pod-azuredisk.yaml

The output of the command resembles the following example:

persistentvolumeclaim/pvc-azuredisk created
pod/nginx-azuredisk created

After the pod is in the running state, run the following command to create a new file called test.txt.

kubectl exec nginx-azuredisk -- touch /mnt/azuredisk/test.txt

To validate the disk is correctly mounted, run the following command and verify you see the test.txt file in the output:

kubectl exec nginx-azuredisk -- ls /mnt/azuredisk

lost+found
outfile
test.txt

Create a custom storage class

The default storage classes are suitable for most common scenarios. For some cases, you might want to have your own storage class customized with your own parameters. For example, you might want to change the volumeBindingMode class.

You can use a volumeBindingMode: Immediate class that guarantees it occurs immediately once the PVC is created. When your node pools are topology constrained, for example when using availability zones, PVs would be bound or provisioned without knowledge of the pod's scheduling requirements.

To address this scenario, you can use volumeBindingMode: WaitForFirstConsumer, which delays the binding and provisioning of a PV until a pod that uses the PVC is created. This way, the PV conforms and is provisioned in the availability zone (or other topology) that's specified by the pod's scheduling constraints. The default storage classes use volumeBindingMode: WaitForFirstConsumer class.

Create a file named sc-azuredisk-csi-waitforfirstconsumer.yaml, and then paste the following manifest. The storage class is the same as our managed-csi storage class, but with a different volumeBindingMode class.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: azuredisk-csi-waitforfirstconsumer
provisioner: disk.csi.azure.com
parameters:
  skuname: StandardSSD_LRS
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

Create the storage class by running the kubectl apply command and specify your sc-azuredisk-csi-waitforfirstconsumer.yaml file:

kubectl apply -f sc-azuredisk-csi-waitforfirstconsumer.yaml

The output of the command resembles the following example:

storageclass.storage.k8s.io/azuredisk-csi-waitforfirstconsumer created

Volume snapshots

The Azure Disk CSI driver supports creating snapshots of persistent volumes. As part of this capability, the driver can perform either full or incremental snapshots depending on the value set in the incremental parameter (by default, it's true).

The following table provides details for all of the parameters.

Name	Meaning	Available Value	Mandatory	Default value
resourceGroup	Resource group for storing snapshot shots	EXISTING RESOURCE GROUP	No	If not specified, snapshot will be stored in the same resource group as source Azure Disks
incremental	Take full or incremental snapshot	true, false	No	true
tags	Azure Disks tags	Tag format: 'key1=val1,key2=val2'	No	""
userAgent	User agent used for customer usage attribution		No	Generated Useragent formatted driverName/driverVersion compiler/version (OS-ARCH)
subscriptionID	Specify Azure subscription ID where Azure Disks will be created	Azure subscription ID	No	If not empty, resourceGroup must be provided, incremental must set as false

Create a volume snapshot

For an example of this capability, create a volume snapshot class with the kubectl apply command:

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/master/deploy/example/snapshot/storageclass-azuredisk-snapshot.yaml

The output of the command resembles the following example:

volumesnapshotclass.snapshot.storage.k8s.io/csi-azuredisk-vsc created

Now let's create a volume snapshot from the PVC that we dynamically created at the beginning of this tutorial, pvc-azuredisk.

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/master/deploy/example/snapshot/azuredisk-volume-snapshot.yaml

The output of the command resembles the following example:

volumesnapshot.snapshot.storage.k8s.io/azuredisk-volume-snapshot created

To verify that the snapshot was created correctly, run the following command:

kubectl describe volumesnapshot azuredisk-volume-snapshot

The output of the command resembles the following example:

Name:         azuredisk-volume-snapshot
Namespace:    default
Labels:       <none>
Annotations:  API Version:  snapshot.storage.k8s.io/v1
Kind:         VolumeSnapshot
Metadata:
  Creation Timestamp:  2020-08-27T05:27:58Z
  Finalizers:
    snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
    snapshot.storage.kubernetes.io/volumesnapshot-bound-protection
  Generation:        1
  Resource Version:  714582
  Self Link:         /apis/snapshot.storage.k8s.io/v1/namespaces/default/volumesnapshots/azuredisk-volume-snapshot
  UID:               dd953ab5-6c24-42d4-ad4a-f33180e0ef87
Spec:
  Source:
    Persistent Volume Claim Name:  pvc-azuredisk
  Volume Snapshot Class Name:      csi-azuredisk-vsc
Status:
  Bound Volume Snapshot Content Name:  snapcontent-dd953ab5-6c24-42d4-ad4a-f33180e0ef87
  Creation Time:                       2020-08-31T05:27:59Z
  Ready To Use:                        true
  Restore Size:                        10Gi
Events:                                <none>

Create a new PVC based on a volume snapshot

You can create a new PVC based on a volume snapshot. Use the snapshot created in the previous step, and create a new PVC and a new pod to consume it.

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/master/deploy/example/snapshot/pvc-azuredisk-snapshot-restored.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/master/deploy/example/snapshot/nginx-pod-restored-snapshot.yaml

The output of the command resembles the following example:

persistentvolumeclaim/pvc-azuredisk-snapshot-restored created
pod/nginx-restored created

Finally, let's make sure it's the same PVC created before by checking the contents by running the following command:

kubectl exec nginx-restored -- ls /mnt/azuredisk

The output of the command resembles the following example:

lost+found
outfile
test.txt

As expected, we can still see our previously created test.txt file.

Clone volumes

A cloned volume is defined as a duplicate of an existing Kubernetes volume. For more information on cloning volumes in Kubernetes, see the conceptual documentation for volume cloning.

The CSI driver for Azure Disks supports volume cloning. To demonstrate, create a cloned volume of the previously created azuredisk-pvc and a new pod to consume it.

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/master/deploy/example/cloning/pvc-azuredisk-cloning.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/master/deploy/example/cloning/nginx-pod-restored-cloning.yaml

The output of the command resembles the following example:

persistentvolumeclaim/pvc-azuredisk-cloning created
pod/nginx-restored-cloning created

You can verify the content of the cloned volume by running the following command and confirming the file test.txt is created:

kubectl exec nginx-restored-cloning -- ls /mnt/azuredisk

The output of the command resembles the following example:

lost+found
outfile
test.txt

Resize a persistent volume without downtime

You can request a larger volume for a PVC. Edit the PVC object, and specify a larger size. This change triggers the expansion of the underlying volume that backs the PV.

In AKS, the built-in managed-csi storage class already supports expansion, so use the PVC created earlier with this storage class. The PVC requested a 10-Gi persistent volume. You can confirm by running the following command:

kubectl exec -it nginx-azuredisk -- df -h /mnt/azuredisk

The output of the command resembles the following example:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc        9.8G   42M  9.8G   1% /mnt/azuredisk

Expand the PVC by increasing the spec.resources.requests.storage field running the following command:

kubectl patch pvc pvc-azuredisk --type merge --patch '{"spec": {"resources": {"requests": {"storage": "15Gi"}}}}'

The output of the command resembles the following example:

persistentvolumeclaim/pvc-azuredisk patched

Run the following command to confirm the volume size has increased:

kubectl get pv

The output of the command resembles the following example:

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                     STORAGECLASS   REASON   AGE
pvc-391ea1a6-0191-4022-b915-c8dc4216174a   15Gi       RWO            Delete           Bound    default/pvc-azuredisk                     managed-csi             2d2h
(...)

And after a few minutes, run the following commands to confirm the size of the PVC:

kubectl get pvc pvc-azuredisk

The output of the command resembles the following example:

NAME            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc-azuredisk   Bound    pvc-391ea1a6-0191-4022-b915-c8dc4216174a   15Gi       RWO            managed-csi    2d2h

Run the following command to confirm the size of the disk inside the pod:

kubectl exec -it nginx-azuredisk -- df -h /mnt/azuredisk

The output of the command resembles the following example:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc         15G   46M   15G   1% /mnt/azuredisk

On-demand bursting

On-demand disk bursting model allows disk bursts whenever its needs exceed its current capacity. This model generates extra charges anytime the disk bursts. On-demand bursting is only available for premium SSDs larger than 512 GiB. For more information on premium SSDs provisioned IOPS and throughput per disk, see Premium SSD size. Alternatively, credit-based bursting is where the disk will burst only if it has burst credits accumulated in its credit bucket. Credit-based bursting doesn't generate extra charges when the disk bursts. Credit-based bursting is only available for premium SSDs 512 GiB and smaller, and standard SSDs 1024 GiB and smaller. For more information on on-demand bursting, see On-demand bursting.

To create a premium SSD persistent volume with on-demand bursting enabled, you can create a new storage class with the enableBursting parameter set to true as shown in the following YAML template. For more information on enabling on-demand bursting, see On-demand bursting. For more information on building your own storage class with on-demand bursting enabled, see Create a Burstable Managed CSI Premium Storage Class.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: burstable-managed-csi-premium
provisioner: disk.csi.azure.com
parameters:
  skuname: Premium_LRS
  enableBursting: "true"
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

Windows containers

The Azure Disk CSI driver supports Windows nodes and containers. If you want to use Windows containers, follow the Windows containers quickstart to add a Windows node pool.

After you have a Windows node pool, you can now use the built-in storage classes like managed-csi. You can deploy an example Windows-based stateful set that saves timestamps into the file data.txt by running the following kubectl apply command:

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/master/deploy/example/windows/statefulset.yaml

The output of the command resembles the following example:

statefulset.apps/busybox-azuredisk created

To validate the content of the volume, run the following command:

kubectl exec -it busybox-azuredisk-0 -- cat c:\\mnt\\azuredisk\\data.txt # on Linux/MacOS Bash
kubectl exec -it busybox-azuredisk-0 -- cat c:\mnt\azuredisk\data.txt # on Windows Powershell/CMD

The output of the command resembles the following example:

2020-08-27 08:13:41Z
2020-08-27 08:13:42Z
2020-08-27 08:13:44Z
(...)

Next steps

• To learn how to use CSI driver for Azure Files, see Use Azure Files with CSI driver.
• To learn how to use CSI driver for Azure Blob storage, see Use Azure Blob storage with CSI driver.
• For more information about storage best practices, see Best practices for storage and backups in Azure Kubernetes Service.
• For more information about disk-based storage solutions, see Disk-based solutions in AKS.