Deploy and manage Azure Arc-enabled Kubernetes cluster extensions

You can create extension instances in an Arc-enabled Kubernetes cluster, setting required and optional parameters including options related to updates and configurations. You can also view, list, update, and delete extension instances.

Before you begin, read the conceptual overview of Arc-enabled Kubernetes cluster extensions and review the list of currently available extensions.

Prerequisites

• The latest version of Azure CLI.

• The latest versions of the connectedk8s and k8s-extension Azure CLI extensions. Install these extensions by running the following commands:

  az extension add --name connectedk8s
  az extension add --name k8s-extension
  

  If the connectedk8s and k8s-extension extensions are already installed, make sure they're updated to the latest version using the following commands:

  az extension update --name connectedk8s
  az extension update --name k8s-extension
  

• An existing Azure Arc-enabled Kubernetes connected cluster, with at least one node of operating system and architecture type linux/amd64. If deploying Flux (GitOps), you can use an ARM64-based cluster without a linux/amd64 node.

  • If you haven't connected a cluster yet, use our quickstart.
  • Upgrade your agents to the latest version.

Create extension instance

To create a new extension instance, use k8s-extension create, passing in values for the required parameters.

This example creates an Azure Key Vault Secrets Provider extension instance on an Azure Arc-enabled Kubernetes cluster:

az k8s-extension create --name akvsecretsprovider  --extension-type Microsoft.AzureKeyVaultSecretsProvider --scope cluster --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type connectedClusters

Output:

{
  "aksAssignedIdentity": null,
  "autoUpgradeMinorVersion": true,
  "configurationProtectedSettings": {},
  "configurationSettings": {},
  "customLocationSettings": null,
  "errorInfo": null,
  "extensionType": "microsoft.azurekeyvaultsecretsprovider",
  "id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/demo/providers/Microsoft.Kubernetes/connectedClusters/chenyeaks/providers/Microsoft.KubernetesConfiguration/extensions/akvsecretsprovider",
  "identity": {
    "principalId": "3f9c9566-1d32-4861-9f36-9739bf853420",
    "tenantId": null,
    "type": "SystemAssigned"
  },
  "installedVersion": null,
  "name": "akvsecretsprovider",
  "packageUri": null,
  "provisioningState": "Succeeded",
  "releaseTrain": "Stable",
  "resourceGroup": "demo",
  "scope": {
    "cluster": {
      "releaseNamespace": "kube-system"
    },
    "namespace": null
  },
  "statuses": [],
  "systemData": {
    "createdAt": "2023-09-13T02:34:19.549739+00:00",
    "createdBy": null,
    "createdByType": null,
    "lastModifiedAt": "2023-09-13T02:34:19.549739+00:00",
    "lastModifiedBy": null,
    "lastModifiedByType": null
  },
  "type": "Microsoft.KubernetesConfiguration/extensions",
  "version": "1.4.2"
}

Required parameters

The following parameters are required when using az k8s-extension create to create an extension instance.

Parameter name	Description
--name	Name of the extension instance
--extension-type	The type of extension you want to install on the cluster. For example: Microsoft.AzureMonitor.Containers, microsoft.azuredefender.kubernetes
--scope	Scope of installation for the extension: cluster or namespace
--cluster-name	Name of the Azure Arc-enabled Kubernetes resource on which the extension instance has to be created
--resource-group	The resource group containing the Azure Arc-enabled Kubernetes resource
--cluster-type	The cluster type on which the extension instance has to be created. For most scenarios, use connectedClusters, which corresponds to Azure Arc-enabled Kubernetes clusters.

Optional parameters

Use one or more of these optional parameters as needed for your scenarios, along with the required parameters.

Parameter name	Description
--auto-upgrade-minor-version	Boolean property that determines whether the extension minor version is automatically upgraded. The default setting is true. If this parameter is set to true, you can't set the version parameter, as the version will be dynamically updated. If set to false, the extension won't be automatically upgraded, even for patch versions.
--version	Version of the extension to be installed (specific version to pin the extension instance to). Must not be supplied if auto-upgrade-minor-version is set to true.
--configuration-settings	Settings that can be passed into the extension to control its functionality. These are passed in as space-separated key=value pairs after the parameter name. If this parameter is used in the command, then --configuration-settings-file can't be used in the same command.
--configuration-settings-file	Path to a JSON file with key=value pairs to be used for passing configuration settings into the extension. If this parameter is used in the command, then --configuration-settings can't be used in the same command.
--configuration-protected-settings	Settings that aren't retrievable using GET API calls or az k8s-extension show commands. Typically used to pass in sensitive settings. These are passed in as space-separated key=value pairs after the parameter name. If this parameter is used in the command, then --configuration-protected-settings-file can't be used in the same command.
--configuration-protected-settings-file	Path to a JSON file with key=value pairs to be used for passing sensitive settings into the extension. If this parameter is used in the command, then --configuration-protected-settings can't be used in the same command.
--release-namespace	This parameter indicates the namespace within which the release will be created. Only relevant if scope is set to cluster.
--release-train	Extension authors can publish versions in different release trains such as Stable, Preview, etc. If this parameter isn't set explicitly, Stable is used as default.
--target-namespace	Indicates the namespace within which the release will be created. Permission of the system account created for this extension instance will be restricted to this namespace. Only relevant if scope is set to namespace.

Show extension details

To view details of a currently installed extension instance, use k8s-extension show, passing in values for the mandatory parameters.

az k8s-extension show --name akvsecretsprovider --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type connectedClusters

Output:

{
  "aksAssignedIdentity": null,
  "autoUpgradeMinorVersion": true,
  "configurationProtectedSettings": {},
  "configurationSettings": {},
  "customLocationSettings": null,
  "errorInfo": null,
  "extensionType": "microsoft.azurekeyvaultsecretsprovider",
  "id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/demo/providers/Microsoft.Kubernetes/connectedClusters/chenyeaks/providers/Microsoft.KubernetesConfiguration/extensions/akvsecretsprovider",
  "identity": {
    "principalId": "3f9c9566-1d32-4861-9f36-9739bf853420",
    "tenantId": null,
    "type": "SystemAssigned"
  },
  "installedVersion": null,
  "name": "akvsecretsprovider",
  "packageUri": null,
  "provisioningState": "Succeeded",
  "releaseTrain": "Stable",
  "resourceGroup": "demo",
  "scope": {
    "cluster": {
      "releaseNamespace": "kube-system"
    },
    "namespace": null
  },
  "statuses": [],
  "systemData": {
    "createdAt": "2023-09-13T02:34:19.549739+00:00",
    "createdBy": null,
    "createdByType": null,
    "lastModifiedAt": "2023-09-13T02:34:19.549739+00:00",
    "lastModifiedBy": null,
    "lastModifiedByType": null
  },
  "type": "Microsoft.KubernetesConfiguration/extensions",
  "version": "1.4.2"
}

List all extensions installed on the cluster

To view a list of all extensions installed on a cluster, use k8s-extension list, passing in values for the mandatory parameters.

az k8s-extension list --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type connectedClusters

Output:

[
  {
    "aksAssignedIdentity": null,
    "autoUpgradeMinorVersion": true,
    "configurationProtectedSettings": {},
    "configurationSettings": {},
    "customLocationSettings": null,
    "errorInfo": null,
    "extensionType": "microsoft.azurekeyvaultsecretsprovider",
    "id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/demo/providers/Microsoft.Kubernetes/connectedClusters/chenyeaks/providers/Microsoft.KubernetesConfiguration/extensions/akvsecretsprovider",
    "identity": {
      "principalId": "3f9c9566-1d32-4861-9f36-9739bf853420",
      "tenantId": null,
      "type": "SystemAssigned"
    },
    "installedVersion": null,
    "name": "akvsecretsprovider",
    "packageUri": null,
    "provisioningState": "Succeeded",
    "releaseTrain": "Stable",
    "resourceGroup": "demo",
    "scope": {
      "cluster": {
        "releaseNamespace": "kube-system"
      },
      "namespace": null
    },
    "statuses": [],
    "systemData": {
      "createdAt": "2023-09-13T02:34:19.549739+00:00",
      "createdBy": null,
      "createdByType": null,
      "lastModifiedAt": "2023-09-13T02:34:19.549739+00:00",
      "lastModifiedBy": null,
      "lastModifiedByType": null
    },
    "type": "Microsoft.KubernetesConfiguration/extensions",
    "version": "1.4.2"
  }
]

Update extension instance

To update an existing extension instance, use k8s-extension update, passing in values for the mandatory and optional parameters. The mandatory and optional parameters are slightly different than those used to create an extension instance.

This example updates the auto-upgrade-minor-version setting for an Azure Machine Learning extension instance to true:

az k8s-extension update --name akvsecretsprovider --cluster-resource-provider Microsoft.AzureKeyvaultSecretsProvider --cluster-name <clusterName> --resource-group <resourceGroupName> --auto-upgrade-minor-version false --cluster-type connectedClusters

Required parameters for update

Parameter name	Description
--name	Name of the extension instance
--cluster-name	Name of the cluster on which the extension instance has to be created
--resource-group	The resource group containing the cluster
--cluster-type	The cluster type on which the extension instance has to be created. For Azure Arc-enabled Kubernetes clusters, use connectedClusters. For AKS clusters, use managedClusters.

Optional parameters for update

Parameter name	Description
--auto-upgrade-minor-version	Boolean property that specifies whether the extension minor version is automatically upgraded. The default setting is true. If this parameter is set to true, you can't set the version parameter, as the version will be dynamically updated. If set to false, the extension won't be automatically upgraded, even for patch versions.
--version	Version of the extension to be installed (specific version to pin the extension instance to). Must not be supplied if auto-upgrade-minor-version is set to true.
--configuration-settings	Settings that can be passed into the extension to control its functionality. These are passed in as space-separated key=value pairs after the parameter name. If this parameter is used in the command, then --configuration-settings-file can't be used in the same command. Only the settings that require an update need to be provided. The provided settings will be replaced with the specified values.
--configuration-settings-file	Path to the JSON file with key=value pairs to be used for passing in configuration settings to the extension. If this parameter is used in the command, then --configuration-settings can't be used in the same command.
--configuration-protected-settings	Settings that aren't retrievable using GET API calls or az k8s-extension show commands. Typically used to pass in sensitive settings. These are passed in as space-separated key=value pairs after the parameter name. If this parameter is used in the command, then --configuration-protected-settings-file can't be used in the same command. When you update a protected setting, all of the protected settings are expected to be specified. If any of these settings are omitted, those settings will be considered obsolete and deleted.
--configuration-protected-settings-file	Path to a JSON file with key=value pairs to be used for passing in sensitive settings to the extension. If this parameter is used in the command, then --configuration-protected-settings can't be used in the same command.
--scope	Scope of installation for the extension - cluster or namespace.
--release-train	Extension authors can publish versions in different release trains such as Stable, Preview, etc. If this parameter isn't set explicitly, Stable is used as default.

Upgrade extension instance

As noted earlier, if you set auto-upgrade-minor-version to true, the extension will automatically be upgraded when a new minor version is released. For most scenarios, we recommend enabling automatic upgrades. If you set auto-upgrade-minor-version to false, you'll have to upgrade the extension manually if you want a newer version.

Manual upgrades are also required to get a new major instance of an extension. You can choose when to upgrade in order to avoid any unexpected breaking changes with major version upgrades.

To manually upgrade an extension instance, use k8s-extension update and set the version parameter to specify a version.

This example updates an zure Key Vault Secrets Provider instance to version x.y.z:

az k8s-extension update --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type connectedClusters --name akvsecretsprovider --version x.y.z

Delete extension instance

To delete an extension instance on a cluster, use k8s-extension delete, passing in values for the mandatory parameters:

az k8s-extension delete --name akvsecretsprovider --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type connectedClusters

AKS hybrid clusters provisioned from Azure (preview)

You can deploy extensions to AKS hybrid clusters provisioned from Azure. However, there are a few key differences to keep in mind in order to deploy successfully:

• The value for the --cluster-type parameter must be provisionedClusters.

• You must add --cluster-resource-provider microsoft.hybridcontainerservice to your commands.

• When deleting an extension instance, you must add --yes to the command:

  az k8s-extension delete --name azuremonitor-containers --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type provisionedClusters --cluster-resource-provider microsoft.hybridcontainerservice --yes
  

In addition, you must be using the latest version of the Azure CLI k8s-extension module (version >= 1.3.3). Use the following commands to add or update to the latest version:

# add if you do not have this installed
az extension add --name k8s-extension

# update if you do have the module installed
az extension update --name k8s-extension

Next steps

• Review the az k8s-extension CLI reference for a comprehensive list of commands and parameters.
• Learn more about how extensions work with Arc-enabled Kubernetes clusters.
• Review the cluster extensions currently available for Azure Arc-enabled Kubernetes.