Deploy and manage an Azure Arc-enabled Kubernetes cluster extension

Article
11/25/2024

You can use an extension in an Azure Arc-enabled Kubernetes cluster to access Azure services and scenarios. This article describes how to create extension instances and set required and optional parameters, including options for updates and configurations. You also learn how to view, list, update, and delete extension instances.

Before you begin, read the overview of Azure Arc-enabled Kubernetes cluster extensions and review the list of currently available extensions.

Prerequisites

The latest version of the Azure CLI.
The latest versions of the connectedk8s and k8s-extension Azure CLI extensions. To install these extensions, run the following commands:
```
az extension add --name connectedk8s
az extension add --name k8s-extension
```
If the connectedk8s and k8s-extension extensions are already installed, make sure that they're updated to the latest version by using these commands:
```
az extension update --name connectedk8s
az extension update --name k8s-extension
```
An existing Azure Arc-enabled Kubernetes connected cluster, with at least one node of operating system and architecture type linux/amd64. If you deploy Flux (GitOps), you can use an ARM64-based cluster without using a linux/amd64 node.
- If you haven't connected a cluster yet, use our quickstart to connect one.
- Upgrade your agents to the latest version.

Note

Installing Azure Arc extensions on AKS hybrid clusters provisioned from Azure is currently in preview, with support for the Azure Key Vault Secrets Provider, and Flux (GitOps) extensions.

Create an extension instance

To create a new extension instance, use k8s-extension create, passing in values for the required parameters.

This example creates an Azure Key Vault Secrets Provider extension instance on an Azure Arc-enabled Kubernetes cluster:

az k8s-extension create --name akvsecretsprovider  --extension-type Microsoft.AzureKeyVaultSecretsProvider --scope cluster --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type connectedClusters

Output:

{
  "aksAssignedIdentity": null,
  "autoUpgradeMinorVersion": true,
  "configurationProtectedSettings": {},
  "configurationSettings": {},
  "customLocationSettings": null,
  "errorInfo": null,
  "extensionType": "microsoft.azurekeyvaultsecretsprovider",
  "id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/demo/providers/Microsoft.Kubernetes/connectedClusters/chenyeaks/providers/Microsoft.KubernetesConfiguration/extensions/akvsecretsprovider",
  "identity": {
    "principalId": "3f9c9566-1d32-4861-9f36-9739bf853420",
    "tenantId": null,
    "type": "SystemAssigned"
  },
  "installedVersion": null,
  "name": "akvsecretsprovider",
  "packageUri": null,
  "provisioningState": "Succeeded",
  "releaseTrain": "Stable",
  "resourceGroup": "demo",
  "scope": {
    "cluster": {
      "releaseNamespace": "kube-system"
    },
    "namespace": null
  },
  "statuses": [],
  "systemData": {
    "createdAt": "2023-09-13T02:34:19.549739+00:00",
    "createdBy": null,
    "createdByType": null,
    "lastModifiedAt": "2023-09-13T02:34:19.549739+00:00",
    "lastModifiedBy": null,
    "lastModifiedByType": null
  },
  "type": "Microsoft.KubernetesConfiguration/extensions",
  "version": "1.4.2"
}

Required parameters

The following table describes parameters that are required when you use az k8s-extension create to create an extension instance:

Parameter name	Description
`--name`	The name of the extension instance.
`--extension-type`	The type of extension you want to install on the cluster. For example, `Microsoft.AzureMonitor.Containers` or `microsoft.azuredefender.kubernetes`.
`--scope`	The scope of installation for the extension. Use `cluster` or `namespace`.
`--cluster-name`	The name of the Azure Arc-enabled Kubernetes resource on which to create the extension instance.
`--resource-group`	The resource group that contains the Azure Arc-enabled Kubernetes resource.
`--cluster-type`	The cluster type on which to create the extension instance. For most scenarios, use `connectedClusters`, the cluster type for an Azure Arc-enabled Kubernetes cluster.

Optional parameters

You can use one or more of these optional parameters with the required parameters for your scenario.

Note

You can choose to automatically upgrade your extension instance to the latest minor and patch versions by setting auto-upgrade-minor-version to true. You also can set the version of the extension instance manually using the --version parameter. We recommend enabling automatic upgrades for minor and patch versions so that you always have the latest security patches and capabilities.

Because major version upgrades may include breaking changes, automatic upgrades for new major versions of an extension instance aren't supported. You can choose when to manually upgrade an extension instances to a new major version.

Parameter name	Description
`--auto-upgrade-minor-version`	A Boolean property that sets whether the extension minor version upgrades automatically. The default setting is `true`. If this parameter is set to `true`, you can't set the `version` parameter because the version is dynamically updated. If this parameter is set to `false`, the extension isn't be automatically upgraded, even for patch versions.
`--version`	The version of the extension to be installed (the specific version to pin the extension instance to). You can't set the `version` parameter if `auto-upgrade-minor-version` is set to `true`.
`--configuration-settings`	Settings that can be passed into the extension to control its functionality. These settings are passed in as space-separated `key=value` pairs after the parameter name. If this parameter is used in the command, you can't pass `--configuration-settings-file` in the same command.
`--configuration-settings-file`	The path to a JSON file with `key=value` pairs to be used for passing configuration settings into the extension. If this parameter is used in the command, you can't use `--configuration-settings` in the same command.
`--configuration-protected-settings`	Settings that aren't retrievable using `GET` API calls or `az k8s-extension show` commands. Typically used to pass in sensitive settings. These settings are passed in as space-separated `key=value` pairs after the parameter name. If this parameter is used in the command, you can't use `--configuration-protected-settings-file` in the same command.
`--configuration-protected-settings-file`	The path to a JSON file with `key=value` pairs to be used to pass sensitive settings into the extension. If this parameter is used in the command, you can't use `--configuration-protected-settings` in the same command.
`--release-namespace`	This parameter indicates the namespace in which to create the release. This parameter is relevant only if `scope` is set to `cluster`.
`--release-train`	The author of an extension can publish versions in different release trains, such as `Stable` or `Preview`. If this parameter isn't set explicitly, `Stable` is the default.
`--target-namespace`	Indicates the namespace within which to create the release. Permissions for the system account that's created for this extension instance is restricted to this namespace. This setting is relevant only if `scope` is set to `namespace`.

Show extension details

To view details of a currently installed extension instance, use the k8s-extension show command. In the code, use values from your scenario for the required parameter placeholders.

az k8s-extension show --name akvsecretsprovider --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type connectedClusters

Output:

{
  "aksAssignedIdentity": null,
  "autoUpgradeMinorVersion": true,
  "configurationProtectedSettings": {},
  "configurationSettings": {},
  "customLocationSettings": null,
  "errorInfo": null,
  "extensionType": "microsoft.azurekeyvaultsecretsprovider",
  "id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/demo/providers/Microsoft.Kubernetes/connectedClusters/chenyeaks/providers/Microsoft.KubernetesConfiguration/extensions/akvsecretsprovider",
  "identity": {
    "principalId": "3f9c9566-1d32-4861-9f36-9739bf853420",
    "tenantId": null,
    "type": "SystemAssigned"
  },
  "installedVersion": null,
  "name": "akvsecretsprovider",
  "packageUri": null,
  "provisioningState": "Succeeded",
  "releaseTrain": "Stable",
  "resourceGroup": "demo",
  "scope": {
    "cluster": {
      "releaseNamespace": "kube-system"
    },
    "namespace": null
  },
  "statuses": [],
  "systemData": {
    "createdAt": "2023-09-13T02:34:19.549739+00:00",
    "createdBy": null,
    "createdByType": null,
    "lastModifiedAt": "2023-09-13T02:34:19.549739+00:00",
    "lastModifiedBy": null,
    "lastModifiedByType": null
  },
  "type": "Microsoft.KubernetesConfiguration/extensions",
  "version": "1.4.2"
}

List all extensions installed on the cluster

To view a list of all extensions that are installed on a cluster, use the k8s-extension list command. In the code, use values from your scenario for the required parameter placeholders.

az k8s-extension list --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type connectedClusters

Output:

[
  {
    "aksAssignedIdentity": null,
    "autoUpgradeMinorVersion": true,
    "configurationProtectedSettings": {},
    "configurationSettings": {},
    "customLocationSettings": null,
    "errorInfo": null,
    "extensionType": "microsoft.azurekeyvaultsecretsprovider",
    "id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/demo/providers/Microsoft.Kubernetes/connectedClusters/chenyeaks/providers/Microsoft.KubernetesConfiguration/extensions/akvsecretsprovider",
    "identity": {
      "principalId": "3f9c9566-1d32-4861-9f36-9739bf853420",
      "tenantId": null,
      "type": "SystemAssigned"
    },
    "installedVersion": null,
    "name": "akvsecretsprovider",
    "packageUri": null,
    "provisioningState": "Succeeded",
    "releaseTrain": "Stable",
    "resourceGroup": "demo",
    "scope": {
      "cluster": {
        "releaseNamespace": "kube-system"
      },
      "namespace": null
    },
    "statuses": [],
    "systemData": {
      "createdAt": "2023-09-13T02:34:19.549739+00:00",
      "createdBy": null,
      "createdByType": null,
      "lastModifiedAt": "2023-09-13T02:34:19.549739+00:00",
      "lastModifiedBy": null,
      "lastModifiedByType": null
    },
    "type": "Microsoft.KubernetesConfiguration/extensions",
    "version": "1.4.2"
  }
]

Update an extension instance

Note

To understand the specific settings in --configuration-settings and --configuration-protected-settings that can be updated, see the documentation for the specific extension type. For --configuration-protected-settings, provide all settings, even if only one setting is updated. If any of these settings are omitted, the omitted settings will be considered obsolete, and they are deleted.

To update an existing extension instance, use k8s-extension update. Pass in values for the mandatory and optional parameters. The mandatory and optional parameters are slightly different from the parameters that you use to create an extension instance.

This example updates the auto-upgrade-minor-version setting for an Azure Machine Learning extension instance to true:

az k8s-extension update --name akvsecretsprovider --cluster-resource-provider Microsoft.AzureKeyvaultSecretsProvider --cluster-name <clusterName> --resource-group <resourceGroupName> --auto-upgrade-minor-version false --cluster-type connectedClusters

Required parameters for an update

Parameter name	Description
`--name`	The name of the extension instance.
`--cluster-name`	The name of the cluster on which the extension instance is created.
`--resource-group`	The resource group that contains the cluster.
`--cluster-type`	The cluster type on which the extension instance has to be created. For Azure Arc-enabled Kubernetes clusters, use `connectedClusters`. For AKS clusters, use `managedClusters`.

Optional parameters for an update

Parameter name	Description
`--auto-upgrade-minor-version`	A Boolean property that specifies whether the extension minor version is automatically upgraded. The default setting is `true`. If this parameter is set to `true`, you can't set the `version` parameter because the version is dynamically updated. If the parameter is set to `false`, the extension isn't automatically upgraded, even for patch versions.
`--version`	The version of the extension to be installed (a specific version to pin the extension instance to). Must not be supplied if `auto-upgrade-minor-version` is set to `true`.
`--configuration-settings`	Settings that can be passed into the extension to control its functionality. These settings are passed in as space-separated `key=value` pairs after the parameter name. If the parameter is used in the command, `--configuration-settings-file` can't be used in the same command. Only the settings that require an update need to be provided. The provided settings are replaced with the specified values.
`--configuration-settings-file`	The path to a JSON file that contains `key=value` pairs to be used for passing in configuration settings to the extension. If this parameter is used in the command, you can't use `--configuration-settings` in the same command.
`--configuration-protected-settings`	Settings that aren't retrievable by using `GET` API calls or `az k8s-extension show` commands. Typically used to pass in sensitive settings. These settings are passed in as space-separated `key=value` pairs after the parameter name. If this parameter is used in the command, the `--configuration-protected-settings-file` can't be used in the same command. When you update a protected setting, configure all protected settings. If any of the settings are omitted, those settings are considered obsolete, and they're deleted.
`--configuration-protected-settings-file`	The path to a JSON file that contains `key=value` pairs to be used for passing in sensitive settings to the extension. If this parameter is used in the command, you can't use `--configuration-protected-settings` in the same command.
`--scope`	The scope of installation for the extension. Use either `cluster` or `namespace`.
`--release-train`	The author of an extension can publish versions in different release trains, such as `Stable` or `Preview`. If this parameter isn't set explicitly, `Stable` is the default.

Upgrade an extension instance

As noted earlier, if you set auto-upgrade-minor-version to true, the extension is automatically upgraded when a new minor version is released. For most scenarios, we recommend that you enable automatic upgrades. If you set auto-upgrade-minor-version to false, you must upgrade the extension manually if you want a more recent version.

Manual upgrades also are required to get a new major instance of an extension. You can choose when to upgrade to avoid any unexpected breaking changes in major version upgrades.

To manually upgrade an extension instance, use k8s-extension update and set the version parameter.

This example updates an zure Key Vault Secrets Provider instance to version x.y.z:

az k8s-extension update --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type connectedClusters --name akvsecretsprovider --version x.y.z

Delete an extension instance

To delete an extension instance on a cluster, use the k8s-extension delete command. Use values from your scenario for the required parameter placeholders.

az k8s-extension delete --name akvsecretsprovider --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type connectedClusters

Note

The Azure resource that represents this extension is immediately deleted. The Helm release on the cluster that's associated with this extension is deleted only when the agents running on the Kubernetes cluster have network connectivity and can reach Azure services to get the desired state.

For a comprehensive list of commands and parameters, review the az k8s-extension CLI reference.
Learn more about how extensions work with Azure Arc-enabled Kubernetes clusters.
Review cluster extensions that are available for Azure Arc-enabled Kubernetes.

Deploy and manage an Azure Arc-enabled Kubernetes cluster extension

Prerequisites

Create an extension instance

Required parameters

Optional parameters

Show extension details

List all extensions installed on the cluster

Update an extension instance

Required parameters for an update

Optional parameters for an update

Upgrade an extension instance

Delete an extension instance

Related content

Additional resources