Manage Entra ID-enabled Azure HDInsight clusters with ARM templates

Azure HDInsight now supports integration with Microsoft Entra ID, enabling secure, role-based access control and simplified identity management for your big data workloads. By using Azure Resource Manager (ARM) templates, you can deploy and manage Entra ID-enabled HDInsight clusters in a repeatable, automated, and consistent way. In this guide, you'll learn how to set up ARM template resources and configure Entra ID. You'll also see how to manage clusters at scale while meeting your organization's security and governance rules.

Prerequisites

Before you begin, ensure you have the following resources:

• Azure subscription with sufficient permissions to deploy resources.

• Microsoft Entra ID tenant configured and accessible.

• Service principal or managed identity with the necessary role assignments (for example, Contributor and User Access Administrator) to create and manage HDInsight clusters.

• Azure CLI, PowerShell, or Azure portal access to validate deployment and monitor cluster status.

• ARM template and parameter files that define the cluster configuration and Entra ID integration settings.

ARM template resource definition

The clusters resource type can be deployed with operations that target Resource groups - See resource group deployment commands

For a list of changed properties in each API version, see change log.

Resource format

To create a Microsoft.HDInsight/clusters resource, add the following JSON to your template.

				{
					"$schema": "https://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#",
					"contentVersion": "0.9.0.0",
					"parameters": {
						"clusterName": {
							"type": "string",
							"metadata": {
								"description": "The name of the HDInsight cluster to create."
							}
						},
						"location": {
							"type": "string",
							"defaultValue": "chinaeast2euap",
							"metadata": {
								"description": "The location where all azure resources will be deployed."
							}
						},
						"clusterVersion": {
							"type": "string",
							"defaultValue": "5.1",
							"metadata": {
								"description": "HDInsight cluster version."
							}
						},
						"clusterWorkerNodeCount": {
							"type": "int",
							"defaultValue": 4,
							"metadata": {
								"description": "The number of nodes in the HDInsight cluster."
							}
						},
						"clusterKind": {
							"type": "string",
							"defaultValue": "SPARK",
							"metadata": {
								"description": "The type of the HDInsight cluster to create."
							}
						},
						"sshUserName": {
							"type": "string",
							"defaultValue": "sshuser",
							"metadata": {
								"description": "These credentials can be used to remotely access the cluster."
							}
						},
						"clusterRestAuthEntraUsers": {
							"type": "string",
							"metadata": {
								"description": "These Micrsoft Entra users can be used to submit jobs to the cluster and to log into cluster dashboards."
							}
						},
						"sshPassword": {
							"type": "securestring",
							"metadata": {
								"description": "The password must be at least 10 characters in length and must contain at least one digit, one non-alphanumeric character, and one upper or lower case letter."
							}
						},
						"minTlsVersionNumber": {
							"type": "string"
						},
						"isEncryptionInTransitEnabled": {
							"type": "bool"
						}
					},
					"resources": [
						{
							"apiVersion": "2025-01-15-preview",
							"name": "[parameters('clusterName')]",
							"type": "Microsoft.HDInsight/clusters",
							"location": "[parameters('location')]",
							"dependsOn": [],
							"tags": {},
							"zones": null,
							"properties": {
								"clusterVersion": "[parameters('clusterVersion')]",
								"osType": "Linux",
								"tier": "standard",
								"clusterDefinition": {
									"kind": "[parameters('clusterKind')]",
									"componentVersion": {
										"Spark": "3.3"
									},
									"configurations": {
										"gateway": {
											"restAuthCredential.isEnabled": false,
											"restAuthEntraUsers": "[parameters('clusterRestAuthEntraUsers')]"
										}
									}
								},
								"storageProfile": {
									"storageaccounts": [
										{
											"name": "<storageAccountName>.dfs.core.chinacloudapi.cn",
											"isDefault": true,
											"fileSystem": "hmgespark1-2025-10-06t08-34-17-604z",
											"resourceId": "/subscriptions/SubscriptionID>/resourceGroups/<resourceGroupName>/providers/Microsoft.Storage/storageAccounts/<MSIname>",
											"msiResourceId": "/subscriptions/<SubscriptionID>/resourcegroups/<resourceGroupName>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<MSIname>",
											"enableSecureChannel": true
										}
									]
								},
								"computeProfile": {
									"roles": [
										{
											"autoscale": null,
											"name": "headnode",
											"minInstanceCount": 1,
											"targetInstanceCount": 2,
											"hardwareProfile": {
												"vmSize": "Standard_E8_V3"
											},
											"osProfile": {
												"linuxOperatingSystemProfile": {
													"username": "[parameters('sshUserName')]",
													"password": "[parameters('sshPassword')]"
												},
												"windowsOperatingSystemProfile": null
											},
											"virtualNetworkProfile": null,
											"scriptActions": [],
											"dataDisksGroups": null
										},
										{
											"autoscale": {
												"capacity": {
													"minInstanceCount": 4,
													"maxInstanceCount": 5
												},
												"recurrence": null
											},
											"name": "workernode",
											"targetInstanceCount": 4,
											"hardwareProfile": {
												"vmSize": "Standard_E8_V3"
											},
											"osProfile": {
												"linuxOperatingSystemProfile": {
													"username": "[parameters('sshUserName')]",
													"password": "[parameters('sshPassword')]"
												},
												"windowsOperatingSystemProfile": null
											},
											"virtualNetworkProfile": null,
											"scriptActions": [],
											"dataDisksGroups": null
										},
										{
											"autoscale": null,
											"name": "zookeepernode",
											"minInstanceCount": 1,
											"targetInstanceCount": 3,
											"hardwareProfile": {
												"vmSize": "Standard_A2_V2"
											},
											"osProfile": {
												"linuxOperatingSystemProfile": {
													"username": "[parameters('sshUserName')]",
													"password": "[parameters('sshPassword')]"
												},
												"windowsOperatingSystemProfile": null
											},
											"virtualNetworkProfile": null,
											"scriptActions": [],
											"dataDisksGroups": null
										}
									]
								},
								"minSupportedTlsVersion": "[parameters('minTlsVersionNumber')]",
								"encryptionInTransitProperties": {
									"isEncryptionInTransitEnabled": "[parameters('isEncryptionInTransitEnabled')]"
								}
							},
							"identity": {
								"type": "UserAssigned",
								"userAssignedIdentities": {
									"/subscriptions/<subcriptionID>/resourcegroups/<resourceGroupName>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<MSIname>": {}
								}
							}
						}
					]
				}

Parameters

     				{
				  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
				  "contentVersion": "1.0.0.0",
				  "parameters": {
					"clusterName": {
					  "value": "<clustername>"
					},
					"location": {
					  "value": "<region_name>"
					},
					"clusterVersion": {
					  "value": "5.1"
					},
					"clusterWorkerNodeCount": {
					  "value": 4
					},
					"clusterKind": {
					  "value": "SPARK"
					},
					"sshUserName": {
					  "value": "sshuser"
					},
					"clusterRestAuthEntraUsers": {
					  "value": "[{\"displayName\":\"<Name>\",\"objectId\":\"00000000-0000-0000-0000-1ed7871c38e0\",\"upn\":\"user2@contoso.com\"},{\"displayName\":\"<Name>\",\"objectId\":\"00000000-0000-0000-000-b44d6570aa30\",\"upn\":\"user1@contoso.com\"}]"
					},
					"sshPassword": {
					  "value": null
					},
					"minTlsVersionNumber": {
					  "value": "1.2"
					},
					"isEncryptionInTransitEnabled": {
					  "value": true
					}
				  }
				}

Property Values

Autoscale

AutoscaleCapacity

AutoscaleRecurrence

AutoscaleSchedule

AutoscaleTimeAndCapacity

ClusterCreatePropertiesOrClusterGetProperties

ClusterDefinition

ClusterIdentity

ComputeIsolationProperties

ComputeProfile

DataDisksGroups

DiskEncryptionProperties

EncryptionInTransitProperties

Hardware Profile

IP Configuration

IPConfigurationProperties

IpTag

KafkaRestProperties

LinuxOperatingSystemProfile

Microsoft.HDInsight/clusters

NetworkProperties

OsProfile

PrivateLinkConfiguration

PrivateLinkConfigurationProperties

ResourceID

Role

ScriptAction

StorageAccount

StorageProfile

UserAssignedIdentity

VirtualNetworkProfile

By managing Microsoft Entra ID-enabled Azure HDInsight clusters with ARM templates, you gain a consistent, repeatable, and automated way to deploy and configure secure big data environments. ARM templates allow you to define all cluster settings including identity, networking, storage, and security as code, ensuring that deployments are predictable and compliant with organizational standards.

This approach not only streamlines cluster provisioning and management but also integrates seamlessly with CI/CD pipelines, enabling you to scale and govern your HDInsight workloads with confidence.