HDInsight cluster management best practices

Learn best practices for managing HDInsight clusters.

How do I create HDInsight clusters?

Note

If you are creating a cluster and re-using the cluster name from a previously created cluster, wait until the previous cluster deletion is completed before creating your cluster.

How do I customize HDInsight clusters?

What are some errors I might face when creating clusters?

Error More information
No quota There are quotas for the number of cores that you can create on your subscription in each region. For more information, see Capacity planning: quotas.
No more IP addresses available Each VNet has a limited number of IP addresses. When you create a HDInsight cluster, each node (including zookeeper and gateway nodes) uses some of these allotted IP addresses. When all of the IP addresses are in use, you will encounter this error.
Network security group (NSG) rules don't allow communication with HDInsight resource providers If you use NSGs or user-defined routes (UDRs) to control inbound traffic to your HDInsight cluster, you must ensure that your cluster can communicate with critical Azure health and management services. For more information, see Network security group (NSG) service tags for Azure HDInsight
Reuse of cluster name When you use a cluster name that you have used before, you need to wait X number of minutes before recreating the cluster. Otherwise you will see a message that the resource already exists.

How do I manage running HDInsight clusters?

Option Documents
Autoscale Automatically scale Azure HDInsight clusters
Manual scaling Scale Azure HDInsight clusters
Monitoring with Ambari Monitor cluster performance in Azure HDInsight
Service issues, planned maintenance, health & security advisories Subscribe to subscription specific service health alerts

How do I check on deleted HDInsight clusters?

Azure Monitor logs

You can use the following query with Azure Monitor logs to monitor deleted clusters.

AzureActivity
| where ResourceProvider == "Microsoft.HDInsight" and (OperationName == "Create or Update Cluster" or OperationName == "Delete Cluster") and ActivityStatus == "Succeeded"

Next steps