Synchronize Microsoft Entra users to an HDInsight cluster

Article
03/25/2024

HDInsight clusters with Enterprise Security Package (ESP) can use strong authentication with Microsoft Entra users, and use Azure role-based access control (Azure RBAC) policies. As you add users and groups to Microsoft Entra ID, you can synchronize the users who need access to your cluster.

Prerequisites

If you haven't already done so, create a HDInsight cluster with Enterprise Security Package.

Add new Microsoft Entra users

To view your hosts, open the Ambari Web UI. Each node is updated with new unattended upgrade settings.

From the Azure portal, navigate to the Microsoft Entra directory associated with your ESP cluster.
Select All users from the left-hand menu, then select New user.
Complete the new user form. Select groups you created for assigning cluster-based permissions. In this example, create a group named "HiveUsers", to which you can assign new users. The example instructions for creating an ESP cluster include adding two groups, HiveUsers and AAD DC Administrators.
Select Create.

Use the Apache Ambari REST API to synchronize users

User groups specified during the cluster creation process are synchronized at that time. User synchronization occurs automatically once every hour. To synchronize the users immediately, or to synchronize a group other than the groups specified during cluster creation, use the Ambari REST API.

The following method uses POST with the Ambari REST API. For more information, see Manage HDInsight clusters by using the Apache Ambari REST API.

Use ssh command to connect to your cluster. Edit the command by replacing CLUSTERNAME with the name of your cluster, and then enter the command:
```
ssh sshuser@CLUSTERNAME-ssh.azurehdinsight.cn
```

After authenticating, enter the following command:

curl -u admin:PASSWORD -sS -H "X-Requested-By: ambari" \
-X POST -d '{"Event": {"specs": [{"principal_type": "groups", "sync_type": "existing"}]}}' \
"https://CLUSTERNAME.azurehdinsight.cn/api/v1/ldap_sync_events"

The response should look like this:

{
  "resources" : [
    {
      "href" : "http://<ACTIVE-HEADNODE-NAME>.<YOUR DOMAIN>.com:8080/api/v1/ldap_sync_events/1",
      "Event" : {
        "id" : 1
      }
    }
  ]
}

To see the synchronization status, execute a new curl command:

curl -u admin:PASSWORD https://CLUSTERNAME.azurehdinsight.cn/api/v1/ldap_sync_events/1

The response should look like this:

{
  "href" : "http://<ACTIVE-HEADNODE-NAME>.YOURDOMAIN.com:8080/api/v1/ldap_sync_events/1",
  "Event" : {
    "id" : 1,
    "specs" : [
      {
        "sync_type" : "existing",
        "principal_type" : "groups"
      }
    ],
    "status" : "COMPLETE",
    "status_detail" : "Completed LDAP sync.",
    "summary" : {
      "groups" : {
        "created" : 0,
        "removed" : 0,
        "updated" : 0
      },
      "memberships" : {
        "created" : 1,
        "removed" : 0
      },
      "users" : {
        "created" : 1,
        "removed" : 0,
        "skipped" : 0,
        "updated" : 0
      }
    },
    "sync_time" : {
      "end" : 1497994072182,
      "start" : 1497994071100
    }
  }
}

This result shows that the status is COMPLETE, one new user was created, and the user was assigned a membership. In this example, the user is assigned to the "HiveUsers" synchronized LDAP group, since the user was added to that same group in Microsoft Entra ID.

Note

The previous method only synchronizes the Microsoft Entra groups specified in the Access user group property of the domain settings during cluster creation. For more information, see create an HDInsight cluster.

Verify the newly added Microsoft Entra user

Open the Apache Ambari Web UI to verify that the new Microsoft Entra user was added. Access the Ambari Web UI by browsing to https://CLUSTERNAME.azurehdinsight.cn. Enter the cluster administrator username and password.

From the Ambari dashboard, select Manage Ambari under the admin menu.
Select Users under the User + Group Management menu group on the left-hand side of the page.
The new user should be listed within the Users table. The Type is set to LDAP rather than Local.

Log in to Ambari as the new user

When the new user (or any other domain user) logs in to Ambari, they use their full Microsoft Entra user name and domain credentials. Ambari displays a user alias, which is the display name of the user in Microsoft Entra ID. The new example user has the user name hiveuser3@contoso.com. In Ambari, this new user shows up as hiveuser3 but the user logs into Ambari as hiveuser3@contoso.com.