Create and configure Enterprise Security Package clusters in Azure HDInsight

  • Article

Enterprise Security Package (ESP) for Azure HDInsight gives you access to Active Directory-based authentication, multiuser support, and role-based access control for your Apache Hadoop clusters in Azure. HDInsight ESP clusters enable organizations that adhere to strict corporate security policies to process sensitive data securely.

This guide shows how to create an ESP-enabled Azure HDInsight cluster. It also shows how to create a Windows IaaS VM on which Active Directory and Domain Name System (DNS) are enabled. Use this guide to configure the necessary resources to allow on-premises users to sign in to an ESP-enabled HDInsight cluster.

The server you create will act as a replacement for your actual on-premises environment. You'll use it for the setup and configuration steps. Later you'll repeat the steps in your own environment.

This guide will also help you create a hybrid identity environment by using password hash sync with Microsoft Entra ID. The guide complements Use ESP in HDInsight.

Before you use this process in your own environment:

  • Set up Active Directory and DNS.
  • Enable Microsoft Entra ID.
  • Sync on-premises user accounts to Microsoft Entra ID.

Microsoft Entra architecture diagram.

Create an on-premises environment

In this section, you'll use an Azure Quickstart deployment template to create new VMs, configure DNS, and add a new Active Directory forest.

  1. Go to the Quickstart deployment template to Create an Azure VM with a new Active Directory forest.

  2. Select Deploy to Azure.

  3. Sign in to your Azure subscription.

  4. On the Create an Azure VM with a new AD Forest page, provide the following information:

    Property Value
    Subscription Select the subscription where you want to deploy the resources.
    Resource group Select Create new, and enter the name OnPremADVRG
    Location Select a location.
    Admin Username HDIFabrikamAdmin
    Admin Password Enter a password.
    Domain Name HDIFabrikam.com
    Dns Prefix hdifabrikam

    Leave the remaining default values.

    Template for Create an Azure VM with a new Microsoft Entra Forest.

  5. Review the Terms and Conditions, and then select I agree to the terms and conditions stated above.

  6. Select Purchase, and monitor the deployment and wait for it to complete. The deployment takes about 30 minutes to complete.

Configure users and groups for cluster access

In this section, you'll create the users that will have access to the HDInsight cluster by the end of this guide.

  1. Connect to the domain controller by using Remote Desktop.

    1. From the Azure portal, navigate to Resource groups > OnPremADVRG > adVM > Connect.
    2. From the IP address drop-down list, select the public IP address.
    3. Select Download RDP File, and then open the file.
    4. Use HDIFabrikam\HDIFabrikamAdmin as the user name.
    5. Enter the password that you chose for the admin account.
    6. Select OK.

  2. From the domain controller Server Manager dashboard, navigate to Tools > Active Directory Users and Computers.

    On the Server Manager dashboard, open Active Directory Management.

  3. Create two new users: HDIAdmin and HDIUser. These two users will sign in to HDInsight clusters.

    1. From the Active Directory Users and Computers page, right-click HDIFabrikam.com, and then navigate to New > User.

      Create a new Active Directory user.

    2. On the New Object - User page, enter HDIUser for First name and User logon name. The other fields will autopopulate. Then select Next.

      Create the first admin user object.

    3. In the pop-up window that appears, enter a password for the new account. Select Password never expires, and then OK at the pop-up message.

    4. Select Next, and then Finish to create the new account.

    5. Repeat the above steps to create the user HDIAdmin.

      Create a second admin user object.

  4. Create a security group.

    1. From Active Directory Users and Computers, right-click HDIFabrikam.com, and then navigate to New > Group.

    2. Enter HDIUserGroup in the Group name text box.

    3. Select OK.

    Create a new Active Directory group.

    Create a new object.

  5. Add members to HDIUserGroup.

    1. Right-click HDIUser and select Add to a group....

    2. In the Enter the object names to select text box, enter HDIUserGroup. Then select OK, and OK again at the pop-up.

    3. Repeat the previous steps for the HDIAdmin account.

      Add the member HDIUser to the group HDIUserGroup.

You've now created your Active Directory environment. You've added two users and a user group that can access the HDInsight cluster.

The users will be synchronized with Microsoft Entra ID.

Create a Microsoft Entra directory

  1. Sign in to the Azure portal.

  2. Select Create a resource and type directory. Select Microsoft Entra ID > Create.

  3. Under Organization name, enter HDIFabrikam.

  4. Under Initial domain name, enter HDIFabrikamoutlook.

  5. Select Create.

    Create a Microsoft Entra directory.

Create a custom domain

  1. From your new Microsoft Entra ID, under Manage, select Custom domain names.

  2. Select + Add custom domain.

  3. Under Custom domain name, enter HDIFabrikam.com, and then select Add domain.

  4. Then complete Add your DNS information to the domain registrar.

    Create a custom domain.

Create a group

  1. From your new Microsoft Entra ID, under Manage, select Groups.
  2. Select + New group.
  3. In the group name text box, enter AAD DC Administrators.
  4. Select Create.

Configure your Microsoft Entra tenant

Now you'll configure your Microsoft Entra tenant so that you can synchronize users and groups from the on-premises Active Directory instance to the cloud.

Create an Active Directory tenant administrator.

  1. Sign in to the Azure portal and select your Microsoft Entra tenant, HDIFabrikam.

  2. Navigate to Manage > Users > New user.

  3. Enter the following details for the new user:

    Identity

    Property Description
    User name Enter fabrikamazureadmin in the text box. From the domain name drop-down list, select hdifabrikam.com
    Name Enter fabrikamazureadmin.

    Password

    1. Select Let me create the password.
    2. Enter a secure password of your choice.

    Groups and roles

    1. Select 0 groups selected.

    2. Select AAD DC Administrators, and then Select.

      The Microsoft Entra groups dialog box.

    3. Select User.

    4. Select Administrator, and then Select.

      The Microsoft Entra role dialog box.

  4. Select Create.

  5. Then have the new user sign in to the Azure portal where it will be prompted to change the password. You'll need to do this before configuring Microsoft Entra Connect.

Sync on-premises users to Microsoft Entra ID

Configure Microsoft Entra Connect

  1. From the domain controller, download Microsoft Entra Connect.

  2. Open the executable file that you downloaded, and agree to the license terms. Select Continue.

  3. Select Use express settings.

  4. On the Connect to Microsoft Entra ID page, enter the username and password of the administrator for Microsoft Entra ID. Use the username fabrikamazureadmin@hdifabrikam.com that you created when you configured your Active Directory tenant. Then select Next.

    Connect to Microsoft Entra ID.

  5. On the Connect to Active Directory Domain Services page, enter the username and password for an enterprise admin account. Use the username HDIFabrikam\HDIFabrikamAdmin and its password that you created earlier. Then select Next.

    Connect to A D D S page.

  6. On the Microsoft Entra sign-in configuration page, select Next.

    Microsoft Entra sign-in configuration page.

  7. On the Ready to configure page, select Install.

    Ready to configure page.

  8. On the Configuration complete page, select Exit. Configuration complete page.

  9. After the sync completes, confirm that the users you created on the IaaS directory are synced to Microsoft Entra ID.

    1. Sign in to the Azure portal.
    2. Select Microsoft Entra ID > HDIFabrikam > Users.

Create a user-assigned managed identity

Create a user-assigned managed identity that you can use to configure Microsoft Entra Domain Services. For more information, see Create, list, delete, or assign a role to a user-assigned managed identity by using the Azure portal.

  1. Sign in to the Azure portal.
  2. Select Create a resource and type managed identity. Select User Assigned Managed Identity > Create.
  3. For the Resource Name, enter HDIFabrikamManagedIdentity.
  4. Select your subscription.
  5. Under Resource group, select Create new and enter HDIFabrikam-chinanorth.
  6. Under Location, select China North.
  7. Select Create.

Create a new user-assigned managed identity.

Enable Microsoft Entra Domain Services

Follow these steps to enable Microsoft Entra Domain Services. For more information, see Enable Microsoft Entra Domain Services by using the Azure portal.

  1. Create a virtual network to host Microsoft Entra Domain Services. Run the following PowerShell code.

    # Sign in to your Azure subscription
$sub = Get-AzSubscription -ErrorAction SilentlyContinue
if(-not($sub))
{
    Connect-AzAccount -Environment AzureChinaCloud
}

# If you have multiple subscriptions, set the one to use
# Select-AzSubscription -SubscriptionId "<SUBSCRIPTIONID>"

$virtualNetwork = New-AzVirtualNetwork -ResourceGroupName 'HDIFabrikam-chinanorth' -Location 'China North' -Name 'HDIFabrikam-AADDSVNET' -AddressPrefix 10.1.0.0/16
$subnetConfig = Add-AzVirtualNetworkSubnetConfig -Name 'AADDS-subnet' -AddressPrefix 10.1.0.0/24 -VirtualNetwork $virtualNetwork
$virtualNetwork | Set-AzVirtualNetwork

  2. Sign in to the Azure portal.

  3. Select Create resource, enter Domain services, and select Microsoft Entra Domain Services > Create.

  4. On the Basics page:

    1. Under Directory name, select the Microsoft Entra directory you created: HDIFabrikam.

    2. For DNS domain name, enter HDIFabrikam.com.

    3. Select your subscription.

    4. Specify the resource group HDIFabrikam-chinanorth. For Location, select China North.

      Microsoft Entra Domain Services basic details.

  5. On the Network page, select the network (HDIFabrikam-VNET) and the subnet (AADDS-subnet) that you created by using the PowerShell script. Or choose Create new to create a virtual network now.

    Create virtual network step.

  6. On the Administrator group page, you should see a notification that a group named AAD DC Administrators has already been created to administer this group. You can modify the membership of this group if you want to, but in this case you don't need to change it. Select OK.

    View the Microsoft Entra administrator group.

  7. On the Synchronization page, enable complete synchronization by selecting All > OK.

    Enable Microsoft Entra Domain Services synchronization.

  8. On the Summary page, verify the details for Microsoft Entra Domain Services and select OK.

    Enable Microsoft Entra Domain Services.

After you enable Microsoft Entra Domain Services, a local DNS server runs on the Microsoft Entra VMs.

Configure your Microsoft Entra Domain Services virtual network

Use the following steps to configure your Microsoft Entra Domain Services virtual network (HDIFabrikam-AADDSVNET) to use your custom DNS servers.

  1. Locate the IP addresses of your custom DNS servers.

    1. Select the HDIFabrikam.com Microsoft Entra Domain Services resource.
    2. Under Manage, select Properties.
    3. Find the IP addresses under IP address on virtual network.

    Locate custom DNS IP addresses for Microsoft Entra Domain Services.

  2. Configure HDIFabrikam-AADDSVNET to use custom IP addresses 10.0.0.4 and 10.0.0.5.

    1. Under Settings, select DNS Servers.
    2. Select Custom.
    3. In the text box, enter the first IP address (10.0.0.4).
    4. Select Save.
    5. Repeat the steps to add the other IP address (10.0.0.5).

In our scenario, we configured Microsoft Entra Domain Services to use IP addresses 10.0.0.4 and 10.0.0.5, setting the same IP address on the Microsoft Entra Domain Services virtual network:

The custom DNS servers page.

Securing LDAP traffic

Lightweight Directory Access Protocol (LDAP) is used to read from and write to Microsoft Entra ID. You can make LDAP traffic confidential and secure by using Secure Sockets Layer (SSL) or Transport Layer Security (TLS) technology. You can enable LDAP over SSL (LDAPS) by installing a properly formatted certificate.

For more information about secure LDAP, see Configure LDAPS for a Microsoft Entra Domain Services managed domain.

In this section, you create a self-signed certificate, download the certificate, and configure LDAPS for the HDIFabrikam Microsoft Entra Domain Services managed domain.

The following script creates a certificate for HDIFabrikam. The certificate is saved in the LocalMachine path.

$lifetime = Get-Date
New-SelfSignedCertificate -Subject hdifabrikam.com `
-NotAfter $lifetime.AddDays(365) -KeyUsage DigitalSignature, KeyEncipherment `
-Type SSLServerAuthentication -DnsName *.hdifabrikam.com, hdifabrikam.com

Note

Any utility or application that creates a valid Public Key Cryptography Standards (PKCS) #10 request can be used to form the TLS/SSL certificate request.

Verify that the certificate is installed in the computer's Personal store:

  1. Start Microsoft Management Console (MMC).

  2. Add the Certificates snap-in that manages certificates on the local computer.

  3. Expand Certificates (Local Computer) > Personal > Certificates. A new certificate should exist in the Personal store. This certificate is issued to the fully qualified host name.

    Verify local certificate creation.

  4. In pane on the right, right-click the certificate that you created. Point to All Tasks, and then select Export.

  5. On the Export Private Key page, select Yes, export the private key. The computer where the key will be imported needs the private key to read the encrypted messages.

    The Export Private Key page of the Certificate Export Wizard.

  6. On the Export File Format page, leave the default settings, and then select Next.

  7. On the Password page, type a password for the private key. For Encryption, select TripleDES-SHA1. Then select Next.

  8. On the File to Export page, type the path and the name for the exported certificate file, and then select Next. The file name has to have a .pfx extension. This file is configured in the Azure portal to establish a secure connection.

  9. Enable LDAPS for a Microsoft Entra Domain Services managed domain.

    1. From the Azure portal, select the domain HDIFabrikam.com.
    2. Under Manage, select Secure LDAP.
    3. On the Secure LDAP page, under Secure LDAP, select Enable.
    4. Browse for the .pfx certificate file that you exported on your computer.
    5. Enter the certificate password.

    Enable secure LDAP.

  10. Now that you've enabled LDAPS, make sure it's reachable by enabling port 636.

    1. In the HDIFabrikam-chinanorth resource group, select the network security group AADDS-HDIFabrikam.com-NSG.

    2. Under Settings, select Inbound security rules > Add.

    3. On the Add inbound security rule page, enter the following properties, and select Add:

      Property Value
      Source Any
      Source port ranges *
      Destination Any
      Destination port range 636
      Protocol Any
      Action Allow
      Priority <Desired number>
      Name Port_LDAP_636

      The Add inbound security rule dialog box.

HDIFabrikamManagedIdentity is the user-assigned managed identity. The HDInsight Domain Services Contributor role is enabled for the managed identity that will allow this identity to read, create, modify, and delete domain services operations.

Create a user-assigned managed identity.

Create an ESP-enabled HDInsight cluster

This step requires the following prerequisites:

  1. Create a new resource group HDIFabrikam-ChinaNorth in the location China North.

  2. Create a virtual network that will host the ESP-enabled HDInsight cluster.

    $virtualNetwork = New-AzVirtualNetwork -ResourceGroupName 'HDIFabrikam-ChinaNorth' -Location 'China North' -Name 'HDIFabrikam-HDIVNet' -AddressPrefix 10.1.0.0/16
$subnetConfig = Add-AzVirtualNetworkSubnetConfig -Name 'SparkSubnet' -AddressPrefix 10.1.0.0/24 -VirtualNetwork $virtualNetwork
$virtualNetwork | Set-AzVirtualNetwork

  3. Create a peer relationship between the virtual network that hosts Microsoft Entra Domain Services (HDIFabrikam-AADDSVNET) and the virtual network that will host the ESP-enabled HDInsight cluster (HDIFabrikam-HDIVNet). Use the following PowerShell code to peer the two virtual networks.

    Add-AzVirtualNetworkPeering -Name 'HDIVNet-AADDSVNet' -RemoteVirtualNetworkId (Get-AzVirtualNetwork -ResourceGroupName 'HDIFabrikam-chinanorth').Id -VirtualNetwork (Get-AzVirtualNetwork -ResourceGroupName 'HDIFabrikam-ChinaNorth')

Add-AzVirtualNetworkPeering -Name 'AADDSVNet-HDIVNet' -RemoteVirtualNetworkId (Get-AzVirtualNetwork -ResourceGroupName 'HDIFabrikam-ChinaNorth').Id -VirtualNetwork (Get-AzVirtualNetwork -ResourceGroupName 'HDIFabrikam-chinanorth')

  4. Create a new Azure Data Lake Storage Gen2 account called Hdigen2store. Configure the account with the user-managed identity HDIFabrikamManagedIdentity. For more information, see Use Azure Data Lake Storage Gen2 with Azure HDInsight clusters.

  5. Set up custom DNS on the HDIFabrikam-AADDSVNET virtual network.

    1. Go to the Azure portal > Resource groups > OnPremADVRG > HDIFabrikam-AADDSVNET > DNS servers.

    2. Select Custom and enter 10.0.0.4 and 10.0.0.5.

    3. Select Save.

      Save custom DNS settings for a virtual network.

  6. Create a new ESP-enabled HDInsight Spark cluster.

    1. Select Custom (size, settings, apps).

    2. Enter details for Basics (section 1). Ensure that the Cluster type is Spark 2.3 (HDI 3.6). Ensure that the Resource group is HDIFabrikam-chinanorth.

    3. For Security + networking (section 2), fill in the following details:

      • Under Enterprise Security Package, select Enabled.

      • Select Cluster admin user and select the HDIAdmin account that you created as the on-premises admin user. Click Select.

      • Select Cluster access group > HDIUserGroup. Any user that you add to this group in the future will be able to access HDInsight clusters.

        Select the cluster access group HDIUserGroup.

    4. Complete the other steps of the cluster configuration and verify the details on the Cluster summary. Select Create.

  7. Sign in to the Ambari UI for the newly created cluster at https://CLUSTERNAME.azurehdinsight.cn. Use your admin username hdiadmin@hdifabrikam.com and its password.

    The Apache Ambari UI sign-in window.

  8. From the cluster dashboard, select Roles.

  9. On the Roles page, under Assign roles to these, next to the Cluster Administrator role, enter the group hdiusergroup.

    Assign the cluster admin role to hdiusergroup.

  10. Open your Secure Shell (SSH) client and sign in to the cluster. Use the hdiuser that you created in the on-premises Active Directory instance.

    Sign in to the cluster by using the SSH client.

If you can sign in with this account, you've configured your ESP cluster correctly to sync with your on-premises Active Directory instance.

Next steps

Read An introduction to Apache Hadoop security with ESP.