Migrate to granular role-based access for cluster configurations

  • Article

We are introducing some important changes to support more fine-grained role-based access to obtain sensitive information. As part of these changes, some action may be required by September 3, 2019 if you are using one of the affected entities/scenarios.

What is changing?

Previously, secrets could be obtained via the HDInsight API by cluster users possessing the Owner, Contributor, or Reader Azure roles, as they were available to anyone with the */read permission. Secrets are defined as values that could be used to obtain more elevated access than a user's role should allow. These include values such as cluster gateway HTTP credentials, storage account keys, and database credentials.

Beginning on September 3, 2019, accessing these secrets will require the Microsoft.HDInsight/clusters/configurations/action permission, user cannot access it with the Reader role. The roles that have this permission are Contributor, Owner, and the new HDInsight Cluster Operator role.

We are also introducing a new HDInsight Cluster Operator role that able to retrieve secrets without being granted the administrative permissions of Contributor or Owner. To summarize:

Role Previously Going Forward
Reader - Read access, including secrets. - Read access, excluding secrets
HDInsight Cluster Operator
(New Role)		 N/A - Read/write access, including secrets
Contributor - Read/write access, including secrets.
- Create and manage all of types of Azure resources.
- Execute script actions.		 No change
Owner - Read/write access including secrets.
- Full access to all resources
- Delegate access to others.
- Execute script actions.		 No change

For information on how to add the HDInsight Cluster Operator role assignment to a user to grant them read/write access to cluster secrets, see the below section, Add the HDInsight Cluster Operator role assignment to a user.

Am I affected by these changes?

The following entities and scenarios are affected:

API

The following APIs are changed or deprecated:

  • GET /configurations/{configurationName} (sensitive information removed)
    • Previously used to obtain individual configuration types (including secrets).
    • Beginning on September 3, 2019, this API call will now return individual configuration types with secrets omitted. To obtain all configurations, including secrets, use the new POST /configurations call. To obtain just gateway settings, use the new POST /getGatewaySettings call.
  • GET /configurations (deprecated)
    • Previously used to obtain all configurations (including secrets)
    • Beginning on September 3, 2019, this API call will be deprecated and no longer be supported. To obtain all configurations going forward, use the new POST /configurations call. To obtain configurations with sensitive parameters omitted, use the GET /configurations/{configurationName} call.
  • POST /configurations/{configurationName} (deprecated)
    • Previously used to update gateway credentials.
    • Beginning on September 3, 2019, this API call will be deprecated and no longer supported. Use the new POST /updateGatewaySettings instead.

The following replacement APIs have been added:

Azure HDInsight Tools for Visual Studio Code

If you are using version 1.1.1 or below, update to the latest version of Azure HDInsight Tools for Visual Studio Code to avoid interruptions.

Azure Toolkit for IntelliJ

If you are using version 3.20.0 or below, update to the latest version of the Azure Toolkit for IntelliJ plugin to avoid interruptions.

Azure Data Lake and Stream Analytics Tools for Visual Studio

Update to version 2.3.9000.1 or later of Azure Data Lake and Stream Analytics Tools for Visual Studio to avoid interruptions. For help with updating, see our documentation, Update Data Lake Tools for Visual Studio.

Azure Toolkit for Eclipse

If you are using version 3.15.0 or below, update to the latest version of the Azure Toolkit for Eclipse to avoid interruptions.

SDK for .NET

Versions 1.x and 2.x

Update to version 2.1.0 of the HDInsight SDK for .NET. Minimal code modifications may be required if you are using a method affected by these changes:

  • ClusterOperationsExtensions.GetClusterConfigurations will no longer return sensitive parameters like storage keys (core-site) or HTTP credentials (gateway).

    • To retrieve all configurations, including sensitive parameters, use ClusterOperationsExtensions.ListConfigurations going forward. Users with the 'Reader' role are not able to use this method. It allows for granular control over which users can access sensitive information for a cluster.
    • To retrieve just HTTP gateway credentials, use ClusterOperationsExtensions.GetGatewaySettings.

  • ClusterOperationsExtensions.GetConnectivitySettings is now deprecated and has been replaced by ClusterOperationsExtensions.GetGatewaySettings.

  • ClusterOperationsExtensions.ConfigureHttpSettings is now deprecated and has been replaced by ClusterOperationsExtensions.UpdateGatewaySettings.

  • ConfigurationsOperationsExtensions.EnableHttp and DisableHttp are now deprecated. HTTP is now always enabled, so these methods are no longer needed.

Versions 3.x and up

Update to version 5.0.0 or later of the HDInsight SDK for .NET. Minimal code modifications may be required if you are using a method affected by these changes:

SDK for Python

Update to version 1.0.0 or later of the HDInsight SDK for Python. Minimal code modifications may be required if you are using a method affected by these changes:

SDK For Java

Update to version 1.0.0 or later of the HDInsight SDK for Java. Minimal code modifications may be required if you are using a method affected by these changes:

  • ConfigurationsInner.get will no longer return sensitive parameters like storage keys (core-site) or HTTP credentials (gateway).
  • ConfigurationsInner.update is now deprecated.

SDK For Go

Update to version 27.1.0 or later of the HDInsight SDK for Go. Minimal code modifications may be required if you are using a method affected by these changes:

Az.HDInsight PowerShell

Update to Az PowerShell version 2.0.0 or later to avoid interruptions. Minimal code modifications may be required if you are using a method affected by these changes.

  • Grant-AzHDInsightHttpServicesAccess is now deprecated and has been replaced by the new Set-AzHDInsightGatewayCredential cmdlet.
  • Get-AzHDInsightJobOutput has been updated to support granular role-based access to the storage key.
    • Users with HDInsight Cluster Operator, Contributor, or Owner roles are not affected.
    • Users with only the Reader role need to specify the DefaultStorageAccountKey parameter explicitly.
  • Revoke-AzHDInsightHttpServicesAccess is now deprecated. HTTP is now always enabled, so this cmdlet is no longer needed. See the az.HDInsight migration guide for more details.

Add the HDInsight Cluster Operator role assignment to a user

A user with the Owner role can assign the HDInsight Cluster Operator role to users that you would want to have read/write access to sensitive HDInsight cluster configuration values (such as cluster gateway credentials and storage account keys).

Using the Azure CLI

The simplest way to add this role assignment is by using the az role assignment create command in Azure CLI.

Note

This command must be run by a user with the Owner role, as only they can grant these permissions. The --assignee is the name of the service principal or email address of the user to whom you want to assign the HDInsight Cluster Operator role. If you receive an insufficient permissions error, see the FAQ.

Grant role at the resource (cluster) level

az role assignment create --role "HDInsight Cluster Operator" --assignee <user@domain.com> --scope /subscriptions/<SubscriptionId>/resourceGroups/<ResourceGroupName>/providers/Microsoft.HDInsight/clusters/<ClusterName>

Grant role at the resource group level

az role assignment create --role "HDInsight Cluster Operator" --assignee user@domain.com -g <ResourceGroupName>

Grant role at the subscription level

az role assignment create --role "HDInsight Cluster Operator" --assignee user@domain.com

Using the Azure portal

You can alternatively use the Azure portal to add the HDInsight Cluster Operator role assignment to a user. See the documentation, Assign Azure roles using the Azure portal.

FAQ

Why am I seeing a 403 (Forbidden) response after updating my API requests and/or tool?

Cluster configurations are now behind granular role-based access control and require the Microsoft.HDInsight/clusters/configurations/* permission to access them. To obtain this permission, assign the HDInsight Cluster Operator, Contributor, or Owner role to the user or service principal trying to access configurations.

Why do I see "Insufficient privileges to complete the operation" when running the Azure CLI command to assign the HDInsight Cluster Operator role to another user or service principal?

In addition to having the Owner role, the user or service principal executing the command needs to have sufficient Microsoft Entra permissions to look up the object IDs of the assignee. This message indicates insufficient Microsoft Entra permissions. Try replacing the --assignee argument with -assignee-object-id and provide the object ID of the assignee as the parameter instead of the name (or the principal ID in the case of a managed identity). See the optional parameters section of the az role assignment create documentation for more info.

If it still does not work, contact your Microsoft Entra admin to acquire the correct permissions.

What will happen if I take no action?

Beginning on September 3, 2019, GET /configurations and POST /configurations/gateway calls will no longer return any information and the GET /configurations/{configurationName} call will no longer return sensitive parameters, such as storage account keys or the cluster password. The same is true of corresponding SDK methods and PowerShell cmdlets.

If you are using an older version of one of the tools for Visual Studio, VSCode, IntelliJ or Eclipse mentioned, it is no longer function until you update.

For more detailed information, see the corresponding section of this document for your scenario.