Set up authentication between Azure Machine Learning and other services

APPLIES TO: Azure CLI ml extension v2 (current) Python SDK azure-ai-ml v2 (current)

Azure Machine Learning is composed of multiple Azure services. There are multiple ways that authentication can happen between Azure Machine Learning and the services it relies on.

• The Azure Machine Learning workspace uses a managed identity to communicate with other services. By default, this is a system-assigned managed identity. You can also use a user-assigned managed identity instead.
• Azure Machine Learning uses Azure Container Registry (ACR) to store Docker images used to train and deploy models. If you allow Azure Machine Learning to automatically create ACR, it will enable the admin account.
• The Azure Machine Learning compute cluster uses a managed identity to retrieve connection information for datastores from Azure Key Vault and to pull Docker images from ACR. You can also configure identity-based access to datastores, which will instead use the managed identity of the compute cluster.
• Data access can happen along multiple paths depending on the data storage service and your configuration. For example, authentication to the datastore may use an account key, token, security principal, managed identity, or user identity.
• Managed online endpoints can use a managed identity to access Azure resources when performing inference. For more information, see Access Azure resources from an online endpoint.

Prerequisites

Before following the steps in this article, make sure you have the following prerequisites:

• An Azure Machine Learning workspace. If you don't have one, use the steps in the Quickstart: Create workspace resources article to create one.

• The Azure CLI and the ml extension or the Azure Machine Learning Python SDK v2:

  • To install the Azure CLI and extension, see Install, set up, and use the CLI (v2).

    Important

    The CLI examples in this article assume that you are using the Bash (or compatible) shell. For example, from a Linux system or Windows Subsystem for Linux.

  • To install the Python SDK v2, use the following command:

    pip install azure-ai-ml azure-identity
    

    To update an existing installation of the SDK to the latest version, use the following command:

    pip install --upgrade azure-ai-ml azure-identity
    

    For more information, see Install the Python SDK v2 for Azure Machine Learning.

• To assign roles, the login for your Azure subscription must have the Managed Identity Operator role, or other role that grants the required actions (such as Owner).

• You must be familiar with creating and working with Managed Identities.

Workspace identity types

The Azure Machine Learning workspace uses a managed identity to communicate with other services. Multiple identity types are supported for Azure Machine Learning.

Once a workspace is created with SAI identity type, it can be updated to SAI+UAI, but not back from SAI+UAI to SAI. You may assign multiple user-assigned identities to the same workspace.

User-assigned managed identity

Workspace

You can add a user-assigned managed identity when creating an Azure Machine Learning workspace from the Azure portal. Use the following steps while creating the workspace:

1. From the Basics page, select the Azure Storage Account, Azure Container Registry, and Azure Key Vault you want to use with the workspace.
2. From the Identity page, select User-assigned identity and then select the managed identity to use.

The following Azure RBAC role assignments are required on your user-assigned managed identity for your Azure Machine Learning workspace to access data on the workspace-associated resources.

For automated creation of role assignments on your user-assigned managed identity, you may use this ARM template.

To create a workspace with multiple user assigned identities, use one of the following methods:

az ml workspace create -f workspace_creation_with_multiple_UAIs.yml --subscription <subscription ID> --resource-group <resource group name> --name <workspace name>

location: <region name>
identity:
   type: user_assigned
   user_assigned_identities:
    '<UAI resource ID 1>': {}
    '<UAI resource ID 2>': {}
storage_account: <storage acccount resource ID>
key_vault: <key vault resource ID>
image_build_compute: <compute(virtual machine) resource ID>
primary_user_assigned_identity: <one of the UAI resource IDs in the above list>

from azure.ai.ml import MLClient, load_workspace
from azure.identity import DefaultAzureCredential

sub_id="<subscription ID>"
rg_name="<resource group name>"
ws_name="<workspace name>"

client = MLClient(DefaultAzureCredential(), sub_id, rg_name)
wps = load_workspace("workspace_creation_with_multiple_UAIs.yml")

workspace = client.workspaces.begin_create(workspace=wps).result()

To update user assigned identities for a workspace, includes adding a new one or deleting the existing ones, use one of the following methods:

az ml workspace update -f workspace_update_with_multiple_UAIs.yml --subscription <subscription ID> --resource-group <resource group name> --name <workspace name>

identity:
   type: user_assigned
   user_assigned_identities:
    '<UAI resource ID 1>': {}
    '<UAI resource ID 2>': {}
primary_user_assigned_identity: <one of the UAI resource IDs in the above list>

from azure.ai.ml import MLClient, load_workspace
from azure.identity import DefaultAzureCredential

sub_id="<subscription ID>"
rg_name="<resource group name>"
ws_name="<workspace name>"

client = MLClient(DefaultAzureCredential(), sub_id, rg_name)
wps = load_workspace("workspace_update_with_multiple_UAIs.yml")

workspace = client.workspaces.begin_update(workspace=wps).result()

Add a user-assigned managed identity to a workspace in addition to a system-assigned identity

In some scenarios, you may need to use a user-assigned managed identity in addition to the default system-assigned workspace identity. To add a user-assigned managed identity, without changing the existing workspace identity, use the following steps:

1. Create a user-assigned managed identity. Save the ID for the managed identity that you create.

2. To attach the managed identity to your workspace, you need a YAML file that specifies the identity. The following is an example of the YAML file contents. Replace the <TENANT_ID>, <RESOURCE_GROUP>, and <USER_MANAGED_ID> with your values.

   identity:
       type: system_assigned,user_assigned
       tenant_id: <TENANT_ID>
       user_assigned_identities:
           '/subscriptions/<SUBSCRIPTION_ID/resourceGroups/<RESOURCE_GROUP>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<USER_MANAGED_ID>':
           {}
   

3. Use the Azure CLI az ml workspace update command to update your workspace. Specify the YAML file from the previous step using the --file parameter. The following example shows what this command looks like:

   az ml workspace update --resource-group <RESOURCE_GROUP> --name <WORKSPACE_NAME> --file <YAML_FILE_NAME>.yaml
   

Compute cluster

The default managed identity is the system-assigned managed identity or the first user-assigned managed identity.

During a run there are two applications of an identity:

1. The system uses an identity to set up the user's storage mounts, container registry, and datastores.

   • In this case, the system will use the default-managed identity.

2. You apply an identity to access resources from within the code for a submitted job:

   • In this case, provide the client_id corresponding to the managed identity you want to use to retrieve a credential.
   • Alternatively, get the user-assigned identity's client ID through the DEFAULT_IDENTITY_CLIENT_ID environment variable.

   For example, to retrieve a token for a datastore with the default-managed identity:

   client_id = os.environ.get('DEFAULT_IDENTITY_CLIENT_ID')
   credential = ManagedIdentityCredential(client_id=client_id)
   token = credential.get_token('https://storage.azure.com/')
   

To configure a compute cluster with managed identity, use one of the following methods:

az ml compute create -f create-cluster.yml

$schema: https://azuremlschemas.azureedge.net/latest/amlCompute.schema.json 
name: basic-example
type: amlcompute
size: STANDARD_DS3_v2
min_instances: 0
max_instances: 2
idle_time_before_scale_down: 120
identity:
  type: user_assigned
  user_assigned_identities: 
    - resource_id: "identity_resource_id"

$schema: https://azuremlschemas.azureedge.net/latest/amlCompute.schema.json 
name: basic-example
type: amlcompute
size: STANDARD_DS3_v2
min_instances: 0
max_instances: 2
idle_time_before_scale_down: 120
identity:
  type: system_assigned

export MSI_NAME=my-cluster-identity
export COMPUTE_NAME=mycluster-msi

does_compute_exist()
{
  if [ -z $(az ml compute show -n $COMPUTE_NAME --query name) ]; then
    echo false
  else
    echo true
  fi
}

echo "Creating MSI $MSI_NAME"
# Get the resource id of the identity
IDENTITY_ID=$(az identity show --name "$MSI_NAME" --query id -o tsv | tail -n1 | tr -d "[:cntrl:]" || true)
if [[ -z $IDENTITY_ID ]]; then
    IDENTITY_ID=$(az identity create -n "$MSI_NAME" --query id -o tsv | tail -n1 | tr -d "[:cntrl:]")
fi
echo "MSI created: $MSI_NAME"
sleep 15 # Let the previous command finish: https://github.com/Azure/azure-cli/issues/8530


echo "Checking if compute $COMPUTE_NAME already exists"
if [ "$(does_compute_exist)" == "true" ]; then
  echo "Skipping, compute: $COMPUTE_NAME exists"
else
  echo "Provisioning compute: $COMPUTE_NAME"
  az ml compute create --name "$COMPUTE_NAME" --type amlcompute --identity-type user_assigned --user-assigned-identities "$IDENTITY_ID"
fi
az ml compute update --name "$COMPUTE_NAME" --identity-type user_assigned --user-assigned-identities "$IDENTITY_ID"

export COMPUTE_NAME=mycluster-sa

does_compute_exist()
{
  if [ -z $(az ml compute show -n $COMPUTE_NAME --query name) ]; then
    echo false
  else
    echo true
  fi
}

echo "Checking if compute $COMPUTE_NAME already exists"
if [ "$(does_compute_exist)" == "true" ]; then
  echo "Skipping, compute: $COMPUTE_NAME exists"
else
  echo "Provisioning compute: $COMPUTE_NAME"
  az ml compute create --name "$COMPUTE_NAME" --type amlcompute
fi

az ml compute update --name "$COMPUTE_NAME" --identity-type system_assigned

from azure.ai.ml.entities import ManagedIdentityConfiguration, IdentityConfiguration, AmlCompute
from azure.ai.ml.constants import ManagedServiceIdentityType

# Create an identity configuration from the user-assigned managed identity
managed_identity = ManagedIdentityConfiguration(resource_id="/subscriptions/<subscription_id>/resourcegroups/<resource_group>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<identity>")
identity_config = IdentityConfiguration(type = ManagedServiceIdentityType.USER_ASSIGNED, user_assigned_identities=[managed_identity])

# specify aml compute name.
cpu_compute_target = "cpu-cluster"

try:
    ml_client.compute.get(cpu_compute_target)
except Exception:
    print("Creating a new cpu compute target...")
    # Pass the identity configuration
    compute = AmlCompute(
        name=cpu_compute_target, size="STANDARD_D2_V2", min_instances=0, max_instances=4, identity=identity_config
    )
    ml_client.compute.begin_create_or_update(compute)

Data storage

When you create a datastore that uses identity-based data access, your Azure account (Microsoft Entra token) is used to confirm you have permission to access the storage service. In the identity-based data access scenario, no authentication credentials are saved. Only the storage account information is stored in the datastore.

In contrast, datastores that use credential-based authentication cache connection information, like your storage account key or SAS token, in the key vault that's associated with the workspace. This approach has the limitation that other workspace users with sufficient permissions can retrieve those credentials, which may be a security concern for some organization.

For more information on how data access is authenticated, see the Data administration article. For information on configuring identity based access to data, see Create datastores.

There are two scenarios in which you can apply identity-based data access in Azure Machine Learning. These scenarios are a good fit for identity-based access when you're working with confidential data and need more granular data access management:

• Accessing storage services
• Training machine learning models

The identity-based access allows you to use role-based access controls (RBAC) to restrict which identities, such as users or compute resources, have access to the data.

Accessing storage services

You can connect to storage services via identity-based data access with Azure Machine Learning datastores.

When you use identity-based data access, Azure Machine Learning prompts you for your Microsoft Entra token for data access authentication instead of keeping your credentials in the datastore. That approach allows for data access management at the storage level and keeps credentials confidential.

The same behavior applies when you work with data interactively via a Jupyter Notebook on your local computer or compute instance.

To help ensure that you securely connect to your storage service on Azure, Azure Machine Learning requires that you have permission to access the corresponding data storage.

Identity-based data access supports connections to only the following storage services.

• Azure Blob Storage
• Azure Data Lake Storage Gen1
• Azure Data Lake Storage Gen2

To access these storage services, you must have at least Storage Blob Data Reader access to the storage account. Only storage account owners can change your access level via the Azure portal.

Access data for training jobs on compute using managed identity

Certain machine learning scenarios involve working with private data. In such cases, data scientists may not have direct access to data as Microsoft Entra users. In this scenario, the managed identity of a compute can be used for data access authentication. In this scenario, the data can only be accessed from a compute instance or a machine learning compute cluster executing a training job. With this approach, the admin grants the compute instance or compute cluster managed identity Storage Blob Data Reader permissions on the storage. The individual data scientists don't need to be granted access.

To enable authentication with compute managed identity:

• Create compute with managed identity enabled. See the compute cluster section, or for compute instance, the Assign managed identity section.

  Important

  If the compute instance is also configured for idle shutdown, the compute instance won't shut down due to inactivity unless the managed identity has contributor access to the Azure Machine Learning workspace. For more information on assigning permissions, see Manage access to Azure Machine Learning workspaces.

• Grant compute managed identity at least Storage Blob Data Reader role on the storage account.

• Create any datastores with identity-based authentication enabled. See Create datastores.

Once the identity-based authentication is enabled, the compute managed identity is used by default when accessing data within your training jobs. Optionally, you can authenticate with user identity using the steps described in next section.

For information on using configuring Azure RBAC for the storage, see role-based access controls.

Access data for training jobs on compute clusters using user identity

APPLIES TO: Azure CLI ml extension v2 (current)

When training on Azure Machine Learning compute clusters, you can authenticate to storage with your user Microsoft Entra token.

This authentication mode allows you to:

• Set up fine-grained permissions, where different workspace users can have access to different storage accounts or folders within storage accounts.
• Let data scientists re-use existing permissions on storage systems.
• Audit storage access because the storage logs show which identities were used to access data.

The following steps outline how to set up data access with user identity for training jobs on compute clusters from CLI.

1. Grant the user identity access to storage resources. For example, grant StorageBlobReader access to the specific storage account you want to use or grant ACL-based permission to specific folders or files in Azure Data Lake Gen 2 storage.

2. Create an Azure Machine Learning datastore without cached credentials for the storage account. If a datastore has cached credentials, such as storage account key, those credentials are used instead of user identity.

3. Submit a training job with property identity set to type: user_identity, as shown in following job specification. During the training job, the authentication to storage happens via the identity of the user that submits the job.

   Note

   If the identity property is left unspecified and datastore does not have cached credentials, then compute managed identity becomes the fallback option.

   command: |
   echo "--census-csv: ${{inputs.census_csv}}"
   python hello-census.py --census-csv ${{inputs.census_csv}}
   code: src
   inputs:
   census_csv:
       type: uri_file 
       path: azureml://datastores/mydata/paths/census.csv
   environment: azureml:AzureML-sklearn-1.0-ubuntu20.04-py38-cpu@latest
   compute: azureml:cpu-cluster
   identity:
   type: user_identity
   

The following steps outline how to set up data access with user identity for training jobs on compute clusters from Python SDK.

1. Grant data access and create data store as described above for CLI.

2. Submit a training job with identity parameter set to azure.ai.ml.UserIdentityConfiguration. This parameter setting enables the job to access data on behalf of user submitting the job.

   from azure.ai.ml import command
   from azure.ai.ml.entities import Data, UriReference
   from azure.ai.ml import Input
   from azure.ai.ml.constants import AssetTypes
   from azure.ai.ml import UserIdentityConfiguration
   
   # Specify the data location
   my_job_inputs = {
       "input_data": Input(type=AssetTypes.URI_FILE, path="<path-to-my-data>")
   }
   
   # Define the job
   job = command(
       code="<my-local-code-location>", 
       command="python <my-script>.py --input_data ${{inputs.input_data}}",
       inputs=my_job_inputs,
       environment="AzureML-sklearn-0.24-ubuntu18.04-py37-cpu:9",
       compute="<my-compute-cluster-name>",
       identity= UserIdentityConfiguration() 
   )
   # submit the command
   returned_job = ml_client.jobs.create_or_update(job)
   

Work with virtual networks

By default, Azure Machine Learning can't communicate with a storage account that's behind a firewall or in a virtual network.

You can configure storage accounts to allow access only from within specific virtual networks. This configuration requires extra steps to ensure data isn't leaked outside of the network. This behavior is the same for credential-based data access. For more information, see How to prevent data exfiltration.

If your storage account has virtual network settings, that dictates what identity type and permissions access is needed. For example for data preview and data profile, the virtual network settings determine what type of identity is used to authenticate data access.

• In scenarios where only certain IPs and subnets are allowed to access the storage, then Azure Machine Learning uses the workspace MSI to accomplish data previews and profiles.

• If your storage is ADLS Gen 2 or Blob and has virtual network settings, customers can use either user identity or workspace MSI depending on the datastore settings defined during creation.

• If the virtual network setting is "Allow Azure services on the trusted services list to access this storage account", then Workspace MSI is used.

Scenario: Azure Container Registry without admin user

When you disable the admin user for ACR, Azure Machine Learning uses a managed identity to build and pull Docker images. There are two workflows when configuring Azure Machine Learning to use an ACR with the admin user disabled:

• Allow Azure Machine Learning to create the ACR instance and then disable the admin user afterwards.
• Bring an existing ACR with the admin user already disabled.

Azure Machine Learning with auto-created ACR instance

1. Create a new Azure Machine Learning workspace.

2. Perform an action that requires Azure Container Registry. For example, the Tutorial: Train your first model.

3. Get the name of the ACR created by the cluster.

   APPLIES TO: Azure CLI ml extension v2 (current)

   az ml workspace show --name <my workspace name> \
   --resource-group <my resource group> \
   --subscription <my subscription id> \
   --query container_registry
   

   This command returns a value similar to the following text. You only want the last portion of the text, which is the ACR instance name:

   /subscriptions/<subscription id>/resourceGroups/<my resource group>/providers/MicrosoftContainerReggistry/registries/<ACR instance name>
   

4. Update the ACR to disable the admin user:

   az acr update --name <ACR instance name> --admin-enabled false
   

Bring your own ACR

If ACR admin user is disallowed by subscription policy, you should first create ACR without admin user, and then associate it with the workspace. Create ACR from Azure CLI without setting --admin-enabled argument, or from Azure portal without enabling admin user. Then, when creating Azure Machine Learning workspace, specify the Azure resource ID of the ACR. The following example demonstrates creating a new Azure Machine Learning workspace that uses an existing ACR:

APPLIES TO: Azure CLI ml extension v2 (current)

az ml workspace create -n <workspace name> \
-g <workspace resource group> \
-l <region> \
--container-registry /subscriptions/<subscription id>/resourceGroups/<acr resource group>/providers/Microsoft.ContainerRegistry/registries/<acr name>

Also, if you already have an existing ACR with admin user disabled, you can attach it to the workspace by updating it. The following example demonstrates updating an Azure Machine Learning workspace to use an existing ACR:

APPLIES TO: Azure CLI ml extension v2 (current)

az ml workspace update --update-dependent-resources \
--name <workspace name> \
--resource-group <workspace resource group> \
--container-registry /subscriptions/<subscription id>/resourceGroups/<acr resource group>/providers/Microsoft.ContainerRegistry/registries/<acr name>

Create compute with managed identity to access Docker images for training

To access the workspace ACR, create machine learning compute cluster with system-assigned managed identity enabled. You can enable the identity from Azure portal or Studio when creating compute, or from Azure CLI using the below. For more information, see using managed identity with compute clusters.

az ml compute create --name cpu-cluster --type <cluster name>  --identity-type systemassigned

from azure.ai.ml.entities import IdentityConfiguration, AmlCompute
from azure.ai.ml.constants import ManagedServiceIdentityType

# Create an identity configuration for a system-assigned managed identity
identity_config = IdentityConfiguration(type = ManagedServiceIdentityType.SYSTEM_ASSIGNED)

# specify aml compute name.
cpu_compute_target = "cpu-cluster"

try:
    ml_client.compute.get(cpu_compute_target)
except Exception:
    print("Creating a new cpu compute target...")
    # Pass the identity configuration
    compute = AmlCompute(
        name=cpu_compute_target, size="STANDARD_D2_V2", min_instances=0, max_instances=4, identity=identity_config
    )
    ml_client.compute.begin_create_or_update(compute)

A managed identity is automatically granted ACRPull role on workspace ACR to enable pulling Docker images for training.

Use Docker images for inference

Once you've configured ACR without admin user as described earlier, you can access Docker images for inference without admin keys from your Azure Kubernetes service (AKS). When you create or attach AKS to workspace, the cluster's service principal is automatically assigned ACRPull access to workspace ACR.

Scenario: Use a private Azure Container Registry

By default, Azure Machine Learning uses Docker base images that come from a public repository managed by Azure. It then builds your training or inference environment on those images. For more information, see What are ML environments?.

To use a custom base image internal to your enterprise, you can use managed identities to access your private ACR. There are two use cases:

• Use base image for training as is.
• Build Azure Machine Learning managed image with custom image as a base.

Pull Docker base image to machine learning compute cluster for training as is

Create machine learning compute cluster with system-assigned managed identity enabled as described earlier. Then, determine the principal ID of the managed identity.

APPLIES TO: Azure CLI ml extension v2 (current)

az ml compute show --name <cluster name> -n <workspace> -g <resource group>

Optionally, you can update the compute cluster to assign a user-assigned managed identity:

APPLIES TO: Azure CLI ml extension v2 (current)

az ml compute update --name <cluster name> --user-assigned-identities <my-identity-id>

To allow the compute cluster to pull the base images, grant the managed service identity ACRPull role on the private ACR

APPLIES TO: Azure CLI ml extension v2 (current)

az role assignment create --assignee <principal ID> \
--role acrpull \
--scope "/subscriptions/<subscription ID>/resourceGroups/<private ACR resource group>/providers/Microsoft.ContainerRegistry/registries/<private ACR name>"

Finally, create an environment and specify the base image location in the environment YAML file.

APPLIES TO: Azure CLI ml extension v2 (current)

$schema: https://azuremlschemas.azureedge.net/latest/environment.schema.json
name: docker-image-example
image: pytorch/pytorch:latest
description: Environment created from a Docker image.

az ml environment create --file <yaml file>

You can now use the environment in a training job.

Build Azure Machine Learning managed environment into base image from private ACR for training or inference

Next steps

• Learn more about enterprise security in Azure Machine Learning
• Learn about data administration
• Learn about managed identities on compute cluster.