Use authentication credential secrets in Azure Machine Learning training jobs

  • Article

APPLIES TO: Python SDK azureml v1

In this article, you learn how to use secrets in training jobs securely. Authentication information such as your user name and password are secrets. For example, if you connect to an external database in order to query training data, you would need to pass your username and password to the remote job context. Coding such values into training scripts in cleartext is insecure as it would expose the secret.

Instead, your Azure Machine Learning workspace has an associated resource called a Azure Key Vault. Use this Key Vault to pass secrets to remote jobs securely through a set of APIs in the Azure Machine Learning Python SDK.

The standard flow for using secrets is:

  1. On local computer, sign in to Azure and connect to your workspace.
  2. On local computer, set a secret in Workspace Key Vault.
  3. Submit a remote job.
  4. Within the remote job, get the secret from Key Vault and use it.

Set secrets

In the Azure Machine Learning, the Keyvault class contains methods for setting secrets. In your local Python session, first obtain a reference to your workspace Key Vault, and then use the set_secret() method to set a secret by name and value. The set_secret method updates the secret value if the name already exists.

from azureml.core import Workspace
from azureml.core import Keyvault
import os


ws = Workspace.from_config()
my_secret = os.environ.get("MY_SECRET")
keyvault = ws.get_default_keyvault()
keyvault.set_secret(name="mysecret", value = my_secret)

Don't put the secret value in your Python code as it is insecure to store it in file as cleartext. Instead, obtain the secret value from an environment variable, for example Azure DevOps build secret, or from interactive user input.

You can list secret names using the list_secrets() method and there's also a batch version,set_secrets() that allows you to set multiple secrets at a time.

Important

Using list_secrets() will only list secrets created through set_secret() or set_secrets() using the Azure Machine Learning SDK. It will not list secrets created by something other than the SDK. For example, a secret created using the Azure portal or Azure PowerShell will not be listed.

You can use get_secret() to get a secret value from the key vault, regardless of how it was created. So you can retrieve secrets that are not listed by list_secrets().

Get secrets

In your local code, you can use the get_secret() method to get the secret value by name.

For jobs submitted the Experiment.submit , use the get_secret() method with the Run class. Because a submitted run is aware of its workspace, this method shortcuts the Workspace instantiation and returns the secret value directly.

# Code in submitted job
from azureml.core import Experiment, Run

run = Run.get_context()
secret_value = run.get_secret(name="mysecret")

Be careful not to expose the secret value by writing or printing it out.

There's also a batch version, get_secrets() for accessing multiple secrets at once.

Related content