Deploy machine learning models to Azure

  • Article
  • 04/07/2023

APPLIES TO: Azure CLI ml extension v1 Python SDK azureml v1

Learn how to deploy your machine learning or deep learning model as a web service in the Azure cloud.

Note

Azure Machine Learning Endpoints (v2) provide an improved, simpler deployment experience. Endpoints support both real-time and batch inference scenarios. Endpoints provide a unified interface to invoke and manage model deployments across compute types. See What are Azure Machine Learning endpoints?.

Workflow for deploying a model

The workflow is similar no matter where you deploy your model:

  1. Register the model.
  2. Prepare an entry script.
  3. Prepare an inference configuration.
  4. Deploy the model locally to ensure everything works.
  5. Choose a compute target.
  6. Deploy the model to the cloud.
  7. Test the resulting web service.

For more information on the concepts involved in the machine learning deployment workflow, see Manage, deploy, and monitor models with Azure Machine Learning.

Prerequisites

Connect to your workspace

APPLIES TO: Python SDK azureml v1

from azureml.core import Workspace
ws = Workspace(subscription_id="<subscription_id>",
               resource_group="<resource_group>",
               workspace_name="<workspace_name>")

For more information on using the SDK to connect to a workspace, see the Azure Machine Learning SDK for Python documentation.

Register the model

A typical situation for a deployed machine learning service is that you need the following components:

  • Resources representing the specific model that you want deployed (for example: a pytorch model file).
  • Code that you will be running in the service, that executes the model on a given input.

Azure Machine Learnings allows you to separate the deployment into two separate components, so that you can keep the same code, but merely update the model. We define the mechanism by which you upload a model separately from your code as "registering the model".

When you register a model, we upload the model to the cloud (in your workspace's default storage account) and then mount it to the same compute where your webservice is running.

The following examples demonstrate how to register a model.

Important

You should use only models that you create or obtain from a trusted source. You should treat serialized models as code, because security vulnerabilities have been discovered in a number of popular formats. Also, models might be intentionally trained with malicious intent to provide biased or inaccurate output.

Register a model from a local file

You can register a model by providing the local path of the model. You can provide the path of either a folder or a single file on your local machine.

import urllib.request
from azureml.core.model import Model

# Download model
urllib.request.urlretrieve("https://aka.ms/bidaf-9-model", "model.onnx")

# Register model
model = Model.register(ws, model_name="bidaf_onnx", model_path="./model.onnx")

To include multiple files in the model registration, set model_path to the path of a folder that contains the files.

For more information, see the documentation for the Model class.

Register a model from an Azure ML training job

When you use the SDK to train a model, you can receive either a Run object or an AutoMLRun object, depending on how you trained the model. Each object can be used to register a model created by an experiment run.

  • Register a model from an azureml.core.Run object:

    APPLIES TO: Python SDK azureml v1

    model = run.register_model(model_name='bidaf_onnx',
                           tags={'area': 'qna'},
                           model_path='outputs/model.onnx')
print(model.name, model.id, model.version, sep='\t')

    The model_path parameter refers to the cloud location of the model. In this example, the path of a single file is used. To include multiple files in the model registration, set model_path to the path of a folder that contains the files. For more information, see the Run.register_model documentation.

  • Register a model from an azureml.train.automl.run.AutoMLRun object:

    APPLIES TO: Python SDK azureml v1

    description = 'My AutoML Model'
model = run.register_model(description = description,
                            tags={'area': 'qna'})

print(run.model_id)

    In this example, the metric and iteration parameters aren't specified, so the iteration with the best primary metric will be registered. The model_id value returned from the run is used instead of a model name.

    For more information, see the AutoMLRun.register_model documentation.

    To deploy a registered model from an AutoMLRun, we recommend doing so via the one-click deploy button in Azure Machine Learning studio.

Define a dummy entry script

The entry script receives data submitted to a deployed web service and passes it to the model. It then returns the model's response to the client. The script is specific to your model. The entry script must understand the data that the model expects and returns.

The two things you need to accomplish in your entry script are:

  1. Loading your model (using a function called init())
  2. Running your model on input data (using a function called run())

For your initial deployment, use a dummy entry script that prints the data it receives.

import json


def init():
    print("This is init")


def run(data):
    test = json.loads(data)
    print(f"received data {test}")
    return f"test is {test}"

Save this file as echo_score.py inside of a directory called source_dir. This dummy script returns the data you send to it, so it doesn't use the model. But it is useful for testing that the scoring script is running.

Define an inference configuration

An inference configuration describes the Docker container and files to use when initializing your web service. All of the files within your source directory, including subdirectories, will be zipped up and uploaded to the cloud when you deploy your web service.

The inference configuration below specifies that the machine learning deployment will use the file echo_score.py in the ./source_dir directory to process incoming requests and that it will use the Docker image with the Python packages specified in the project_environment environment.

You can use any Azure Machine Learning inference curated environments as the base Docker image when creating your project environment. We will install the required dependencies on top and store the resulting Docker image into the repository that is associated with your workspace.

Note

Azure machine learning inference source directory upload does not respect .gitignore or .amlignore

The following example demonstrates how to create a minimal environment with no pip dependencies, using the dummy scoring script you defined above.

from azureml.core import Environment
from azureml.core.model import InferenceConfig

env = Environment(name="project_environment")
dummy_inference_config = InferenceConfig(
    environment=env,
    source_directory="./source_dir",
    entry_script="./echo_score.py",
)

For more information on environments, see Create and manage environments for training and deployment.

For more information on inference configuration, see the InferenceConfig class documentation.

Define a deployment configuration

A deployment configuration specifies the amount of memory and cores your webservice needs in order to run. It also provides configuration details of the underlying webservice. For example, a deployment configuration lets you specify that your service needs 2 gigabytes of memory, 2 CPU cores, 1 GPU core, and that you want to enable autoscaling.

The options available for a deployment configuration differ depending on the compute target you choose. In a local deployment, all you can specify is which port your webservice will be served on.

The following Python demonstrates how to create a local deployment configuration:

from azureml.core.webservice import LocalWebservice

deployment_config = LocalWebservice.deploy_configuration(port=6789)

Deploy your machine learning model

You are now ready to deploy your model.

service = Model.deploy(
    ws,
    "myservice",
    [model],
    dummy_inference_config,
    deployment_config,
    overwrite=True,
)
service.wait_for_deployment(show_output=True)

print(service.get_logs())

For more information, see the documentation for Model.deploy() and Webservice.

Call into your model

Let's check that your echo model deployed successfully. You should be able to do a simple liveness request, as well as a scoring request:

import requests
import json

uri = service.scoring_uri
requests.get("http://localhost:6789")
headers = {"Content-Type": "application/json"}
data = {
    "query": "What color is the fox",
    "context": "The quick brown fox jumped over the lazy dog.",
}
data = json.dumps(data)
response = requests.post(uri, data=data, headers=headers)
print(response.json())

Define an entry script

Now it's time to actually load your model. First, modify your entry script:

import json
import numpy as np
import os
import onnxruntime
from nltk import word_tokenize
import nltk


def init():
    nltk.download("punkt")
    global sess
    sess = onnxruntime.InferenceSession(
        os.path.join(os.getenv("AZUREML_MODEL_DIR"), "model.onnx")
    )


def run(request):
    print(request)
    text = json.loads(request)
    qw, qc = preprocess(text["query"])
    cw, cc = preprocess(text["context"])

    # Run inference
    test = sess.run(
        None,
        {"query_word": qw, "query_char": qc, "context_word": cw, "context_char": cc},
    )
    start = np.asscalar(test[0])
    end = np.asscalar(test[1])
    ans = [w for w in cw[start : end + 1].reshape(-1)]
    print(ans)
    return ans


def preprocess(word):
    tokens = word_tokenize(word)

    # split into lower-case word tokens, in numpy array with shape of (seq, 1)
    words = np.asarray([w.lower() for w in tokens]).reshape(-1, 1)

    # split words into chars, in numpy array with shape of (seq, 1, 1, 16)
    chars = [[c for c in t][:16] for t in tokens]
    chars = [cs + [""] * (16 - len(cs)) for cs in chars]
    chars = np.asarray(chars).reshape(-1, 1, 1, 16)
    return words, chars

Save this file as score.py inside of source_dir.

Notice the use of the AZUREML_MODEL_DIR environment variable to locate your registered model. Now that you've added some pip packages.

APPLIES TO: Python SDK azureml v1

env = Environment(name='myenv')
python_packages = ['nltk', 'numpy', 'onnxruntime']
for package in python_packages:
    env.python.conda_dependencies.add_pip_package(package)

inference_config = InferenceConfig(environment=env, source_directory='./source_dir', entry_script='./score.py')

For more information, see the documentation for LocalWebservice, Model.deploy(), and Webservice.

Deploy again and call your service

Deploy your service again:

service = Model.deploy(
    ws,
    "myservice",
    [model],
    inference_config,
    deployment_config,
    overwrite=True,
)
service.wait_for_deployment(show_output=True)

print(service.get_logs())

For more information, see the documentation for Model.deploy() and Webservice.

Then ensure you can send a post request to the service:

import requests

uri = service.scoring_uri

headers = {"Content-Type": "application/json"}
data = {
    "query": "What color is the fox",
    "context": "The quick brown fox jumped over the lazy dog.",
}
data = json.dumps(data)
response = requests.post(uri, data=data, headers=headers)
print(response.json())

Choose a compute target

The compute target you use to host your model will affect the cost and availability of your deployed endpoint. Use this table to choose an appropriate compute target.

Compute target Used for GPU support Description
Local web service Testing/debugging   Use for limited testing and troubleshooting. Hardware acceleration depends on use of libraries in the local system.
Azure Machine Learning endpoints (SDK/CLI v2 only) Real-time inference

Batch inference		 Yes Fully managed computes for real-time (managed online endpoints) and batch scoring (batch endpoints) on serverless compute.
Azure Machine Learning Kubernetes Real-time inference

Batch inference		 Yes Run inferencing workloads on on-premises, cloud, and edge Kubernetes clusters.
Azure Container Instances (SDK/CLI v1 only) Real-time inference

Recommended for dev/test purposes only.		   Use for low-scale CPU-based workloads that require less than 48 GB of RAM. Doesn't require you to manage a cluster.

Supported in the designer.

Note

When choosing a cluster SKU, first scale up and then scale out. Start with a machine that has 150% of the RAM your model requires, profile the result and find a machine that has the performance you need. Once you've learned that, increase the number of machines to fit your need for concurrent inference.

Note

Container instances require the SDK or CLI v1 and are suitable only for small models less than 1 GB in size.

Deploy to cloud

Once you've confirmed your service works locally and chosen a remote compute target, you are ready to deploy to the cloud.

Change your deploy configuration to correspond to the compute target you've chosen, in this case Azure Container Instances:

from azureml.core.webservice import AciWebservice

deployment_config = AciWebservice.deploy_configuration(
    cpu_cores=0.5, memory_gb=1, auth_enabled=True
)

Deploy your service again:

service = Model.deploy(
    ws,
    "myservice",
    [model],
    inference_config,
    deployment_config,
    overwrite=True,
)
service.wait_for_deployment(show_output=True)

print(service.get_logs())

For more information, see the documentation for Model.deploy() and Webservice.

Call your remote webservice

When you deploy remotely, you may have key authentication enabled. The example below shows how to get your service key with Python in order to make an inference request.

import requests
import json
from azureml.core import Webservice

service = Webservice(workspace=ws, name="myservice")
scoring_uri = service.scoring_uri

# If the service is authenticated, set the key or token
key, _ = service.get_keys()

# Set the appropriate headers
headers = {"Content-Type": "application/json"}
headers["Authorization"] = f"Bearer {key}"

# Make the request and display the response and logs
data = {
    "query": "What color is the fox",
    "context": "The quick brown fox jumped over the lazy dog.",
}
data = json.dumps(data)
resp = requests.post(scoring_uri, data=data, headers=headers)
print(resp.text)

print(service.get_logs())

See the article on client applications to consume web services for more example clients in other languages.

How to configure emails in the studio

To start receiving emails when your job, online endpoint, or batch endpoint is complete or if there's an issue (failed, canceled), use the following steps:

  1. In Azure ML studio, go to settings by selecting the gear icon.
  2. Select the Email notifications tab.
  3. Toggle to enable or disable email notifications for a specific event.

Screenshot of Azure ML studio's settings on the email notifications tab.

Understanding service state

During model deployment, you may see the service state change while it fully deploys.

The following table describes the different service states:

Webservice state Description Final state?
Transitioning The service is in the process of deployment. No
Unhealthy The service has deployed but is currently unreachable. No
Unschedulable The service cannot be deployed at this time due to lack of resources. No
Failed The service has failed to deploy due to an error or crash. Yes
Healthy The service is healthy and the endpoint is available. Yes

Tip

When deploying, Docker images for compute targets are built and loaded from Azure Container Registry (ACR). By default, Azure Machine Learning creates an ACR that uses the basic service tier. Changing the ACR for your workspace to standard or premium tier may reduce the time it takes to build and deploy images to your compute targets. For more information, see Azure Container Registry service tiers.

Note

If you are deploying a model to Azure Kubernetes Service (AKS), we advise you enable Azure Monitor for that cluster. This will help you understand overall cluster health and resource usage.

If you are trying to deploy a model to an unhealthy or overloaded cluster, it is expected to experience issues. If you need help troubleshooting AKS cluster problems please contact AKS Support.

Delete resources

service.delete()
model.delete()

To delete a deployed web service, use service.delete(). To delete a registered model, use model.delete().

For more information, see the documentation for WebService.delete() and Model.delete().

Next steps