Deploy a flow to online endpoint for real-time inference with CLI

In this article, you'll learn to deploy your flow to a managed online endpoint or a Kubernetes online endpoint for use in real-time inferencing with Azure Machine Learning v2 CLI.

Before beginning make sure that you have tested your flow properly, and feel confident that it's ready to be deployed to production. To learn more about testing your flow, see test your flow. After testing your flow you'll learn how to create managed online endpoint and deployment, and how to use the endpoint for real-time inferencing.

• This article will cover how to use the CLI experience.
• The Python SDK isn't covered in this article. See the GitHub sample notebook instead. To use the Python SDK, you must have The Python SDK v2 for Azure Machine Learning. To learn more, see Install the Python SDK v2 for Azure Machine Learning.

Prerequisites

• The Azure CLI and the Azure Machine Learning extension to the Azure CLI. For more information, see Install, set up, and use the CLI (v2).
• An Azure Machine Learning workspace. If you don't have one, use the steps in the Quickstart: Create workspace resources article to create one.
• Azure role-based access controls (Azure RBAC) are used to grant access to operations in Azure Machine Learning. To perform the steps in this article, your user account must be assigned the owner or contributor role for the Azure Machine Learning workspace, or a custom role allowing "Microsoft.MachineLearningServices/workspaces/onlineEndpoints/". If you use studio to create/manage online endpoints/deployments, you'll need an additional permission "Microsoft.Resources/deployments/write" from the resource group owner. For more information, see Manage access to an Azure Machine Learning workspace.

Virtual machine quota allocation for deployment

For managed online endpoints, Azure Machine Learning reserves 20% of your compute resources for performing upgrades. Therefore, if you request a given number of instances in a deployment, you must have a quota for ceil(1.2 * number of instances requested for deployment) * number of cores for the VM SKU available to avoid getting an error. For example, if you request 10 instances of a Standard_DS3_v2 VM (that comes with four cores) in a deployment, you should have a quota for 48 cores (12 instances four cores) available. To view your usage and request quota increases, see View your usage and quotas in the Azure portal.

Get the flow ready for deploy

Each flow will have a folder which contains codes/prompts, definition and other artifacts of the flow. If you have developed your flow with UI, you can download the flow folder from the flow details page. If you have developed your flow with CLI or SDK, you should have the flow folder already.

This article will use the sample flow "basic-chat" as an example to deploy to Azure Machine Learning managed online endpoint.

Set default workspace

Use the following commands to set the default workspace and resource group for the CLI.

az account set --subscription <subscription ID>
az configure --defaults workspace=<Azure Machine Learning workspace name> group=<resource group>

Register the flow as a model (optional)

In the online deployment, you can either refer to a registered model, or specify the model path (where to upload the model files from) inline. It's recommended to register the model and specify the model name and version in the deployment definition. Use the form model:<model_name>:<version>.

Following is a model definition example for a chat flow.

$schema: https://azuremlschemas.azureedge.net/latest/model.schema.json
name: basic-chat-model
path: ../../../../examples/flows/chat/basic-chat
description: register basic chat flow folder as a custom model
properties:
  # In AuzreML studio UI, endpoint detail UI Test tab needs this property to know it's from prompt flow
  azureml.promptflow.source_flow_id: basic-chat
  
  # Following are properties only for chat flow 
  # endpoint detail UI Test tab needs this property to know it's a chat flow
  azureml.promptflow.mode: chat
  # endpoint detail UI Test tab needs this property to know which is the input column for chat flow
  azureml.promptflow.chat_input: question
  # endpoint detail UI Test tab needs this property to know which is the output column for chat flow
  azureml.promptflow.chat_output: answer

Use az ml model create --file model.yaml to register the model to your workspace.

Define the endpoint

To define an endpoint, you need to specify:

• Endpoint name: The name of the endpoint. It must be unique in the Azure region. For more information on the naming rules, see endpoint limits.
• Authentication mode: The authentication method for the endpoint. Choose between key-based authentication and Azure Machine Learning token-based authentication. A key doesn't expire, but a token does expire. For more information on authenticating, see Authenticate to an online endpoint. Optionally, you can add a description and tags to your endpoint.
• Optionally, you can add a description and tags to your endpoint.
• If you want to deploy to a Kubernetes cluster (AKS or Arc enabled cluster) which is attaching to your workspace, you can deploy the flow to be a Kubernetes online endpoint.

Following is an endpoint definition example which by default uses system-assigned identity.

$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineEndpoint.schema.json
name: basic-chat-endpoint
auth_mode: key
properties:
# this property only works for system-assigned identity.
# if the deploy user has access to connection secrets, 
# the endpoint system-assigned identity will be auto-assigned connection secrets reader role as well
  enforce_access_to_default_secret_stores: enabled

$schema: https://azuremlschemas.azureedge.net/latest/kubernetesOnlineEndpoint.schema.json
name: basic-chat-endpoint
compute: azureml:<Kubernetes compute name>
auth_mode: key

If you create a Kubernetes online endpoint, you need to specify the following additional attributes:

For more configurations of endpoint, see managed online endpoint schema.

Use user-assigned identity

By default, when you create an online endpoint, a system-assigned managed identity is automatically generated for you. You can also specify an existing user-assigned managed identity for the endpoint.

If you want to use user-assigned identity, you can specify the following additional attributes in the endpoint.yaml:

identity:
  type: user_assigned
  user_assigned_identities:
    - resource_id: user_identity_ARM_id_place_holder

Besides, you also need to specify the Client ID of the user-assigned identity under environment_variables the deployment.yaml as following. You can find the Client ID in the Overview of the managed identity in Azure portal.

environment_variables:
  AZURE_CLIENT_ID: <client_id_of_your_user_assigned_identity>

Define the deployment

A deployment is a set of resources required for hosting the model that does the actual inferencing.

Following is a deployment definition example, in which the model section refers to the registered flow model. You can also specify the flow model path in line.

$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineDeployment.schema.json
name: blue
endpoint_name: basic-chat-endpoint
model: azureml:basic-chat-model:1
  # You can also specify model files path inline
  # path: examples/flows/chat/basic-chat
environment: 
  image: mcr.microsoft.com/azureml/promptflow/promptflow-runtime:latest
  # inference config is used to build a serving container for online deployments
  inference_config:
    liveness_route:
      path: /health
      port: 8080
    readiness_route:
      path: /health
      port: 8080
    scoring_route:
      path: /score
      port: 8080
instance_type: Standard_E16s_v3
instance_count: 1
environment_variables:
  # for pulling connections from workspace
  PRT_CONFIG_OVERRIDE: deployment.subscription_id=<subscription_id>,deployment.resource_group=<resource_group>,deployment.workspace_name=<workspace_name>,deployment.endpoint_name=<endpoint_name>,deployment.deployment_name=<deployment_name>

  # (Optional) When there are multiple fields in the response, using this env variable will filter the fields to expose in the response.
  # For example, if there are 2 flow outputs: "answer", "context", and I only want to have "answer" in the endpoint response, I can set this env variable to '["answer"]'.
  # If you don't set this environment, by default all flow outputs will be included in the endpoint response.
  # PROMPTFLOW_RESPONSE_INCLUDED_FIELDS: '["category", "evidence"]'

$schema: https://azuremlschemas.azureedge.net/latest/kubernetesOnlineDeployment.schema.json
name: blue
type: kubernetes
endpoint_name: basic-chat-endpoint
model: azureml:basic-chat-model:1
  # You can also specify model files path inline
  # path: examples/flows/chat/basic-chat
environment: 
  image: mcr.microsoft.com/azureml/promptflow/promptflow-runtime:latest
  # inference config is used to build a serving container for online deployments
  inference_config:
    liveness_route:
      path: /health
      port: 8080
    readiness_route:
      path: /health
      port: 8080
    scoring_route:
      path: /score
      port: 8080
instance_type: <kubernetes custom instance type>
instance_count: 1
environment_variables:

  # for pulling connections from workspace
  PRT_CONFIG_OVERRIDE: deployment.subscription_id=<subscription_id>,deployment.resource_group=<resource_group>,deployment.workspace_name=<workspace_name>,deployment.endpoint_name=<endpoint_name>,deployment.deployment_name=<deployment_name>

  # (Optional) When there are multiple fields in the response, using this env variable will filter the fields to expose in the response.
  # For example, if there are 2 flow outputs: "answer", "context", and I only want to have "answer" in the endpoint response, I can set this env variable to '["answer"]'.
  # If you don't set this environment, by default all flow outputs will be included in the endpoint response.
  # PROMPTFLOW_RESPONSE_INCLUDED_FIELDS: '["category", "evidence"]'

If you create a Kubernetes online deployment, you need to specify the following additional attributes:

Deploy your online endpoint to Azure

To create the endpoint in the cloud, run the following code:

az ml online-endpoint create --file endpoint.yml

To create the deployment named blue under the endpoint, run the following code:

az ml online-deployment create --file blue-deployment.yml --all-traffic

Check status of the endpoint and deployment

To check the status of the endpoint, run the following code:

az ml online-endpoint show -n basic-chat-endpoint

To check the status of the deployment, run the following code:

az ml online-deployment get-logs --name blue --endpoint basic-chat-endpoint

Invoke the endpoint to score data by using your model

You can create a sample-request.json file like this:

{
  "question": "What is Azure Machine Learning?",
  "chat_history":  []
}

az ml online-endpoint invoke --name basic-chat-endpoint --request-file sample-request.json

You can also call it with an HTTP client, for example with curl:

ENDPOINT_KEY=<your-endpoint-key>
ENDPOINT_URI=<your-endpoint-uri>

curl --request POST "$ENDPOINT_URI" --header "Authorization: Bearer $ENDPOINT_KEY" --header 'Content-Type: application/json' --data '{"question": "What is Azure Machine Learning?", "chat_history":  []}'

You can get your endpoint key and your endpoint URI from the Azure Machine Learning workspace in Endpoints > Consume > Basic consumption info.

Advanced configurations

Deploy with different connections from flow development

You might want to override connections of the flow during deployment.

For example, if your flow.dag.yaml file uses a connection named my_connection, you can override it by adding environment variables of the deployment yaml like following:

Option 1: override connection name

environment_variables:
  my_connection: <override_connection_name>

If you want to override a specific field of the connection, you can override by adding environment variables with naming pattern <connection_name>_<field_name>. For example, if your flow uses a connection named my_connection with a configuration key called chat_deployment_name, the serving backend will attempt to retrieve chat_deployment_name from the environment variable 'MY_CONNECTION_CHAT_DEPLOYMENT_NAME' by default. If the environment variable is not set, it will use the original value from the flow definition.

Option 2: override by referring to asset

environment_variables:
  my_connection: ${{azureml://connections/<override_connection_name>}}

Deploy with a custom environment

This section will show you how to use a docker build context to specify the environment for your deployment, assuming you have knowledge of Docker and Azure Machine Learning environments.

1. In your local environment, create a folder named image_build_with_reqirements contains following files:

   |--image_build_with_reqirements
   |  |--requirements.txt
   |  |--Dockerfile
   

   • The requirements.txt should be inherited from the flow folder, which has been used to track the dependencies of the flow.

   • The Dockerfile content is as following:

     FROM mcr.microsoft.com/azureml/promptflow/promptflow-runtime:latest
     COPY ./requirements.txt .
     RUN pip install -r requirements.txt
     

2. replace the environment section in the deployment definition yaml file with the following content:

   environment: 
     build:
       path: image_build_with_reqirements
       dockerfile_path: Dockerfile
     # deploy prompt flow is BYOC, so we need to specify the inference config
     inference_config:
       liveness_route:
         path: /health
         port: 8080
       readiness_route:
         path: /health
         port: 8080
       scoring_route:
         path: /score
         port: 8080
   

Use FastAPI serving engine (preview)

By default prompt flow serving uses FLASK serving engine. Starting from prompt flow SDK version 1.10.0, FastAPI based serving engine is supported. You can use fastapi serving engine by specifying an environment variable PROMPTFLOW_SERVING_ENGINE.

environment_variables:
  PROMPTFLOW_SERVING_ENGINE=fastapi

Configure concurrency for deployment

When deploying your flow to online deployment, there are two environment variables, which you configure for concurrency: PROMPTFLOW_WORKER_NUM and PROMPTFLOW_WORKER_THREADS. Besides, you'll also need to set the max_concurrent_requests_per_instance parameter.

Below is an example of how to configure in the deployment.yaml file.

request_settings:
  max_concurrent_requests_per_instance: 10
environment_variables:
  PROMPTFLOW_WORKER_NUM: 4
  PROMPTFLOW_WORKER_THREADS: 1

• PROMPTFLOW_WORKER_NUM: This parameter determines the number of workers (processes) that will be started in one container. The default value is equal to the number of CPU cores, and the maximum value is twice the number of CPU cores.

• PROMPTFLOW_WORKER_THREADS: This parameter determines the number of threads that will be started in one worker. The default value is 1.

  Note

  When setting PROMPTFLOW_WORKER_THREADS to a value greater than 1, ensure that your flow code is thread-safe.

• max_concurrent_requests_per_instance: The maximum number of concurrent requests per instance allowed for the deployment. The default value is 10.

  The suggested value for max_concurrent_requests_per_instance depends on your request time:

  • If your request time is greater than 200 ms, set max_concurrent_requests_per_instance to PROMPTFLOW_WORKER_NUM * PROMPTFLOW_WORKER_THREADS.
  • If your request time is less than or equal to 200 ms, set max_concurrent_requests_per_instance to (1.5-2) * PROMPTFLOW_WORKER_NUM * PROMPTFLOW_WORKER_THREADS. This can improve total throughput by allowing some requests to be queued on the server side.
  • If you're sending cross-region requests, you can change the threshold from 200 ms to 1 s.

While tuning above parameters, you need to monitor the following metrics to ensure optimal performance and stability:

• Instance CPU/Memory utilization of this deployment
• Non-200 responses (4xx, 5xx)
  • If you receive a 429 response, this typically indicates that you need to either retune your concurrency settings following the above guide or scale your deployment.

Monitor endpoints

Collect general metrics

You can view general metrics of online deployment (request numbers, request latency, network bytes, CPU/GPU/Disk/Memory utilization, and more).

Collect tracing data and system metrics during inference time

You can also collect tracing data and prompt flow deployment specific metrics (token consumption, flow latency, etc.) during inference time to workspace linked Application Insights by adding a property app_insights_enabled: true in the deployment yaml file. Learn more about trace and metrics of prompt flow deployment.

Prompt flow specific metrics and trace can be specified to other Application Insights other than the workspace linked one. You can specify an environment variable in the deployment yaml file as following. You can find the connection string of your Application Insights in the Overview page in Azure portal.

environment_variables:
  APPLICATIONINSIGHTS_CONNECTION_STRING: <connection_string>

Common errors

Upstream request timeout issue when consuming the endpoint

Such error is usually caused by timeout. By default the request_timeout_ms is 5000. You can specify at max to 5 minutes, which is 300,000 ms. Following is example showing how to specify request timeout in the deployment yaml file. Learn more about the deployment schema here.

request_settings:
  request_timeout_ms: 300000

properties:
  # indicate a deployment from prompt flow
  azureml.promptflow.source_flow_id: <value>

Next steps

• Learn more about managed online endpoint schema and managed online deployment schema.
• Learn more about how to test the endpoint in UI and monitor the endpoint.
• Learn more about how to troubleshoot managed online endpoints.
• Troubleshoot prompt flow deployments.
• Once you improve your flow, and would like to deploy the improved version with safe rollout strategy, see Safe rollout for online endpoints.
• Learn more about deploy flows to other platforms, such as a local development service, Docker container, Azure APP service, etc.