Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
APPLIES TO: All API Management tiers
The AI gateway in Azure API Management is a set of capabilities that help you manage your AI backends effectively. Use these capabilities to secure, scale, monitor, and govern AI models, agents, and tools that back your intelligent apps and workloads.
Use the AI gateway to manage a wide range of AI endpoints, including:
- Remote MCP servers and A2A agent APIs
- Self-hosted models and endpoints
Note
- The AI gateway, including MCP server capabilities, extends API Management's existing API gateway; it's not a separate offering.
- Capability availability varies by API Management service tier. See the linked documentation articles for details.
Why use an AI gateway?
AI adoption in organizations involves several phases:
- Defining requirements and evaluating AI models
- Building AI apps and agents that need access to AI models and services
- Operationalizing and deploying AI apps and backends to production
As AI adoption matures, especially in larger enterprises, the AI gateway helps address key challenges. It helps you:
- Authenticate and authorize access to AI services
- Load balance across multiple AI endpoints
- Monitor and log AI interactions
- Manage token usage and quotas across multiple applications
- Enable self-service for developer teams
Traffic mediation and control
By using the AI gateway, you can:
- Govern chat completions, responses, and real-time APIs.
- Expose your existing REST APIs as MCP servers, and support passthrough to MCP servers.
- Import and manage A2A agent APIs.
More information:
Native scaling features in API Management
API Management also provides built-in scaling features to help the gateway handle high volumes of requests to your AI APIs. These features include automatic or manual addition of gateway scale units and addition of regional gateways for multiregion deployments. Specific capabilities depend on the API Management service tier.
More information:
Note
While API Management can scale gateway capacity, you also need to scale and distribute traffic to your AI backends to accommodate increased load (see the Resiliency section). For example, to take advantage of geographical distribution of your system in a multiregion configuration, you should deploy backend AI services in the same regions as your API Management gateways.
Security and safety
An AI gateway secures and controls access to your AI APIs. By using the AI gateway, you can:
- Use managed identities to authenticate to AI services in Azure, so you don't need API keys for authentication.
- Configure OAuth authorization for AI apps and agents to access APIs or MCP servers by using API Management's credential manager.
More information:
Resiliency
One challenge when building intelligent applications is ensuring that the applications are resilient to backend failures and can handle high loads. By configuring your LLM endpoints with backends in Azure API Management, you can balance the load across them. You can also define circuit breaker rules to stop forwarding requests to AI service backends if they're not responsive.
Load balancer
The backend load balancer supports round-robin, weighted, priority-based, and session-aware load balancing. You can define a load distribution strategy that meets your specific requirements. For example, define priorities within the load balancer configuration to ensure optimal utilization of specific Microsoft ai-services endpoints, particularly those purchased as PTU instances.
Circuit breaker
The backend circuit breaker features dynamic trip duration, applying values from the Retry-After header provided by the backend. This feature ensures precise and timely recovery of the backends, maximizing the utilization of your priority backends.
More information:
Observability and governance
API Management provides comprehensive monitoring and analytics capabilities to track token usage patterns, optimize costs, ensure compliance with your AI governance policies, and troubleshoot problems with your AI APIs. Use these capabilities to:
- Log prompts and completions to Azure Monitor.
- Track token metrics per consumer in Application Insights.
- View the built-in monitoring dashboard.
- Configure policies with custom expressions.
- Manage token quotas across applications.
Early access to AI gateway features
As an API Management customer, you can get early access to new features and capabilities through the AI Gateway Early release channel. This access lets you try out the latest AI gateway innovations before they're generally available and provide feedback to help shape the product. Currently, the AI gateway early release channel is available in the classic tiers of Azure API Management.
More information: