Agentic retrieval in Azure AI Search

Note

Some agentic retrieval features are generally available in the 2026-04-01 REST API version via programmatic access.

If you choose to use a preview REST API version, you can access agentic retrieval capabilities that aren't yet generally available. Preview features are provided without a service-level agreement and aren't recommended for production workloads.

Important

These features and functionality are part of the 2026-05-01-preview REST API. The 2026-05-01-preview is licensed to you as part of your Azure subscription and is subject to the terms applicable to "Previews" in the Microsoft Product Terms, the Microsoft Products and Services Data Protection Addendum ("DPA"), and the Supplemental Terms of Use for Azure Previews.

The 2026-05-01-preview supports connections to other Microsoft services and third-party services. Use of these services is subject to their respective terms and might result in data processing or storage outside of the Azure compliance boundary, as well as data flowing into the Azure compliance boundary.

It's your responsibility to manage whether your data will flow outside of your organization's compliance and geographic boundaries and any related implications, and that appropriate permissions, boundaries, and approvals are provisioned.

You're responsible for carefully reviewing and testing applications you build in the context of your specific use cases and making all appropriate decisions and customizations. This includes implementing your own responsible AI mitigations, such as metaprompts, content filters, or other safety systems, and ensuring your applications meet appropriate quality, reliability, security, and trustworthiness standards.

In Azure AI Search, agentic retrieval is a multi-query pipeline designed for complex questions posed by users or agents in chat and copilot apps. It's intended for retrieval-augmented generation (RAG) patterns and agent-to-agent workflows.

Here's what it does:

  • Can use a large language model (LLM) to break down a complex query into smaller, focused subqueries for better coverage over proprietary and external content. Subqueries can include chat history for extra context.

  • Runs subqueries in parallel. Each subquery is semantically reranked to promote the most relevant matches.

  • Combines the best results into a unified response that an LLM can use to generate grounded answers.

  • Can return source references and an activity log alongside the merged content, so you can use just the grounding data or pass it to an LLM for a full answer.

This high-performance pipeline helps you generate high-quality grounding data or answers for your chat application, with the ability to answer complex questions quickly.

Why use agentic retrieval?

Use agentic retrieval when you want to provide agents and apps with the most relevant content for answering harder questions, drawing on chat context, your proprietary content, and external sources.

Agentic retrieval adds latency compared to a single-query pipeline, but it handles query complexity that a single query can't. For example, it can handle:

  • Questions with multiple asks, such as "find me a hotel near the beach, with airport transportation, and that's within walking distance of vegetarian restaurants."

  • Questions that depend on earlier context in the conversation.

  • Queries that benefit from rewriting, using synonym maps and LLM-generated paraphrasing to expand coverage across your content.

  • Spelling mistakes.

Diagram of a complex query showing how agentic retrieval handles implied context and an intentional typo.

Architecture and workflow

The agentic retrieval process works as follows:

  1. Workflow initiation: Your application calls a knowledge base with a retrieve action that provides a query and conversation history.

  2. Query planning: At low and medium retrieval reasoning effort, the knowledge base sends your query and conversation history to an LLM, which generates focused subqueries. At minimal effort, this step is skipped and queries are issued directly to knowledge sources. Reasoning effort defaults to low and is configured on the knowledge base.

  3. Query execution: The knowledge base sends the subqueries to your knowledge sources. All subqueries run simultaneously and can be keyword, vector, or hybrid search. Each subquery undergoes semantic reranking to find the most relevant matches. References are extracted and retained for citation purposes.

  4. Result synthesis: The system combines all results into a unified response. Merged content is always returned. Source references and an execution activity log are optional.

Diagram of agentic retrieval workflow using an example query.

Components

For all agentic retrieval scenarios, a knowledge base and at least one knowledge source are required. Other components are optional and depend on your configuration.

Component Service Role
Knowledge base Azure AI Search Orchestrates the pipeline, managing knowledge sources and query parameters.
Knowledge source Azure AI Search Defines the content used in the pipeline. Can be indexed (backed by a search index on your service) or remote (content retrieved at query time from an external platform).
Search index Azure AI Search Stores searchable content (text and vectors) with a semantic configuration. Determines which query types run and which optimizations apply. Required for indexed knowledge sources only.
Semantic ranker Azure AI Search Used internally by the agentic retrieval pipeline to rerank results for relevance (L2 reranking).

Integration requirements

Your application drives the pipeline by calling the knowledge base and handling the response. The pipeline returns grounding data that you can pass to an LLM for answer generation or use directly in your conversation interface.

How to get started

To create an agentic retrieval solution, you can use the Azure portal, REST APIs, or an Azure SDK package that provides the functionality.

  • Last updated on