Agentic retrieval in Azure AI Search

What is agentic retrieval? In Azure AI Search, agentic retrieval is a new multi-query pipeline designed for complex questions posed by users or agents in chat and copilot apps. It's intended for Retrieval Augmented Generation (RAG) patterns and agent-to-agent workflows.

Here's what it does:

• Uses a large language model (LLM) to break down a complex query into smaller, focused subqueries for better coverage over your indexed content. Subqueries can include chat history for extra context.

• Runs subqueries in parallel. Each subquery is semantically reranked to promote the most relevant matches.

• Combines the best results into a unified response that an LLM can use to generate answers with your proprietary content.

• The response is modular yet comprehensive in how it also includes a query plan and source documents. You can choose to use just the search results as grounding data, or invoke the LLM to formulate an answer.

This high-performance pipeline helps you generate high quality grounding data (or an answer) for your chat application, with the ability to answer complex questions quickly.

Programmatically, agentic retrieval is supported through a new Knowledge Agents object in the 2025-08-01-preview and 2025-05-01-preview data plane REST APIs and in Azure SDK preview packages that provide the feature. A knowledge agent's retrieval response is designed for downstream consumption by other agents and chat apps.

Why use agentic retrieval

You should use agentic retrieval when you want to provide agents and apps with the most relevant content for answering harder questions, leveraging chat context and your proprietary content.

The agentic aspect is a reasoning step in query planning processing that's performed by a supported large language model (LLM) that you provide. The LLM analyzes the entire chat thread to identify the underlying information need. Instead of a single, catch-all query, the LLM breaks down compound questions into focused subqueries based on: user questions, chat history, and parameters on the request. The subqueries target your indexed documents (plain text and vectors) in Azure AI Search. This hybrid approach ensures you surface both keyword matches and semantic similarities at once, dramatically improving recall.

The retrieval component is the ability to run subqueries simultaneously, merge results, semantically rank results, and return a three-part response that includes grounding data for the next conversation turn, reference data so that you can inspect the source content, and an activity plan that shows query execution steps.

Query expansion and parallel execution, plus the retrieval response, are the key capabilities of agentic retrieval that make it the best choice for generative AI (RAG) applications.

Agentic retrieval adds latency to query processing, but it makes up for it by adding these capabilities:

• Reads in chat history as an input to the retrieval pipeline.
• Deconstructs a complex query that contains multiple "asks" into component parts. For example: "find me a hotel near the beach, with airport transportation, and that's within walking distance of vegetarian restaurants."
• Rewrites an original query into multiple subqueries using synonym maps (optional) and LLM-generated paraphrasing.
• Corrects spelling mistakes.
• Executes all subqueries simultaneously.
• Outputs a unified result as a single string. Alternatively, you can extract parts of the response for your solution. Metadata about query execution and reference data is included in the response.

Agentic retrieval invokes the entire query processing pipeline multiple times for each subquery, but it does so in parallel, preserving the efficiency and performance necessary for a reasonable user experience.

Architecture and workflow

Agentic retrieval is designed for conversational search experiences that use an LLM to intelligently break down complex queries. The system coordinates multiple Azure services to deliver comprehensive search results.

How it works

The agentic retrieval process works as follows:

1. Workflow initiation: Your application calls a knowledge agent with retrieve action that provides a query and conversation history.

2. Query planning: A knowledge agent sends your query and conversation history to an LLM, which analyzes the context and breaks down complex questions into focused subqueries. This step is automated and not customizable.

3. Query execution: The knowledge agent sends the subqueries to your knowledge sources. All subqueries run simultaneously and can be keyword, vector, and hybrid search. Each subquery undergoes semantic reranking to find the most relevant matches. References are extracted and retained for citation purposes.

4. Result synthesis: The system combines all results into a unified response with three parts: merged content, source references, and execution details.

Your search index determines query execution and any optimizations that occur during query execution. Specifically, if your index includes searchable text and vector fields, a hybrid query executes. If the only searchable field is a vector field, then only pure vector search is used. The index semantic configuration, plus optional scoring profiles, synonym maps, analyzers, and normalizers (if you add filters) are all used during query execution. You must have named defaults for a semantic configuration and a scoring profile.

Required components

Integration requirements

Your application drives the pipeline by calling the knowledge agent and handling the response. The pipeline returns grounding data that you pass to an LLM for answer generation in your conversation interface. For implementation details, see Tutorial: Build an agent-to-agent retrieval solution.

How to get started

You must use the preview REST APIs or a prerelease Azure SDK package that provides the functionality. At this time, there's no Azure portal or Azure AI Foundry portal support.

Choose any of these options for your next step.

• Quickstart article: Run agentic retrieval in Azure AI Search. Learn the basic workflow using sample data and a prepared index and queries.

• Sample code:

  • Quickstart-Agentic-Retrieval: Python
  • Quickstart-Agentic-Retrieval: .NET
  • Quickstart-Agentic-Retrieval: REST
  • End-to-end with Azure AI Search and Azure AI Agent Service

• How-to guides for a focused look at development tasks:

  • Create a search index knowledge source or a blob knowledge source
  • Create a knowledge agent
  • Use answer synthesis for citation-backed responses
  • Use a knowledge agent to retrieve data
  • Tutorial: Build an agent-to-agent retrieval solution

• REST API reference:

  • Knowledge Sources
  • Knowledge Agents
  • Knowledge Retrieval

• Azure OpenAI Demo, updated to use agentic retrieval.

Availability and pricing

Agentic retrieval is available in all regions that provide semantic ranker, on all tiers except the free tier.

Billing for agentic retrieval has two parts:

• Billing for query planning and answer synthesis (optional) is Standard Pay-in-Advance Offer in Azure OpenAI. It's token based for both input and output tokens. The model you assign to the knowledge agent is the one charged for token usage. For example, if you use gpt-4o, the token charge appears in the bill for gpt-4o.

• Billing for semantic ranking during query execution. Billing is suspended during the initial roll-out phase but then transitions to Standard Pay-in-Advance Offer on the Azure AI Search side through the semantic ranker. Semantic ranker, which is a premium billable feature, is an integral part of agentic retrieval. You're charged on the Azure AI Search side for token inputs to the semantic ranking models.

Semantic ranking is performed for every subquery in the plan. Semantic ranking charges are based on the number of tokens returned by each subquery.