Agentic RAG: When Agents Control Retrieval

The Limitation of Linear RAG

A standard RAG pipeline runs the same sequence on every query: embed the question, search the vector database, rerank, build the prompt, generate. This works well for straightforward factual questions where a single retrieval step finds the relevant documents. But it fails on queries that require reasoning about what to retrieve, combining information from multiple sources, or selecting between different retrieval strategies based on the query type.

Consider the question: "Compare the pricing tiers of our three CRM integrations and recommend one for a team of 50 users." A linear pipeline embeds this question, retrieves the top 5 most similar chunks, and generates an answer. But the pricing information for three different integrations might live in three separate documents, each with its own structure. A single vector search might retrieve chunks from only one or two of the three, or it might retrieve overview chunks that mention all three but lack the pricing details. The query requires three targeted retrievals (one per integration) followed by synthesis, which is a multi-step plan, not a single retrieval.

Or consider: "What changed in our refund policy between Q1 and Q3 this year?" This requires retrieving two versions of the refund policy (Q1 and Q3), comparing them, and identifying the differences. A single retrieval cannot express the temporal comparison; it would return whichever version is more semantically similar to the query, which is essentially random between the two. An agent can decompose this into two targeted retrievals with date filters and then compare the results.

These are not edge cases. In enterprise RAG deployments, 30% to 40% of user queries are complex enough that a single retrieval step is insufficient, according to evaluations published by LangChain, LlamaIndex, and Vectara across production deployments in 2025 and 2026. Agentic RAG addresses this by giving the retrieval process a reasoning layer.

How Agentic RAG Works

An agentic RAG system gives an LLM agent access to retrieval tools and lets it decide how to use them. The agent receives the user's question, reasons about what information it needs, calls the appropriate tools to retrieve that information, evaluates whether the results are sufficient, and either generates a final answer or performs additional retrieval rounds.

The tools available to the agent typically include: a vector search tool (semantic similarity search against the knowledge base), a keyword search tool (BM25 or full-text search for exact matching), metadata-filtered search (search within a specific document collection, date range, or category), a SQL query tool (if structured data is part of the knowledge base), a web search tool (for information not in the internal knowledge base), and a document summary tool (retrieve a summary of a specific document rather than individual chunks). The agent selects the right tool for each sub-task based on its reasoning about the query.

The reasoning loop follows a pattern similar to ReAct (Reason + Act): the agent thinks about what it needs, takes an action (calls a tool), observes the result, and decides whether it has enough information to answer or needs to take another action. This loop continues until the agent determines that the retrieved context is sufficient, or until a maximum iteration limit is reached to prevent infinite loops.

A concrete example of the loop for the CRM pricing question: the agent reasons that it needs pricing information for three specific integrations. It calls the vector search tool with a query for "Salesforce CRM integration pricing," reads the results, then calls it again with "HubSpot CRM integration pricing," reads those results, then calls it with "Pipedrive CRM integration pricing." It now has pricing data from all three integrations and generates a comparison with a recommendation. Three targeted retrievals, each with a focused query, produce much better results than one broad retrieval with the original complex question.

Query Decomposition

Query decomposition is the most impactful capability that agents bring to RAG. Instead of using the user's question as-is for retrieval, the agent breaks it into sub-questions that are individually retrievable and collectively answer the original question.

There are several decomposition strategies. Sequential decomposition breaks the question into sub-questions where each depends on the answer to the previous one. "What is the revenue impact of the feature our largest customer requested last quarter?" decomposes into: (1) who is our largest customer, (2) what feature did they request last quarter, (3) what is the estimated revenue impact of that feature. Each retrieval uses the answer from the previous step to form a more specific query.

Parallel decomposition breaks the question into independent sub-questions that can be retrieved simultaneously. "Compare our mobile and desktop app performance metrics" decomposes into: (1) what are the mobile app performance metrics, (2) what are the desktop app performance metrics. Both retrievals can run in parallel, and the results are combined for generation.

Conditional decomposition adapts the retrieval plan based on intermediate results. "How should we handle GDPR compliance for our European users' data?" might start with a retrieval about the current data handling practices, then, based on what is found, branch into specific retrievals about consent mechanisms, data retention policies, or cross-border transfer rules depending on which aspects are not already covered.

The decomposition itself is performed by the LLM agent, which makes it flexible but also unpredictable. The agent might decompose a simple question unnecessarily, adding latency without improving quality. Or it might fail to decompose a complex question that needs it. Prompt engineering for the decomposition step, with clear instructions about when to decompose and when to use a direct retrieval, significantly affects quality. Including examples of good decompositions in the agent's system prompt is the most effective technique.

Retrieval Self-Evaluation

A key advantage of agentic RAG is that the agent can evaluate whether the retrieved context is sufficient before generating an answer. In a linear pipeline, the system always generates an answer from whatever was retrieved, even if the retrieval returned irrelevant or incomplete results. An agent can look at the retrieved documents and decide: "these results do not contain pricing information, I need to search with different keywords" or "these results cover two of the three topics, I need to retrieve the third."

Self-evaluation manifests in several patterns. Relevance checking is where the agent examines each retrieved chunk and discards ones that are not relevant to the query before including them in the prompt. This is a lightweight alternative to a dedicated reranker: the agent itself acts as the relevance filter. Sufficiency checking is where the agent determines whether the retrieved context collectively contains enough information to answer the question. If not, it formulates a follow-up retrieval query targeting the missing information. Contradiction detection is where the agent notices when retrieved chunks contain conflicting information and either retrieves additional context to resolve the conflict or surfaces the contradiction in the answer.

The trade-off is latency. Each self-evaluation step requires an LLM call, which adds 200 to 1000 milliseconds. A system that does relevance checking, sufficiency checking, and then generates the answer makes three LLM calls instead of one. For interactive applications where users expect sub-3-second responses, aggressive self-evaluation can push total latency past acceptable limits. The practical approach is to limit self-evaluation to one round: retrieve, evaluate, optionally re-retrieve once, then generate. Allowing unlimited evaluation rounds creates latency spikes and rarely improves answer quality beyond the first correction round.

Tool Selection and Routing

Beyond choosing retrieval queries, agents can select between fundamentally different retrieval tools based on the query type. This is called query routing or tool selection, and it handles the reality that production knowledge bases contain diverse data types that require different retrieval strategies.

A production knowledge base might include unstructured documents (PDFs, web pages, email threads), structured data (database tables, spreadsheets, CRM records), and real-time data (APIs, live dashboards, current inventory). A query about company policy should search the document collection. A query about last month's revenue should query the analytics database. A query about current stock levels should call the inventory API. An agentic RAG system gives the LLM access to all of these as tools and lets it decide which to use.

The implementation uses the LLM's tool-calling capability, which is now a standard feature of all major models (GPT-4o, Claude, Gemini, Llama 3.1+). Each retrieval method is defined as a tool with a description of what data it accesses and when to use it. The agent reads the user's question, selects the appropriate tool (or tools), calls them with appropriate parameters, and uses the combined results for generation. The existing guide on AI tool use covers the broader tool-calling pattern.

Router models are a lighter-weight alternative where a small classifier (or a fast LLM call) routes the query to the appropriate retrieval method without the full agent loop. This is faster because the routing decision is a single inference step rather than a multi-step reasoning chain, but less flexible because the router cannot compose multiple tools or iterate based on results. Use a router when your retrieval methods are clearly separable (documents vs. database vs. API) and an agent when queries frequently need multiple methods combined.

Frameworks for Agentic RAG

Several frameworks provide scaffolding for building agentic RAG systems. LangGraph (from LangChain) models the agent as a graph of nodes, where each node is a step (retrieve, evaluate, generate) and edges define the control flow, including conditional branches based on evaluation results. LangGraph provides persistence, streaming, and human-in-the-loop breakpoints, making it the most mature option for production agentic RAG.

LlamaIndex Workflows (formerly LlamaIndex Agents) provide a similar event-driven architecture with built-in RAG tooling: query engines, sub-question query engines, and retriever tools that the agent can call. LlamaIndex's advantage is its deep integration with retrieval components, chunking, indexing, and querying are all first-class concepts.

Anthropic's Agent SDK and OpenAI's Agents API provide lower-level agent primitives with tool calling, where you define the tools and the control flow, giving more flexibility but requiring more custom code. These are better starting points if your retrieval architecture is custom and does not fit neatly into LangChain's or LlamaIndex's abstractions.

CrewAI and AutoGen specialize in multi-agent systems where different agents handle different aspects of the retrieval and generation process. A retrieval agent handles document search, a validation agent checks the results, and a generation agent produces the answer. This is overkill for most RAG applications but powerful for complex enterprise workflows where different knowledge domains have different access controls and retrieval strategies. The existing pillar on AI agent memory covers how persistent memory interacts with agent architectures.

When to Use Agentic RAG vs Linear RAG

Agentic RAG adds latency (multiple LLM calls), cost (more tokens consumed), and complexity (harder to debug, less predictable). It is not always worth it. Use linear RAG when queries are straightforward and can be answered from a single retrieval step, when latency requirements are tight (under 2 seconds), and when the knowledge base is homogeneous (all documents are the same type and structure). This covers the majority of FAQ bots, documentation assistants, and simple knowledge search applications.

Use agentic RAG when queries are complex (multi-part, comparative, temporal), when the knowledge base contains diverse data types requiring different retrieval strategies, when answer quality matters more than latency, and when a significant fraction of queries fail with linear RAG because the single retrieval step does not find the right context. Enterprise knowledge assistants, research tools, compliance systems, and any application where users ask open-ended analytical questions benefit from agentic RAG.

A practical middle ground is a hybrid approach: use linear RAG as the default path and escalate to agentic RAG when the linear pipeline's confidence is low. The linear pipeline retrieves and generates an answer, a lightweight classifier evaluates confidence (based on retrieval scores, answer length, or presence of hedging language), and low-confidence queries are re-processed through the agentic pipeline. This keeps latency low for simple queries while improving quality for complex ones.