How to Use Summarization in RAG Pipelines
The fundamental tension in RAG is between retrieval breadth and generation budget. You want to retrieve enough context to answer the question comprehensively (10-20 chunks from different sources), but the generation model has a finite context window and performs worse when overwhelmed with irrelevant information. Summarization resolves this tension by compressing retrieved content to its essential claims, fitting more information into fewer tokens.
Step 1: Add Pre-Retrieval Chunk Summaries
At indexing time, when you chunk documents and embed them for vector search, also generate a summary of each chunk. Store the summary alongside the raw chunk text and create embeddings for both.
Why chunk summaries improve retrieval: Raw chunks often contain detailed implementation specifics, examples, and tangential information that make the embedding noisy. A 500-token chunk about database connection pooling might spend 300 tokens on a code example and 200 tokens on the concept explanation. The embedding captures the code example's specifics (PostgreSQL, asyncpg library, pool_size=20) as strongly as the core concept. A query about "database connection pooling best practices" might miss this chunk because the embedding is diluted by implementation details.
A 50-100 word summary of the same chunk ("Database connection pooling manages a pool of reusable connections to avoid the overhead of establishing new connections per query. Key parameters are pool size, timeout, and max overflow. PostgreSQL with asyncpg is shown as an example.") produces an embedding that matches conceptual queries more precisely because the noise is stripped.
Dual-representation retrieval: The best approach is to search against both raw chunk embeddings and summary embeddings, then combine the results. Some queries match better against raw text (specific code patterns, exact error messages), while others match better against summaries (conceptual questions, "how does X work" queries). Reciprocal rank fusion or a simple union of top-K from each index gives you the best of both worlds.
Summary prompt for indexing: "Summarize this text chunk in 2-3 sentences. Focus on the main concept, key terms, and what this chunk is about at a topic level. Do not include code examples or specific values unless they are the primary subject." This prompt targets the conceptual layer while stripping implementation noise.
Cost at indexing: Generating a 50-word summary for each chunk costs approximately $0.001 per chunk with Claude Haiku. A corpus of 100,000 chunks costs $100 to summarize, a one-time investment that pays for itself immediately through better retrieval accuracy. Re-summarize only when chunks change (document updates).
Step 2: Compress Retrieved Context Before Generation
After retrieval returns the top-K chunks (typically 5-20), you face a budget decision. Sending all retrieved chunks raw to the generation model uses thousands of tokens, leaving less room for the system prompt, conversation history, and the model's reasoning. Summarizing the retrieved chunks before generation compresses the context by 60-80% while preserving the relevant information.
Simple concatenate-and-summarize: Concatenate all retrieved chunks in relevance order, then summarize the concatenated text into a target length. "The following text passages were retrieved as context for answering a user question. Summarize them into [500 words], preserving all facts, numbers, and specific claims relevant to answering questions about [topic]. Remove redundancy where multiple passages discuss the same point."
Per-chunk extraction then merge: For each retrieved chunk, extract the 2-3 most relevant sentences to the query. Concatenate the extracted sentences from all chunks. This is faster than full summarization (extraction is cheaper than generation) and preserves exact source wording, which reduces hallucination risk in the final answer. Tools like LLMLingua and LongLLMLingua implement this approach as prompt compression.
How much to compress: Target 20-40% of the original retrieved context length. If you retrieve 10 chunks averaging 500 tokens each (5,000 total), compress to 1,000-2,000 tokens. This leaves comfortable room for the system prompt (500-1,000 tokens), conversation history (500-2,000 tokens), and the model's response generation. The optimal compression ratio depends on your context window size and how much space other prompt components consume.
When to skip compression: If your retrieved chunks total less than 2,000 tokens and your context window is 128K+, send them raw. Compression adds latency and risks losing information. Only compress when the retrieved context is large relative to your available context budget.
Step 3: Use Query-Focused Summarization
Generic summarization ("summarize this text") retains information proportional to its prominence in the source. Query-focused summarization ("summarize this text with respect to the question: [user query]") retains information proportional to its relevance to the specific question being asked.
Why this matters for RAG: Retrieved chunks often contain relevant and irrelevant information side by side. A chunk about database performance might discuss both indexing strategies (relevant to a query about slow queries) and backup procedures (irrelevant). Generic summarization keeps both topics. Query-focused summarization drops the backup discussion and expands on indexing, producing compressed context that is entirely relevant to the query.
Query-focused prompt: "Given the user question: '[query]', summarize the following retrieved passages, keeping ONLY the information directly relevant to answering this question. Drop any information that does not help answer the question, even if it is interesting or important in other contexts. Target [300 words]."
Query-focused summarization reduces "lost in the middle" effects: Research shows that LLMs pay less attention to information in the middle of long contexts. By compressing to only query-relevant content, you shorten the context and eliminate the middle positions where information gets overlooked. A 5,000-token retrieved context compressed to 1,500 tokens of query-relevant information performs better than the full 5,000 tokens because every token in the compressed version is relevant and there is no dilution.
Multi-hop query handling: For complex queries that require information from multiple retrieved chunks ("compare the pricing of Pinecone and Weaviate for 1M vectors"), query-focused summarization of each chunk independently may not capture the comparison aspect. In this case, summarize all chunks together with the comparison query: "Summarize these passages to compare [X] and [Y] on [criteria]. Extract the relevant data points from each passage and present them side by side."
Step 4: Summarize Multi-Turn RAG Conversation History
In multi-turn RAG conversations, each turn involves retrieval and generation. After several turns, the conversation history itself becomes a context budget problem: 5 prior question-answer pairs might consume 3,000-5,000 tokens, leaving less room for newly retrieved content.
Conversation summary as retrieval context: After every 3-5 turns, summarize the conversation history into a 200-400 word summary that captures: questions asked so far, answers provided, topics covered, and any user preferences or constraints expressed. Use this summary instead of the raw conversation history in subsequent turns. This frees 2,000-4,000 tokens for retrieval results while maintaining conversational continuity.
Rolling summary pattern: Maintain a running summary that updates after each turn. Each update takes the previous summary plus the latest Q&A pair and produces an updated summary of the same target length. This is the "refine" summarization pattern applied to conversation management, and it keeps the conversation context at a fixed token cost regardless of how many turns have occurred.
Query reformulation from conversation summary: Use the conversation summary to reformulate the user's latest query for better retrieval. "Given this conversation context: [summary], reformulate the user's question '[latest query]' into a self-contained search query that incorporates relevant context from the conversation." This helps retrieval find relevant chunks even when the user's query is ambiguous without conversation context ("what about the pricing?" becomes "what is the pricing for Pinecone vector database" based on the conversation summary).
Architecture: Where Summarization Runs in the Pipeline
Pre-retrieval summarization (indexing time): Runs once per chunk when documents are ingested. Batch process, latency irrelevant, use cheap models. Output is stored alongside chunk embeddings in the vector database.
Post-retrieval summarization (query time): Runs on every user query. Latency-sensitive, adds 1-3 seconds to query processing. Use fast models (Haiku, GPT-4o-mini) and cache summaries of frequently retrieved chunk combinations.
Conversation summarization (between turns): Runs after every N turns or when conversation history exceeds a token threshold. Semi-latency-sensitive (can run asynchronously while the user reads the response). Use the same model as generation for consistency.
These three summarization points operate independently. You can implement any combination: just pre-retrieval summaries (cheapest, improves retrieval), just post-retrieval compression (simplest, reduces generation context), or all three (best quality, highest cost). Start with post-retrieval compression because it has the largest immediate impact on answer quality.
Summarization improves RAG at three points: chunk summaries at indexing time produce better retrieval embeddings, post-retrieval compression fits more information into the generation budget, and query-focused summarization ensures only relevant content reaches the model. Start with post-retrieval compression for the biggest immediate impact, then add pre-retrieval chunk summaries for retrieval precision and conversation summarization for multi-turn coherence.