Refine Summarization: Sequential Processing for Better Coherence
While map-reduce is the standard approach for long-document summarization (and the correct default choice for most production workloads), it has a structural weakness: chunk summaries are generated independently, so cross-chunk relationships, evolving arguments, and narrative threads that span multiple chunks are often lost. The reduce step tries to recover these relationships from the isolated chunk summaries, but it is working from compressed, decontextualized pieces. Refine avoids this problem entirely because every step sees the full accumulated context from all prior chunks.
Step 1: Summarize the First Chunk
The initial summary sets the tone, structure, and coverage baseline for the entire refine chain. It is the only step that works without prior context, so it functions identically to a standard single-chunk summarization.
Initial prompt: "Summarize the following text in [300-500 words]. This is the beginning of a longer document, and your summary will be iteratively refined as subsequent sections are processed. Capture the main topic, key entities introduced, and any thesis or argument established. Preserve all names, numbers, and specific claims."
Length target for the initial summary: Set this to roughly 80-100% of your final target length. If you want a 500-word final summary, start with 400-500 words. This gives subsequent refine steps room to add new information without the summary growing uncontrollably. Starting too short (100 words for a 500-word target) means later chunks have to add enormous amounts of information in single steps, which degrades quality. Starting too long means the summary will need aggressive compression in later steps, which risks losing early information.
Marking uncertainty: Instruct the initial summary to flag information that seems incomplete or context-dependent: "If the text introduces concepts, entities, or arguments without fully resolving them, note these as pending. They may be resolved in subsequent sections." This primes the refine chain to watch for resolutions in later chunks.
Step 2: Refine with Each Subsequent Chunk
For each chunk after the first, send both the running summary and the new chunk to the model. The model must decide what new information from the chunk should be incorporated into the summary, what existing information should be updated or corrected, and what should be left unchanged.
Refine prompt template: "Below is a running summary of a document so far, followed by the next section of the document. Update the summary to incorporate important new information from the new section. Rules: (1) Add new facts, entities, and conclusions from the new section. (2) If the new section updates, corrects, or resolves information from the existing summary, modify the summary accordingly. (3) Do not remove information from the existing summary unless the new section explicitly contradicts it. (4) Keep the summary at approximately [target length] words. If adding new information would exceed this, compress less important details from earlier sections. (5) Maintain narrative coherence. The summary should read as a single cohesive piece, not as a patchwork of additions."
Running summary + new chunk must fit in context: Each refine step requires the running summary (300-500 words, roughly 400-650 tokens) plus the new chunk (typically 2,000-6,000 tokens) to fit within the context window. With modern models at 128K-200K context, this is never a bottleneck. But with smaller context models, keep chunk sizes and summary lengths proportionally smaller.
Processing order matters: Refine is inherently sequential, each step depends on the output of the previous step. This means you cannot parallelize the refine chain. A 20-chunk document requires 20 sequential API calls. At 2-4 seconds per call, the total latency is 40-80 seconds, compared to 2-4 seconds for parallel map-reduce. This latency difference is the primary reason map-reduce is preferred for most production workloads.
Step 3: Apply Recency-Bias Correction
The refine pattern has a systematic bias: later chunks disproportionately influence the final summary. Each refine step sees the new chunk in full detail but sees earlier content only through the compressed running summary. After 15 refinement steps, the first chunk's information has been compressed through 14 successive summaries, while the last chunk's information was added in full detail just one step ago.
Measuring the bias: Compare entity density from the first third, middle third, and last third of the document in the final summary. If the last third contributes 50%+ of entities while the first third contributes under 20%, recency bias is distorting the summary. A well-balanced summary should represent all sections roughly proportionally to their importance, not their position.
Correction strategy 1: Fixed-importance budgeting. Allocate a minimum word budget for each major document section. "Your summary must allocate at least 100 words to the introduction and methodology (chunks 1-5), at least 150 words to the results (chunks 6-12), and at least 100 words to the discussion and conclusions (chunks 13-18)." This prevents late-document content from squeezing out early content.
Correction strategy 2: Post-processing rebalance. After the refine chain completes, run a final rebalancing step: "The following summary was produced by processing a document sequentially. Review it for balance: does the summary fairly represent the beginning, middle, and end of the document? If early sections are under-represented, expand their coverage. If later sections are over-represented, compress them. Maintain the same total length." This adds one extra API call but significantly improves balance.
Correction strategy 3: Bidirectional refine. Process the document in both forward and reverse order, producing two summaries. Then merge them. The forward pass over-represents late content; the reverse pass over-represents early content. The merge naturally balances both perspectives. This doubles the cost (two full refine chains plus a merge) but produces the most balanced results.
Step 4: Choose Refine vs Map-Reduce Based on Document Type
Refine and map-reduce are not competing approaches, they are optimized for different document characteristics. Choosing correctly based on the document type avoids the weaknesses of each pattern.
Use refine for narrative and chronological documents: Novels, case studies, historical accounts, biography chapters, meeting transcripts, legal proceedings with chronological facts. These documents build meaning sequentially: later content depends on earlier context. A character introduced in chapter 1 appears throughout the book; a legal argument established in the facts section is referenced in the holdings. Refine preserves these cross-chunk relationships because each step sees prior context. Map-reduce would summarize each chapter independently, losing character development, argument progression, and narrative arc.
Use map-reduce for analytical and modular documents: Research papers with independent sections, technical documentation with standalone topics, FAQ collections, product catalogs, regulatory filings with self-contained sections. These documents contain information that is largely independent across sections. A risk factors section in an SEC filing does not depend on the financial statements section for meaning. Map-reduce works well because chunk independence aligns with document structure.
Use refine for contradiction-heavy documents: Documents where later content corrects, updates, or refines earlier claims benefit from refine because each step can update the running summary with corrections. Map-reduce treats all chunks as equally authoritative and has no mechanism for later chunks to override earlier ones.
Use map-reduce when latency matters: If the user is waiting for the summary and the document has 20+ chunks, refine will take 40-80 seconds while map-reduce takes 3-5 seconds. Latency-sensitive applications should default to map-reduce unless the document type specifically benefits from refine's coherence advantage.
Hybrid Approaches
Map then refine: Use map-reduce for the initial summarization (fast, parallel), then run one refine pass over the chunk summaries (sequential, coherent). This gives you most of map-reduce's speed with some of refine's coherence. The refine pass processes chunk summaries (short documents) rather than raw chunks (long documents), so it completes in 5-10 seconds rather than 40-80.
Windowed refine: Instead of maintaining a single running summary across all chunks, refine over sliding windows of 3-5 chunks. Window 1 covers chunks 1-3, window 2 covers chunks 2-5 (overlapping by 1), window 3 covers chunks 4-7, and so on. Then merge window summaries. This provides refine's cross-chunk coherence within windows while limiting the recency bias that accumulates across the full document. The window summaries can be merged in parallel, recovering some of map-reduce's speed.
Refine with memory: Store key entities and claims from earlier chunks in a structured memory (a list of facts, not a prose summary). Each refine step receives the running summary, the new chunk, AND the structured memory. The memory ensures early-document facts survive compression even after many refinement steps because they are preserved in structured form outside the summary's token budget.
Refine summarization processes chunks sequentially, building a running summary that incorporates each new section. It produces more coherent summaries than map-reduce for narrative and chronological documents but takes 10-20x longer due to sequential processing and suffers from recency bias favoring later content. Correct for recency bias with post-processing rebalancing or bidirectional processing. Default to map-reduce for speed, and switch to refine only when cross-chunk coherence is critical.