Reward Shaping for Retrieval Systems

Why Sparse Rewards Slow Learning

The fundamental challenge of RL in retrieval is reward sparsity. The true reward signal (did the user achieve their goal?) arrives after the full interaction is complete and is available for only a fraction of interactions where users provide explicit feedback. Without intermediate signals, the system must connect its ranking decisions to an outcome that may be minutes or hours away, across many intervening steps.

Consider a memory retrieval system that serves five memories in response to a query. The user interacts with the model's response, eventually finishes their task, and the session ends. The system needs to figure out which of the five memories contributed to the outcome, whether the ranking order mattered, and whether a different set of memories would have produced a better result. This credit assignment problem is hard enough with immediate feedback; it becomes nearly impossible when the feedback is delayed by minutes and available for only 5-10% of sessions.

Reward shaping bridges this gap by providing immediate, dense signals based on known good practices. Returning results with high similarity scores? Small positive reward. Including a diverse set of memories rather than near-duplicates? Small positive reward. Matching the query type with an appropriate ranking strategy? Small positive reward. These shaped rewards accelerate learning by giving the system guidance during the long stretches between sparse true rewards.

Potential-Based Shaping

Naive reward shaping can change the optimal policy: the system might learn to maximize the shaped reward rather than the true reward. If you add a large bonus for returning recent results, the system might learn to always prioritize recency even when older, more relevant results would better serve the user. The shaped reward becomes the target instead of a guide toward the real target.

Potential-based reward shaping (Ng, Harada, and Russell, 1999) solves this problem with a mathematical guarantee. The shaped reward is computed as the difference in a potential function between the current and next state: F(s, s') = gamma * phi(s') - phi(s), where phi is the potential function and gamma is the discount factor. This formulation guarantees that the optimal policy remains the same regardless of the shaping function. Shaping only affects learning speed, not the final outcome.

In practice, potential-based shaping for retrieval defines a potential function over the state space (query context, current ranking, user history). States that are closer to satisfying the user have higher potential. The shaped reward nudges the system toward high-potential states without distorting what constitutes a truly good outcome. The key insight is that the potential function captures your prior knowledge about what makes a retrieval state "good" without overriding what the system learns from actual user feedback.

Practical Shaping Patterns for Retrieval

Diversity bonus. Add a small reward for returning results that cover multiple aspects of the query rather than redundant near-duplicates. Measure diversity as the average pairwise dissimilarity among the returned results. When five results all have similarity above 0.95 to each other, the set is redundant. When they cover distinct subtopics (similarity below 0.7 to each other), the set provides comprehensive coverage. The diversity bonus shapes the system toward the latter pattern without requiring explicit diversity optimization in the ranking function.

Confidence alignment. Add a small reward for returning results whose confidence matches the query context. High-stakes queries (about production configurations, architecture decisions, important customer details) should retrieve high-confidence memories because errors in these contexts are costly. Exploratory queries (brainstorming, "what else do we know about X") can tolerate lower-confidence results because the user is exploring rather than relying on the information. The alignment bonus shapes the system to calibrate its confidence threshold to the query intent.

Recency appropriateness. Add a small reward for matching recency to query intent. Questions about current state ("what database are we using now?") should retrieve recent memories. Questions about history ("what did we decide last month?") should retrieve older memories. Questions about stable facts ("what is the API rate limit?") should retrieve the most-corroborated memory regardless of age. The recency bonus shapes the system to detect temporal intent and adjust its recency weighting accordingly.

Source variety. Add a small reward for retrieving memories from different sources (different conversations, different time periods, different extraction types). A retrieval set that draws from three independent conversations provides more robust evidence than one that draws from a single conversation. The variety bonus shapes the system toward comprehensive answers that are corroborated across multiple independent sources.

Anti-Patterns and Common Mistakes

The most common mistake in reward shaping is making the shaped reward too strong relative to the true reward. If the diversity bonus is large enough to override the relevance signal, the system learns to return maximally diverse but irrelevant results. The shaped rewards should be small nudges (5-15% of the magnitude of the true reward), not dominant signals. They guide the system's exploration, they do not define its objective.

Another common mistake is shaping for observable proxies rather than actual value. Click-through rate is an observable proxy for relevance, but optimizing for clicks leads to clickbait: results with engaging titles that do not actually answer the query. Dwell time is a proxy for content quality, but optimizing for dwell time rewards lengthy content regardless of whether it is useful. Always validate that your shaped reward correlates with the outcome you actually care about, not just with the metric you can measure easily.

A subtler mistake is layering too many shaping signals. Each additional shaped reward adds complexity, introduces potential interactions between signals, and makes the system harder to debug. If you have five shaped rewards that sometimes conflict (the diversity bonus wants to include a less relevant memory, the confidence bonus wants to exclude it), the system oscillates between strategies. Start with one or two shaped rewards and add more only when you have evidence that the existing ones are insufficient.

Measuring Whether Shaping Works

Reward shaping should accelerate learning, not change the final outcome. Measure this by comparing two systems: one with shaping and one without. Both systems should converge to the same quality level (measured by true user satisfaction), but the shaped system should get there faster. If the shaped system converges to a different quality level, the shaping is distorting the objective and needs adjustment.

Track the shaped reward and the true reward separately. Early in training, the shaped reward should increase faster than the true reward (the system learns the shaped behaviors quickly). Over time, the true reward should catch up as the system translates shaped behaviors into actual user satisfaction. If the shaped reward increases but the true reward stays flat, the shaping is not actually helping users, it is just teaching the system to game the shaped metric.

Shaping in Adaptive Recall

Adaptive Recall's multi-signal scoring system provides implicit reward shaping through its architecture. The combination of similarity, recency, frequency, and confidence scores creates a composite ranking that inherently rewards diversity (different signals promote different memories), recency appropriateness (the recency component handles temporal matching), and confidence alignment (the confidence component handles reliability matching). Because these signals are baked into the scoring equation rather than added as external bonuses, they guide retrieval from the first query without requiring a separate shaping infrastructure.

The ACT-R activation framework that underpins Adaptive Recall's scoring is itself a form of theoretically grounded reward shaping. The activation equation encodes decades of cognitive science research about what makes information useful to humans: recently accessed, frequently needed, contextually connected, and well-corroborated. These factors shape the retrieval ranking from day one, providing the system with strong priors about what constitutes good retrieval while still allowing usage patterns to refine the rankings over time.