Experience Replay: Game AI Concepts Power Memory

Origins in Game AI

DeepMind's DQN paper (2013) introduced experience replay as a key technique for stabilizing deep RL training. The problem it solved was that consecutive game frames are highly correlated: frame N+1 looks almost identical to frame N. Training on consecutive frames caused the network to overfit to local patterns and forget broader strategies. By storing experiences in a buffer and sampling randomly from it during training, DQN broke the temporal correlations and learned more stable, general policies.

The improvement was dramatic. Without experience replay, DQN trained unstably and performed inconsistently. With experience replay, DQN achieved superhuman performance on dozens of Atari games. The technique proved that reusing past experiences (rather than learning only from the most recent interaction) is fundamental to effective learning.

How It Maps to Retrieval

Retrieval systems face the same correlation problem as game AI. Consecutive queries from the same user tend to be about the same topic. If the system updates its ranking parameters after each query, it overfits to the current topic and loses the ability to rank well for other topics. Experience replay solves this by training on a diverse sample of past queries, ensuring the ranking function works well across the full range of query types.

Each "experience" in a retrieval system is a tuple of (query, context, results served, user feedback, reward). The replay buffer stores thousands of these experiences. During training, the system samples a batch from the buffer, recomputes what the ranking would have produced under the current parameters, and adjusts parameters to improve the expected reward. This decouples learning from serving, which means ranking updates can happen asynchronously without adding latency to live queries.

Prioritized Experience Replay

Not all experiences are equally valuable for learning. An interaction where the system served perfect results and the user was fully satisfied provides little learning signal (the system already knew what to do). An interaction where the system confidently served the wrong results provides a strong learning signal (the system was wrong and needs to update). Prioritized experience replay weights the sampling probability of each experience by its learning value, typically measured by the prediction error (how far the actual reward was from the expected reward).

High-surprise experiences (large prediction errors) get replayed more frequently because they contain the most information about what the system still needs to learn. Low-surprise experiences (small prediction errors) get replayed less frequently because the system already handles them well. This focus on informative experiences accelerates learning significantly compared to uniform sampling.

Buffer Management

The replay buffer has finite capacity, so old experiences must be discarded as new ones arrive. The simplest approach is a circular buffer that overwrites the oldest experiences. More sophisticated approaches maintain a diverse buffer by keeping a mix of recent and old experiences, or by retaining experiences that are still informative (high prediction error) even if they are old.

Buffer capacity involves a tradeoff. A larger buffer provides more diverse training samples but includes older experiences that may not reflect current user behavior. A smaller buffer focuses on recent behavior but provides less diversity and more variance in training. For retrieval systems, a buffer holding 5,000 to 50,000 experiences provides a good balance for most applications.

Experience Replay in Memory Systems

Adaptive Recall's ACT-R activation system implements a form of continuous experience replay without an explicit buffer. The activation equation for each memory considers its entire access history: every time the memory was retrieved, stored as a timestamp. When computing current activation, the equation sums the decayed contributions of all past access events. This is mathematically equivalent to replaying all past experiences with exponential temporal weighting, but it is computed as a single function evaluation rather than requiring batch sampling and gradient updates.

This approach has two advantages over explicit experience replay. First, it requires no buffer management, sampling logic, or batch training pipeline. Second, it naturally handles non-stationarity because the temporal weighting ensures that recent experiences dominate without requiring manual tuning of the decay rate. The cognitive science foundation of ACT-R provides theoretically grounded decay parameters that match empirical human memory performance.