Is RL Overkill for Search Ranking

What "Full RL" Actually Requires

Full reinforcement learning for search ranking means training a neural network to optimize result ordering using policy gradient methods (REINFORCE, PPO, or actor-critic architectures). The system learns a policy that maps query features and result features to an optimal ranking. This requires a reward model (trained on human preferences or behavioral data), a policy network (the ranking model itself), an experience collection pipeline (logging interactions for training), a training infrastructure (GPUs for policy updates), and a deployment pipeline (serving the learned policy in production).

The engineering cost is substantial. Building and maintaining this stack requires dedicated ML engineers who understand RL training stability, reward model calibration, and policy deployment. The data requirements are significant: policy gradient methods typically need millions of interactions to converge to a stable policy. The operational complexity is high: you need to monitor for policy degradation, reward hacking, and distribution shift. And the debugging is hard: when the system serves bad results, tracing the cause through a neural policy is much harder than inspecting a weighted scoring formula.

When Full RL Is Worth It

Full RL justifies its complexity at the scale where marginal ranking improvements translate to significant business value. Google uses RL for search ranking because they serve billions of queries per day, and a 0.1% improvement in click-through rate affects millions of users. Spotify uses RL for playlist recommendations because a 0.5% improvement in listening time translates to measurable retention improvements across hundreds of millions of users.

At this scale, the engineering investment in full RL is amortized across so many interactions that the per-query cost of the infrastructure is negligible. The millions of daily interactions provide enough data for policy gradient methods to converge quickly. And the business value of even small improvements is large enough to justify a dedicated ML team.

For most AI applications, the traffic volume is orders of magnitude smaller. A memory system serving 1,000 queries per day does not generate enough data for policy gradient methods to converge reliably. A retrieval system with 10,000 users does not generate enough revenue from marginal improvements to fund a dedicated RL team. At this scale, simpler approaches deliver nearly the same quality improvement at a fraction of the cost.

Simpler Approaches That Work

Weighted scoring with usage-driven adjustment. Define a ranking function as a weighted sum of signals (similarity, recency, frequency, confidence). Track which memories are useful and adjust the weights gradually based on observed patterns. This requires no ML infrastructure, no training pipeline, and no GPU. The weights can be adjusted with a few lines of code based on aggregated feedback metrics. For most applications, this provides 60-70% of the ranking improvement that full RL would achieve.

Multi-armed bandits. Define 3-5 ranking configurations (different weight combinations) and use a bandit algorithm to discover which works best. Thompson sampling converges quickly (hundreds of interactions, not millions), requires minimal infrastructure (just a Beta distribution per arm), and provides principled exploration. This adds 10-15% improvement on top of the best fixed weighting, bringing the total to 70-80% of what full RL would achieve.

ACT-R cognitive scoring. Use activation equations from cognitive science that model how human memory prioritizes information. The equations factor in recency, frequency, contextual associations, and confidence, producing rankings that feel natural to users. The equations update automatically based on access patterns, requiring no explicit training or reward engineering. This is the approach Adaptive Recall uses, and it provides ranking quality comparable to what simple RL implementations achieve, without any RL infrastructure.

Cross-encoder reranking. Use a pre-trained cross-encoder model to rerank the top results from a simpler retrieval stage. The cross-encoder evaluates each (query, result) pair for relevance, producing high-quality rankings with no application-specific training. This adds 50-200ms of latency per query but significantly improves precision at the top of the result list.

The Complexity Ladder

Think of adaptive ranking approaches as a ladder of increasing complexity, where each rung provides diminishing marginal improvement over the previous one:

Rung 1: Static weighted scoring. Set weights once, never change them. This is the baseline that most retrieval systems ship with. Zero engineering cost beyond the initial setup. Performance is fixed and cannot improve with usage.

Rung 2: Usage-driven weight adjustment. Adjust weights based on aggregated feedback metrics. Requires basic instrumentation (logging retrievals and outcomes) and periodic analysis. Improves over the baseline by 20-40% in retrieval quality metrics after a few weeks of data collection.

Rung 3: Cognitive scoring or bandits. Use principled frameworks (ACT-R, Thompson sampling) that adapt automatically. Requires more sophisticated instrumentation but no ML training infrastructure. Improves over rung 2 by 10-20%.

Rung 4: Full RL with neural policies. Train a neural ranking model with policy gradients. Requires dedicated ML infrastructure, millions of interactions, and ongoing operational monitoring. Improves over rung 3 by 5-15%, but only at sufficient scale.

Most applications get the best return on investment at rung 2 or rung 3. The jump from rung 3 to rung 4 provides the smallest marginal improvement at the highest marginal cost. Only climb to rung 4 when you have exhausted the improvements available at lower rungs and have the scale to justify the investment.

The Pragmatic Choice

Adaptive Recall uses rung 3 (cognitive scoring through ACT-R) because it provides the best quality-to-complexity ratio for memory retrieval applications. The ACT-R equations are mathematically principled, computationally efficient (no GPU required), automatically adaptive (no manual tuning), and grounded in decades of cognitive science validation. They deliver ranking quality that matches or exceeds simple RL implementations while requiring zero ML infrastructure, zero training data, and zero dedicated ML engineering.

This is the pragmatic choice for any application that does not operate at Google-scale traffic volumes. You get adaptive ranking that improves with every interaction, without the engineering overhead of policy gradients, reward models, and training pipelines. If your application eventually reaches the scale where full RL would provide a meaningful uplift, you can add it on top of the cognitive scoring foundation rather than replacing it.