RLHF Beyond LLMs: Human Feedback for Any System

What RLHF Actually Is

RLHF is a training methodology with three stages. First, collect human feedback on system outputs, typically as pairwise comparisons ("output A is better than output B for this input"). Second, train a reward model that predicts human preferences from output features. Third, use the reward model to guide the system's behavior through reinforcement learning, optimizing the system to produce outputs the reward model scores highly.

For LLM training, these stages involve human raters comparing model responses, a neural network learning to predict which response humans prefer, and a policy gradient algorithm adjusting the model weights to produce preferred responses. The same three stages can be instantiated for any system where humans have opinions about output quality.

RLHF for Retrieval Systems

A retrieval system returns ranked results for a query. Human feedback tells the system which rankings are better. This feedback can be explicit (a user rates results as helpful or unhelpful) or implicit (the user clicks on the third result, suggesting the first two were not useful).

The reward model for retrieval learns which ranking characteristics humans prefer. Do users prefer results that are more recent? More similar to the query? Shorter and more specific? From higher-confidence sources? The reward model captures these preferences as a function of result features, and the retrieval system uses this function to rank results in a way that aligns with human expectations.

This is particularly valuable when the "right" ranking depends on context. Academic queries might benefit from comprehensive, authoritative results. Quick lookups might benefit from concise, recent results. The reward model can learn these context-dependent preferences if the feedback data includes enough diversity.

RLHF for Memory Systems

Memory systems decide what to store, what to retrieve, and what to surface as context. Each of these decisions can benefit from human feedback. If a user says "no, that's outdated" in response to an injected memory, that is feedback on the retrieval decision. If a user finds an answer quickly because the right memory was surfaced, that is positive feedback on the storage and retrieval decisions.

The challenge for memory systems is that the feedback is indirect. Users interact with the model's response, not with the individual memories that informed it. Attributing the quality of the response to specific memories requires inference: was the response good because the right memory was injected, or would it have been good anyway? Was the response bad because the wrong memory was injected, or because the model misused the context?

Adaptive Recall handles this attribution through its activation dynamics. Memories that are consistently retrieved in sessions with positive outcomes gain activation over time. Memories that are retrieved in sessions with negative outcomes, or that are consistently ignored by the model, lose activation. The system does not need perfect attribution because the statistical patterns over many interactions reveal which memories are genuinely useful.

Implicit vs Explicit Feedback

Explicit feedback (ratings, thumbs up/down, comparisons) is high quality but sparse. Most users never rate results. Implicit feedback (clicks, dwell time, reformulations, task completion) is noisy but abundant. Production systems need to use both.

The practical approach is to use explicit feedback for reward model training (calibrating what "good" means) and implicit feedback for ongoing learning (adapting to changing patterns). When a user explicitly rates a retrieval result, that data point has high weight in the reward model. When a user implicitly signals through behavior, that data point has lower weight but contributes to the ongoing adaptation.

Adaptive Recall uses implicit feedback exclusively, through access patterns. The frequency and recency of memory access are natural implicit signals that require no user action. A memory that is retrieved across many sessions is implicitly validated by the pattern of use. A memory that is never retrieved is implicitly demoted by the absence of use. This zero-friction approach to feedback collection means the system learns from every interaction without asking users to rate anything.

When RLHF Is Worth the Effort

RLHF adds significant complexity to a system. The feedback collection infrastructure, reward model training, and policy optimization pipeline all require engineering investment. The payoff is justified when the gap between what your system currently optimizes (cosine similarity, recency, etc.) and what users actually value is large enough to produce measurable quality differences.

If your retrieval system already produces good results and user satisfaction is high, RLHF may not be worth the investment. If users frequently complain about irrelevant results, if engagement metrics are flat despite content improvements, or if the "right" ranking depends on context in ways your current system does not capture, RLHF can close the gap between what the system optimizes and what users need.