How to Ground LLM Responses with Persistent Memory
Before You Start
You need a persistent memory system that stores structured observations with metadata. At minimum, each memory should carry a content field, a confidence score, a timestamp, and entity tags that link it to relevant concepts. Adaptive Recall provides all of this through its MCP or REST API, but the pattern works with any memory store that supports semantic search and metadata filtering. You also need an LLM that accepts system prompts with custom context injection, which covers every major provider.
The core idea is simple: instead of asking the LLM a question and hoping it knows the answer, you first search your memory store for relevant context, then ask the LLM the question while providing the retrieved memories as grounding material. The model generates from the provided context rather than from parametric recall, which dramatically reduces fabrication for any topic the memory store covers.
Step-by-Step Setup
Every memory should be more than a text blob. Include the content itself, a confidence score (how well-corroborated the fact is), the timestamp of last access or update, entity tags (named concepts, people, technologies, projects that the memory relates to), and optionally a source field indicating how the memory was created (user stated, system observed, consolidation output). This metadata is what makes memory grounding more powerful than simple RAG, because you can filter and rank by reliability, not just similarity.
{
"content": "Project uses FastAPI with OAuth2PasswordBearer for auth",
"confidence": 8.7,
"entities": ["FastAPI", "OAuth2", "authentication"],
"lastAccessed": "2026-05-10T14:22:00Z",
"source": "user_stated"
}Before every LLM call, take the user's query and search the memory store for relevant context. Use semantic search to find memories whose content relates to the query topic, then apply cognitive scoring to rank results by a combination of semantic relevance, recency, access frequency, and confidence. The goal is to retrieve the 5 to 15 most relevant, most reliable memories for the current question. Too few memories leave gaps that the model fills with guesses. Too many dilute the signal and consume context window space.
# Pseudocode for pre-query retrieval
def get_grounding_context(user_query, user_id):
memories = memory_store.recall(
query=user_query,
user_id=user_id,
limit=10,
min_confidence=3.0
)
return format_grounding_block(memories)Format retrieved memories into a clear, parseable block that the model can reference during generation. Include the memory content, its confidence score, and a brief source indicator. Ordering matters: put the highest-confidence, most relevant memories first, because models attend more strongly to content near the beginning of their context. Label the block clearly so the model knows what it is and how to use it.
VERIFIED CONTEXT (base your response on these facts):
[Confidence: 9.2] Project uses FastAPI with OAuth2PasswordBearer
for authentication. Tokens stored in Redis with 30-minute TTL.
Source: confirmed by user across multiple sessions.
[Confidence: 7.8] Team migrated from session-based auth to JWT
in January 2026. Migration completed without data loss.
Source: user stated on 2026-01-15.
[Confidence: 5.1] Rate limiting is handled at the nginx level,
not in the application code.
Source: observed in one conversation, not reconfirmed.Add the grounding block to your system prompt alongside clear rules about how the model should use it. The instructions should tell the model to prefer the provided context over its own knowledge when they conflict, to cite the source when making claims based on the context, to qualify statements when the confidence score is below a threshold, and to say "I don't have enough information" rather than guessing when no relevant context exists. Be specific in these instructions, because vague guidance like "try to be accurate" does not change model behavior meaningfully.
system_prompt = f"""You are a technical assistant with access to
verified project context. Follow these rules strictly:
1. When the VERIFIED CONTEXT below contains relevant information,
base your answer on it. Do not contradict it.
2. When the context has confidence above 7.0, state the
information as established fact.
3. When the context has confidence between 3.0 and 7.0,
qualify with "based on earlier discussions" or similar.
4. When no relevant context exists for the question,
say so explicitly rather than guessing.
5. Never fabricate project-specific details like file names,
API endpoints, or configuration values.
{grounding_block}
"""The confidence scores on your memories should influence how the model presents information. High-confidence memories (above 7 or 8, depending on your scale) can be presented as established facts. Medium-confidence memories should be qualified with hedging language that signals uncertainty without undermining usefulness. Low-confidence memories should be used as context clues rather than stated facts, helping the model generate in the right direction without making specific claims that might be wrong. This graduated approach gives users calibrated trust in the output rather than forcing a binary choice between "the AI said it confidently so it must be true" and "the AI might be wrong about everything."
After the model generates a response and the user reacts to it, extract any new facts or corrections from the interaction and store them as memories. If the user confirms a detail, the corresponding memory's confidence increases. If the user corrects a detail, the old memory gets updated and the new fact starts with higher confidence because it came directly from the user. Over time, this creates a flywheel effect where each interaction makes the grounding context more comprehensive and more accurate, which produces better responses, which generate more useful interactions. The system gets better at avoiding hallucinations the more it is used.
Handling Edge Cases
The most important edge case is conflicting context. When multiple retrieved memories disagree about a fact (perhaps reflecting a change over time or a genuine ambiguity), the model needs a clear rule for resolving the conflict. The simplest approach is to prefer the most recent, highest-confidence memory. A more nuanced approach presents both versions to the user with timestamps: "As of January 2026, the project used session-based auth, but as of March 2026, the team migrated to JWT." This approach avoids hallucination by presenting verified historical facts rather than forcing a single answer.
Another important edge case is memory staleness. Memories that have not been accessed or reinforced in a long time may reflect outdated information. The confidence decay mechanism in cognitive scoring handles this naturally: old, unreinforced memories lose activation over time and rank lower in retrieval results. But for critical facts (like production infrastructure details), you may want explicit staleness checks that flag memories older than a certain threshold and prompt the user to reconfirm before the system relies on them.
Partial coverage is the edge case where the memory store has some relevant context but not enough to fully answer the question. In this situation, the model should use the available context as far as it goes and clearly indicate where its knowledge ends and its inference begins. A response like "Based on your project context, the auth layer uses JWT tokens. I do not have specific information about how refresh tokens are handled in your implementation, so the following is general guidance" gives the user a clear boundary between grounded facts and model-generated suggestions.
Measuring the Impact
Track hallucination rates before and after implementing memory grounding by sampling responses and checking them against ground truth. For project-specific questions, ground truth is whatever the user confirms or corrects. For factual questions, ground truth is the content of the memory store. A well-implemented memory grounding system typically reduces hallucination rates by 40% to 70% for questions within the memory store's coverage, with the largest improvements on questions about specific, factual details (names, versions, configurations, dates) and smaller improvements on open-ended analytical questions where the model still needs to synthesize beyond the provided context.
Ground your AI in verified facts. Adaptive Recall provides persistent memory with confidence scoring, knowledge graph grounding, and cognitive retrieval that anchors every response in real context.
Get Started Free