Home » AI Cost Optimization » Memory to Reduce Tokens

How to Use Persistent Memory to Cut Token Usage

Persistent memory replaces the most expensive parts of your AI context window with curated, compact information stored outside the request. Instead of resending 5,000 tokens of conversation history every turn, you recall 300 tokens of key facts. Instead of retrieving 2,500 tokens of RAG chunks, you recall 400 tokens of relevant knowledge. The result is a 50 to 80 percent reduction in input tokens per request, which translates directly to a 50 to 80 percent reduction in the largest component of your API bill.

Before You Start

You need a persistent memory service (Adaptive Recall, Mem0, Zep, or a custom implementation), per-request token logging so you can measure before and after, and a multi-turn or knowledge-intensive application where context tokens dominate the bill. Single-turn applications with minimal context have less to optimize, though even they benefit from memory-based system prompt reduction.

Step-by-Step Implementation

Step 1: Measure current token distribution.
Before making changes, measure exactly where your input tokens go. Log a representative sample of 1,000 requests and break down each into: system prompt tokens, conversation history tokens, RAG retrieval tokens, tool definition tokens, and user message tokens. Calculate the average and p95 for each component. In most applications, conversation history and RAG chunks together represent 50 to 70 percent of input tokens. These are the primary targets for memory-based reduction.
Step 2: Replace conversation history with memory summaries.
This is the highest-impact change. Instead of appending every message to a growing conversation history that is resent with each turn, store key information after each interaction and recall only what is relevant to the current turn. After each user turn, store observations about what was discussed, what the user asked for, what was decided, and any facts learned. On the next turn, recall relevant memories instead of including the full transcript.
# Before: conversation history grows every turn messages = [ {"role": "system", "content": system_prompt}, # Full history resent every time {"role": "user", "content": "My name is Sarah and I need help..."}, {"role": "assistant", "content": "Hi Sarah, I'd be happy to help..."}, {"role": "user", "content": "I tried resetting my password but..."}, {"role": "assistant", "content": "I see. Let me look into..."}, # ... 10 more turns, 4000+ tokens {"role": "user", "content": current_message} ] # Average input: 8,500 tokens at turn 10 # After: memory replaces history # Store after each turn: memory.store("User Sarah is troubleshooting password reset. " "Tried standard reset flow, got error code 403. " "Account is enterprise tier, SSO enabled.") # Recall before each turn: context = memory.recall(current_message) messages = [ {"role": "system", "content": system_prompt}, {"role": "system", "content": f"Relevant context:\n{context}"}, {"role": "user", "content": current_message} ] # Average input: 2,800 tokens at turn 10
Step 3: Replace generic RAG with memory recall.
Standard RAG retrieves 3 to 5 document chunks on every request, whether or not they are needed. A memory-augmented approach first checks if the question can be answered from stored knowledge (memories from previous interactions, curated domain knowledge, or previously processed documents), and only falls back to RAG retrieval when memory does not have a confident answer. For applications where users ask similar questions repeatedly, this eliminates 40 to 60 percent of RAG retrievals entirely. For the remaining retrievals, memory can narrow the search scope (specifying which document sections are relevant), reducing the number of chunks retrieved from 5 to 1 or 2.
Step 4: Slim the system prompt.
Many system prompts grow over time as developers add edge case handling, persona details, formatting instructions, and context-specific rules. Review your system prompt and identify content that is either rarely needed (edge case instructions that apply to less than 5 percent of requests) or user-specific (preferences, history, behavioral customizations). Move these to memory, where they are recalled only when relevant. A system prompt that was 2,500 tokens might be reduced to 800 tokens of core instructions, with the remaining 1,700 tokens of situational content recalled from memory only when the current request warrants it. Combined with prompt caching on the remaining 800 tokens, this reduces the effective cost of the system prompt by over 95 percent.
Step 5: Measure the reduction.
After implementing memory-based optimizations, run the same measurement from Step 1 on a new sample of 1,000 requests. Compare per-request token counts for each component: system prompt, history (now memory recall), RAG (now targeted recall), tools, and user message. Calculate the percentage reduction in total input tokens and translate it to dollar savings using your API pricing. Track these metrics continuously to ensure the reduction is maintained and to catch regressions. A properly implemented memory layer typically shows: conversation history tokens reduced by 80 to 90 percent, RAG tokens reduced by 40 to 60 percent, and system prompt tokens reduced by 50 to 70 percent.

The Economics of Memory vs Context

The cost math is straightforward. Context tokens are processed by the LLM on every request at $3 per million (Sonnet input pricing). Memory tokens are stored once and recalled as needed, with the memory service cost amortized across all requests that benefit from the stored knowledge. A memory that is stored once (one API call to extract and store the information) and recalled 100 times (100 requests that use it instead of raw context) has a per-use cost that approaches zero while eliminating thousands of context tokens from each of those 100 requests.

Consider a specific example. A customer support bot processes 10,000 conversations per day, averaging 6 turns each. Without memory, each turn includes 4,000 tokens of conversation history (average across turns). That is 4,000 tokens per turn, 6 turns per conversation, 10,000 conversations per day = 240 million history tokens per day at $3 per million = $720 per day just for conversation history. With memory, each turn includes 400 tokens of recalled context instead. That is 400 tokens per turn, same volume = 24 million tokens per day = $72 per day. The memory service that stores and recalls these summaries costs perhaps $50 per day. Net savings: $598 per day, or roughly $18,000 per month, from a single optimization.

Quality improves too: Memory-based context is not just cheaper; it is better. A 400-token memory recall contains curated, relevant information: the user's name, their problem, what has been tried, what was decided. A 4,000-token raw history contains all of that buried in greetings, filler, tangents, and repeated information. The model works more accurately with curated context because there is less noise to reason through.

Start cutting token costs today. Adaptive Recall stores conversation knowledge, domain facts, and user context, then delivers exactly the right information on each recall. The cognitive scoring ensures that relevant, recent, high-confidence memories surface first, so your context is both smaller and better.

Start Free Trial

Related Articles