Working Memory vs Long-Term Memory for AI Agents

Working Memory in Agent Systems

Working memory for an LLM-based agent has a direct physical analog: the context window. Everything the LLM can "see" at any given moment, the system prompt, conversation history, tool results, and any injected context, is its working memory. This working memory has a hard capacity limit (the token limit of the model) and everything in it is immediately accessible without a retrieval step.

In practice, agent working memory extends slightly beyond the raw context window. Many agent frameworks maintain state variables that track the current plan, completed steps, and intermediate results outside the conversation. These variables are injected into the LLM prompt at each reasoning step, effectively expanding working memory beyond the conversation itself. LangGraph's graph state, CrewAI's task context, and AutoGen's message history all serve as extended working memory.

The limitation of working memory is capacity. A 200,000-token context window sounds large, but a complex agent task can consume it quickly. Each tool call and result adds hundreds or thousands of tokens. Intermediate reasoning, especially when the agent explores multiple hypotheses, adds more. After 20 to 30 tool calls, the working memory is half full and the LLM starts losing track of earlier findings because they have scrolled far up in the context. This is the "lost in the middle" problem applied to agent execution: the LLM attends more to recent messages and may forget important findings from early in the session. The lost-in-the-middle guide covers the research behind this effect and mitigation strategies.

The practical consequence is that agents on long tasks (15 or more steps) need active working memory management. The most common technique is context summarization: after every N steps, the agent summarizes the conversation so far, keeping key findings and the current plan while discarding verbose tool outputs and intermediate reasoning. This frees up context window capacity for the remaining steps. The trade-off is that summarization is lossy, specific details that were in the original messages may be compressed or dropped, which is why the most important findings should be promoted to long-term memory before they are at risk of being summarized away.

Long-Term Memory for Agent Systems

Long-term memory is a separate store that the agent explicitly writes to and reads from. It is not part of the LLM context window; the agent has to take an action (call a tool) to store or retrieve information. This extra step is the trade-off for persistence: information in long-term memory survives indefinitely, is searchable across all sessions, and is not constrained by the context window size.

Long-term memory serves three purposes in agent systems. First, it carries knowledge across sessions. An agent that discovered the root cause of a bug yesterday can recall that knowledge today without re-investigating. Second, it offloads working memory. When the context window is filling up, the agent can store important findings in long-term memory and then summarize the conversation to free up working memory space, knowing the details are safely persisted. Third, it enables multi-agent collaboration. Multiple agents can write to and read from the same long-term memory store, sharing knowledge without direct communication.

The quality of long-term memory depends entirely on what gets stored. A memory store full of raw conversation snippets is barely better than no memory at all, because retrieval returns context-free fragments that require the original conversation to interpret. A memory store of clean, self-contained facts with metadata (timestamps, confidence, entity tags) is a powerful knowledge base that any agent can query and get immediately useful results. The difference between these two outcomes is the promotion policy, which is the hardest design problem in agent memory.

The Promotion Problem

The most critical design decision in agent memory is the promotion policy: what moves from working memory to long-term storage, and when. Four approaches exist, ordered from least to most effective.

Store everything. When the session ends, dump the entire conversation history into long-term memory. This is easy to implement but produces a memory store full of noise. Intermediate reasoning, failed hypotheses, tool call formatting, and clarification exchanges all get stored alongside the actual findings. Retrieval quality degrades rapidly because useful memories are outnumbered by noise. At scale, store-everything approaches can produce memory stores with 95% noise and 5% signal, making retrieval essentially random.

Store summaries. At the end of the session, use the LLM to summarize what was accomplished and store the summary. This is better than storing everything because the summary is concise and focused on outcomes. The limitation is that summaries lose specific details (exact metric values, specific command sequences, precise error messages) that may be needed in future sessions. A summary that says "the database query was slow because of a missing index" is useful, but a summary that specifies "adding a B-tree index on orders.customer_id reduced the query from 4.2 seconds to 12 milliseconds" is far more useful when the same issue arises again.

Store at decision points. Instrument the agent to store a memory at specific moments: after discovering a fact, after completing a task, after making a key decision. Each memory is self-contained and specific. This produces the highest-quality memory store but requires careful instrumentation of the agent loop. You need to identify which moments are worth storing and add explicit store calls at those points. The risk is under-storing: if you miss a decision point, the knowledge is lost when the session ends.

Continuous selective storage. The agent stores observations throughout execution but is trained (through its system prompt) to be selective about what is worth storing. It stores facts, outcomes, and surprises. It does not store intermediate reasoning, expected results, or information it retrieved from memory (which is already stored). This combines the coverage of continuous storage with the quality of selective storage. The challenge is that the model's judgment about what is worth storing is imperfect, so some noise still gets through, but far less than the store-everything approach.

Adaptive Recall works best with the third and fourth approaches. The store tool is designed for agents to call during execution when they discover something worth remembering. The metadata system (confidence, tags, entity extraction) ensures that each stored memory is well-structured and retrievable. The consolidation process handles cleanup by merging redundant memories and fading low-value ones, which means even imperfect promotion policies improve over time as the lifecycle system refines the memory store.

Common Mistakes That Degrade Agent Memory

Several patterns consistently cause agent memory to become less useful over time rather than more useful. Recognizing them helps you avoid building a system that starts strong and gradually degrades.

Storing opinions as facts. When the agent makes a judgment call ("this API is probably deprecated"), storing that judgment as a fact causes problems later when the agent retrieves it and treats it as established truth. The fix is to store the judgment with a confidence score and the reasoning behind it, so future retrievals carry the uncertainty forward. Adaptive Recall's confidence scoring handles this natively: memories stored with lower confidence are weighted lower in retrieval results.

Never updating stored memories. If the agent stored that "the billing API endpoint is api.example.com/v2/billing" six months ago and the endpoint has since changed to v3, the stale memory actively causes errors. Long-term memory needs a mechanism for updating or invalidating outdated information. The simplest approach is to treat contradictions as update signals: when the agent encounters information that contradicts a stored memory, it should update the stored memory with the new information and a timestamp. The memory lifecycle pillar covers strategies for keeping stored knowledge current.

Storing too much context. A memory that says "When I was debugging the login issue for customer #4521 on Tuesday, I discovered that the session token was expired because the JWT signing key had been rotated" contains useful information (JWT key rotation causes session expiration) wrapped in irrelevant context (customer number, day of week). The irrelevant context hurts retrieval because a future query about JWT key rotation may not mention customer numbers or Tuesdays. Strip the context and store the generalized fact: "JWT signing key rotation invalidates existing session tokens. Symptom: users get logged out simultaneously after key rotation."

Not categorizing memories. A flat memory store where every memory is the same type becomes increasingly hard to search as it grows. When the agent searches for "API rate limit," it should be able to distinguish between procedural knowledge (how to handle rate limits), factual knowledge (what the rate limits are for specific APIs), and project knowledge (which of our services hit rate limits most often). Categorization, using types like general_knowledge, learned_procedure, and work_project, makes retrieval dramatically more precise.

Practical Architecture: Three-Tier Memory

The typical production architecture has three tiers. The bottom tier is the LLM context window: small, fast, volatile. This holds the current message, the current plan, and the most recent tool results. The middle tier is an extended scratchpad (a JSON state object or a short conversation buffer): medium-sized, persisted for the duration of the task, discarded when the task completes. This holds the full plan, all step results, and running notes. The top tier is long-term memory: large, persistent, searchable. This holds all accumulated knowledge from all sessions.

Information flows upward through promotion. A finding starts in the context window (the agent notices it in a tool result), gets recorded in the scratchpad (as part of the step results), and gets promoted to long-term memory (when the agent recognizes it as a durable fact worth remembering). Information flows downward through retrieval. Before starting a new task, the agent queries long-term memory for relevant context, which is loaded into the scratchpad and injected into the context window.

The three-tier architecture solves the fundamental tension in agent memory: working memory needs to be small and focused (for LLM attention quality), but the agent's total knowledge needs to be large and comprehensive (for handling diverse tasks). By separating the tiers and managing the flow between them, you get both: a focused working memory that changes with each step, backed by a comprehensive long-term memory that accumulates over the agent's entire lifetime.

For implementation specifics, the guide to adding long-term memory walks through the integration of persistent storage into an existing agent, and the file vs database vs hybrid architecture comparison helps you choose the right storage backend for your scale and access patterns.