AI agent memory is how an agent keeps track of what it has done and learned across a long, multi-step task, instead of starting each step as if the last one never happened. A language model only ever sees what is inside its current context window. Everything else — the goal it agreed, the decision it made three steps ago, the file it already read — is gone unless something stored it and something brings it back.
Why long tasks break without memory
A long task breaks an agent in a specific way: not with an error, but with drift. The agent re-reads a file it already read, asks a question it already answered, or quietly drops a constraint from the system prompt. Nothing crashed. The context window is simply crowded, and the model is spending its attention on stale tool output instead of the instructions that matter.
The reason is structural. Studies on needle-in-a-haystack style benchmarking have uncovered a pattern Anthropic calls context rot: as the number of tokens in the context window increases, the model’s ability to accurately recall information from that context decreases. A study by Chroma tested 18 models across 8 input lengths and 11 needle positions and found accuracy dropping as input grows, unevenly, and faster when the question and the answer do not share vocabulary.
The cause sits in the transformer architecture itself. Every token attends to every other token, which creates n-squared pairwise relationships for n tokens, so as context grows the model’s ability to capture those relationships stretches thin. Anthropic frames this as an attention budget: every new token depletes it. Context is therefore a finite resource with diminishing marginal returns, and a bigger window does not remove the problem — it only moves it.
Long-horizon tasks are exactly the tasks that run into this wall. Anthropic defines them as tasks that require an agent to maintain coherence, context, and goal-directed behavior over sequences of actions where the token count exceeds the context window — work spanning tens of minutes to multiple hours, like large codebase migrations or comprehensive research projects. For those tasks, memory is not an enhancement. It is the mechanism that keeps the agent coherent at all.
What ai agent memory means
Memory in an agent is not one thing. Production designs split it into four tiers, borrowed loosely from cognitive science. Working memory is the current context window: the active conversation, retrieved documents, and tool outputs. It is bounded, expensive, and gone at the end of the request unless something writes it out. Episodic memory is what happened in past sessions — time-ordered and usually retrieved by recency or similarity. Semantic memory is durable facts: user preferences, account details, learned entities and their relationships. How a store handles a contradicted fact — whether it overwrites, versions, or flags it — is a design decision for that store, not a property of semantic memory itself. Procedural memory is learned behavior: which tool sequence worked last time. It is the least standardized tier, and most systems do not handle it explicitly yet. Memory is additive to retrieval-augmented generation, not a substitute: memory stores personal facts, while retrieval-augmented generation stores external knowledge such as documents and codebases.
Short-term memory: the context window and conversation history
Short-term memory is the memory every agent already has: the conversation thread itself. In LangChain v1, short-term memory is the thread saved by a checkpointer passed to create_agent and keyed by a thread_id; long-term memory is anything that should carry across threads, kept in a store or an external memory service. The old memory classes, such as ConversationBufferMemory, are deprecated since version 0.3.1 and due to be removed in 2.0.
The simplest thread memory keeps the whole conversation, but that is also the most dangerous for production use: every exchange is injected into every prompt, so token costs grow with every turn. The window pattern keeps only the last few exchanges. The summary pattern replaces older messages with a model-written summary once a trigger is hit. LangChain ships this as SummarizationMiddleware.
A minimal working setup shows the shape. It needs the langchain and langgraph packages installed and a model API key set in the environment:
from langchain.agents import create_agent
from langgraph.checkpoint.memory import InMemorySaver
agent = create_agent(
model="gpt-4o",
tools=[],
checkpointer=InMemorySaver(),
)
result = agent.invoke(
{"messages": [{"role": "user", "content": "Remember that I prefer concise summaries."}]},
config={"configurable": {"thread_id": "user-123"}},
)Reusing the same thread_id gives the agent the full history of that thread. But a thread is still bounded by the window: when history outgrows it, something has to compress or move it out.
Long-term memory: summaries, vector stores, and episodic stores
Long-term memory lives outside the thread. A new thread starts empty, while a checkpointed thread can be resumed with its history intact — so long-term memory is what carries facts across threads. The three main approaches are summaries, vector retrieval, and structured stores.
Compaction is the summary approach, and Anthropic calls it typically the first lever for long-term coherence. It takes a conversation nearing the context window limit, summarizes its contents, and reinitiates a new context window with the summary. In Claude Code, the model preserves architectural decisions, unresolved bugs, and implementation details while discarding redundant tool outputs, then continues with the compressed context plus the five most recently accessed files. The art lies in choosing what to keep: overly aggressive compaction can lose subtle but critical context whose importance only becomes apparent later. Clearing tool results is the lightest-touch form of compaction.
Vector stores handle recall by meaning rather than by order. Past exchanges or facts are embedded and retrieved by semantic similarity to the current query. Similarity is a retrieval mechanism, not a memory type: it can return an episodic exchange or a durable fact, and a close match can still be irrelevant or incomplete.
Structured stores keep facts as documents under a namespace. In LangChain, long-term memory is a store of JSON documents grouped by namespace — usually the user ID — that tools read and write through runtime.store. A stored memory record looks like this:
{
"namespace": ["users", "user-123", "preferences"],
"key": "summary-style",
"value": {
"preference": "concise summaries",
"set_on": "2026-11-10"
}
}Named systems package these layers differently. Mem0 is a memory layer you attach to an existing agent: it extracts facts from conversations with a language model, deduplicates them against what is already stored, and retrieves relevant memories by semantic similarity on future turns. Letta, formerly MemGPT and born from a UC Berkeley research paper, treats the language model’s context window like an operating system’s virtual memory: a small core memory that is always in context and editable by the model itself, plus an unbounded archival memory backed by Postgres and pgvector that the agent searches on demand. Zep builds a temporal knowledge graph from conversations and combines graph traversal with vector search, which handles time-aware queries better than pure vector stores. A-MEM, a NeurIPS 2025 research architecture, organizes memories as linked Zettelkasten-style notes; it is a research implementation, not a hosted service.
How ai agent memory retrieval works in practice
Retrieval is a cycle with three decisions: what gets written, what gets recalled, and when. Writing can happen on every step, at milestones, or on demand. Mem0’s documented pipeline has two stages: a language-model call extracts candidate facts from a new exchange, then a decision step compares those candidates against what is already stored.
Recall is triggered by the current turn. A vector store retrieves by semantic similarity to the query; a structured store is read by namespace and key; a note file is loaded when the agent checks it. What comes back re-enters the context window as ordinary tokens, where it competes for the same attention budget as everything else. That is why Anthropic’s guiding principle matters here too: find the smallest set of high-signal tokens that maximize the likelihood of your desired outcome. Retrieving too much memory is its own failure mode, crowding out the current task with old context.
Anthropic also describes a just-in-time style of retrieval that avoids pre-loading everything. Instead of processing all relevant data up front, the agent keeps lightweight identifiers — file paths, stored queries, web links — and dynamically loads data into context at runtime using tools. The same idea applies to memory: a pointer to a stored note is cheap to keep; the full note is loaded only when the agent decides it needs it.
One long task, traced through the memory layers
Take a concrete task, presented here as an illustrative walkthrough: an agent is asked to research a topic and draft a report over many turns, with the user returning between sessions to steer it. The working memory holds the current turn: the latest instruction, the section being drafted, and the tool outputs in front of it. As the conversation grows, summary middleware compresses the early turns into a summary the agent carries forward.
When the user comes back the next day and starts a new thread, that thread is empty, so long-term memory takes over. Resuming the checkpointed thread instead would restore its history — elapsed time alone does not erase a thread. A semantic store returns the standing facts: the report’s audience, the user’s preference for concise summaries, the sources already ruled out. An episodic store returns what happened: last session ended after the agent drafted the first two sections and flagged one claim as unverified. Structured note-taking covers the same ground in simpler systems: the agent wrote its progress to a notes file outside the window, and after the context reset it reads its own notes and continues. Anthropic reports the same pattern in a games setting, where its Pokemon-playing agent reads its own notes after context resets and continues multi-hour training sequences, maintaining a tally of 1,234 steps of training in Route 1, with Pikachu gaining 8 levels toward a target of 10. The walkthrough also shows a miss and its recovery: one morning the agent needs the report’s target word count, a fact nobody wrote down. The stores return nothing, so the agent checks its saved notes, finds no record, and asks the user before continuing — a recovery, not a guess.
Failure modes: stale memories, misses, and bloat
Memory fails in four recognizable ways. The first is staleness: a stored fact is contradicted by a newer one, and the agent acts on the old version. This is why write-time deduplication matters: a store needs an explicit policy for contradicted facts, and that policy has to be designed and verified for the specific store rather than assumed.
The second is the retrieval miss. A vector search returns nothing relevant, or returns something similar but wrong. The Chroma result belongs to a different mechanism — recalling information already inside a context window, where accuracy falls as the window grows — not to vector search over stored memory. A retrieval miss is its own failure mode: the store holds the fact, but the query does not retrieve it. A miss is recoverable if the agent treats absence as absence and re-derives or asks, and dangerous if it fills the gap with a confident guess.
The third is bloat. Too much memory in context is as harmful as too little: context rot means recall accuracy falls as the window fills, so an agent that retrieves generously can end up less accurate than one that retrieves sparingly. Compaction has the same risk in reverse: an aggressive summary can discard a detail whose importance only becomes apparent later.
The fourth is privacy and ownership. A memory that persists across sessions is a record of what the user said, decided, and preferred. Namespaces keyed by user ID exist for a reason: memory must be scoped to the person it belongs to, and a stored fact needs an expiry or a review path, not just a write path.
Choosing a memory approach for your agent
The choice starts with the task, not the tool. Anthropic’s guidance maps the three long-horizon techniques to task shapes: compaction maintains conversational flow for tasks requiring extensive back-and-forth; note-taking excels for iterative development with clear milestones; multi-agent architectures handle complex research where parallel exploration pays off. In a sub-agent design, each sub-agent explores with a clean context window and returns only a distilled summary — often 1,000 to 2,000 tokens after exploring tens of thousands — while the lead agent keeps the plan in external memory, because context exceeding 200,000 tokens will be truncated. Anthropic’s multi-agent research system, with Claude Opus 4 as lead and Claude Sonnet 4 subagents, outperformed single-agent Claude Opus 4 by 90.2 percent on their internal research evaluation.
A practical checklist follows from the evidence. Start with summaries: compaction and note-taking are the documented starting options, and Anthropic maps them to clear task shapes — compaction for back-and-forth work, note-taking for iterative development with milestones. A notes file the agent writes and reads back needs no extra service to set up. Add vector retrieval only when history outgrows the window and recall by meaning beats recall by order. Decide the write policy up front: what gets stored, who can overwrite it, and when a stored fact expires. Keep retrieved memory small and high-signal. And give the agent a way to say it does not remember, rather than letting a retrieval miss become a confident invention.
The difference between an agent that remembers and one that merely runs a long loop is not the size of its context window. It is whether anything outside the window keeps the goal, the decisions, and the facts, and whether the agent can bring exactly the right piece back at the moment it needs it.
