Every interesting thing an AI agent does happens between model calls. The call itself is stateless: the model sees a prompt, produces a reply, and forgets everything. So ai agent memory is the engineering discipline of deciding what the agent remembers, how it stores it, and what gets assembled into each prompt. Memory is also what enterprises want most from agents: 46.9% of organizations cite it as the top capability they expect, according to S&P Global Market Intelligence (451 Research, April 2026) — even as the underlying techniques remain early-stage. This guide explains how AI agents handle memory and context, from the three memory layers to the read/write loop that ties them together.
Why AI agents need memory at all
An agent that forgets everything each turn cannot do multi-step work. It cannot remember your preferences, what it tried yesterday, or the plan it is halfway through executing. Memory, in the agent sense, is the information the agent retains across turns and sessions — both what sits inside the assembled context right now (pinned facts, recent history) and what lives in external stores (conversation archives, fact databases, knowledge graphs) waiting to be retrieved. Storage and visibility are different things: a fact can be durably stored yet invisible to the model until the retrieval step surfaces it. Concretely, that retained information includes conversation history, extracted facts, user preferences, and learned procedures.
Two hard constraints shape every memory design. First, context windows are finite. Second, they are billed per token: OpenAI lists gpt-6.1-sol at $2.00 input / $10.00 output per million tokens, Anthropic prices Claude Haiku 5.5 at $0.10 / $0.50 per million up to 100K tokens of prompt (then 5x above that threshold), and Google tiers Gemini 2.5 Pro from $1.25 / $10.00 per million up to 200K to $2.50 / $15.00 above it. For agents that work over long histories, replaying full conversations can become the dominant token cost — one 2026 study measured ~101,600-token conversations against a memory pipeline that compressed them to ~2,909 tokens. Memory engineering is largely the art of spending that token budget wisely.
The three layers of agent memory
Many agent frameworks use a version of the same layered picture — working, short-term, long-term — though the implementations differ enough that it is better understood as a useful conceptual model than a universal architecture. It is often described with a computer-memory analogy: RAM, disk cache, and cold storage.
Working memory is what is in the prompt right now. It is the only thing the model can directly reason over, and it is precious: everything here costs tokens on every call.
Short-term memory is the recent conversation — a rolling buffer of the last N turns, giving continuity within a session. Short-term describes scope, not guaranteed ephemerality: a purely in-memory buffer evaporates when the process ends, but thread-scoped persistence (LangGraph checkpointers) can outlive the session, and overflow in systems like LlamaIndex is flushed into long-term blocks rather than discarded.
Long-term memory persists across sessions: facts, preferences, and learned procedures stored in databases, vector stores, or knowledge graphs, fetched on demand.
Within long-term memory, frameworks borrow a distinction from cognitive science. LangMem, LangChain’s memory SDK, organizes memory into semantic memory (facts about the world), episodic memory (things that happened), and procedural memory (how to do things). The labels matter because each type is written and read differently: you update a fact, you append an episode, you refine a procedure.
Concrete frameworks map onto these layers in their own vocabulary. In LangGraph, checkpointers persist a thread’s graph state — short-term, thread-scoped memory keyed by thread_id — while stores persist application data across threads: long-term, user- or entity-namespaced memory. In LlamaIndex, the Memory class keeps a short-term FIFO chat history (default token_limit 30,000) and flushes overflow into long-term memory blocks. Letta describes the same stack as core memory (“RAM”, pinned in context), recall memory (“disk cache”, searchable history), and archival memory (“cold storage”, long-term facts in a vector database).
How context gets assembled before every model call
The hottest path in any agent is the prompt-construction loop that runs before every single model call. A typical assembly looks like this:
{
"system": "You are a research assistant. Answer concisely and cite sources.",
"pinned_memory": [
{"label": "user_profile", "value": "Prefers concise answers with sources."}
],
"retrieved_facts": [
{"fact": "User's team deploys on AWS.", "learned_at": "2026-10-20"}
],
"tool_definitions": ["search_docs(query: string)", "read_url(url: string)"],
"recent_history": [
{"role": "user", "content": "How should we evaluate vector databases?"},
{"role": "assistant", "content": "Start with recall@k measured on your own queries..."}
],
"token_budget": {"total": 30000, "used": 1840}
}Each slot has a job. Pinned memory holds facts the agent needs constantly, with no retrieval step. Retrieved facts are fetched on demand for this specific turn. Tool definitions tell the model what it can do. Recent history provides conversational continuity. And the token budget governs the whole assembly: when it is exceeded, something gets truncated — which is why frameworks define truncation rules up front (LlamaIndex, for example, truncates long-term memory blocks by priority).
Order matters inside that assembly, not just size. Liu et al.’s “Lost in the Middle” study (Stanford, 2023) showed that language models attend best to the beginning and end of long contexts, with accuracy sagging in the middle — moving the answer-bearing document from the best to the worst position cost GPT-3.5 roughly 20 accuracy points on multi-document QA. Put the current instruction and the most load-bearing facts where the model actually looks.
Summarization: compressing what the agent already said
When the short-term buffer fills, the standard move is to compress it: a rolling summary distills the conversation so far, discarding verbatim turns but keeping the gist. LangMem ships a summarize_messages utility for exactly this kind of short-term compression.
Summarization is lossy by design: a summary is a bet about what the future will need, and every bet can be wrong. The broader risk of information loss across a whole memory system is documented independently of summarization — the LongMemEval benchmark (Wu et al., ICLR 2025) found that ChatGPT with GPT-4o scored 0.5773 on long-term memory questions versus 0.9184 with the full history available offline, a 37% gap for the entire memory pipeline rather than summarization alone, and observed the assistant tending to overwrite crucial information as chats continued. Summarize conservatively, keep raw recent turns verbatim as long as budget allows, and treat the summary as a fallback rather than the source of truth. One practical check for summary damage: periodically ask the agent to restate key stored facts and compare them against the source turns — silent overwrites show up as confidently wrong answers.
Retrieval: remembering on demand
The alternative to stuffing history into the prompt is fetching memories only when relevant: the agent embeds the current turn, searches a vector store for similar past content, and injects the hits into the prompt — retrieval-augmented generation applied to the agent’s own past rather than an external corpus.
LlamaIndex’s VectorMemoryBlock is the canonical building block here: long-term memories live in a vector store (the docs name qdrant, chroma, weaviate, and milvus as options) and are retrieved by semantic similarity at prompt-assembly time. Letta gives the agent explicit tools for the same job — archival_memory_insert to store, conversation_search to recall. Zep’s Graphiti engine takes a different approach, building a temporal knowledge graph instead of a flat vector index: every fact edge is bi-temporal, recording both when the fact was true in the world and when the system learned or superseded it.
Retrieval is what makes long horizons affordable, but it has its own failure signature. The MRCR v2 multi-needle benchmark shows even the strongest long-context models degrading as the haystack grows: Opus 4.6 falls from roughly 92% retrieval accuracy at 256K tokens to about 78% at 1M, and GPT-5.4 from roughly 80% at 128K to 37% at 1M. Retrieval narrows the haystack so the model never searches the whole thing — but a bad retriever silently starves the model of the facts it needs, which is why retrieval quality deserves its own evaluation, separate from the model’s reasoning.
Writing memory back: the part most agents skip
Reading memory gets all the attention; writing it is where durable agents are made. After each turn, the agent should extract what is worth keeping — a new preference, a corrected fact, a lesson about what worked — and store it. LangMem formalizes this as an extract-and-consolidate loop (create_memory_manager), driven by the LLM itself: the model decides what from this turn deserves to become a lasting memory.
Writes raise the hardest questions in memory design: what happens when the new fact contradicts an old one? When Zep’s knowledge graph resolves a contradiction, it marks the old fact invalid and records the superseding one rather than deleting it — invalidation scoped to contradiction resolution, not a blanket prohibition on deletion. Engram reconciles differently: it extracts memories asynchronously, compares new data against existing records, and explicitly rewrites, merges, keeps, or deletes them (their example: “the user used to work as a machine learning engineer, but has now been promoted to CEO”).
Letta lets the agent edit its own core memory blocks directly — but its docs warn that setting a block’s value replaces the entire block (not an append) and concurrent writers resolve to last-write-wins. Memory writes need the same care as database writes: conflicts, idempotency, and audit trails. As the LongMemEval authors conclude, “even the most capable long-context LLMs currently require an effective memory mechanism to manage an ever-growing interaction history” — and a memory mechanism that only reads is half a mechanism.
A working pattern: the AI agent memory loop in practice
The full loop has six ordered steps: load context from the layers, budget tokens across the slots, act (call the model), observe (fold tool results and other observations into the record), extract durable facts from the turn, and store them — truncating the short-term buffer when it fills. The sketch below illustrates the loop’s shape (the model call and fact extraction are stubs; the store is in-memory and retrieval is keyword overlap, not semantic search):
class ContextBudgetExceeded(Exception):
pass
def estimate_tokens(text):
# Stand-in only: word count approximates tokens.
# Production code uses the model's real tokenizer.
return len(text.split())
class MemoryStore:
"""Three layers: pinned blocks, long-term facts, short-term turns.
Toy limits: in-memory store (no cross-session persistence);
search() is keyword overlap, not semantic retrieval. Production
needs a persistent store plus embedding search.
"""
def __init__(self):
self.blocks = {"user_profile": "Prefers concise answers."}
self.facts = [] # long-term memory
self.turns = [] # short-term memory (recent turns)
def search(self, query):
q = set(query.lower().split())
return [f for f in self.facts if q & set(f.lower().split())]
def upsert(self, fact):
key = fact.split(":", 1)[0]
self.facts = [f for f in self.facts
if f.split(":", 1)[0] != key]
self.facts.append(fact)
def truncate_buffer(self, max_turns=20):
# Truncation, not summarization: older turns are dropped, no summary
# kept. Production summarizes evicted turns; omitted here.
self.turns = self.turns[-max_turns:]
def build_prompt(store, user_msg):
parts = ["## Pinned memory"]
parts += [f"- {k}: {v}" for k, v in store.blocks.items()]
parts.append("## Retrieved facts")
parts += [f"- {f}" for f in store.search(user_msg)] or ["- (none)"]
parts.append("## Recent turns")
parts += [f"User: {u}\nAssistant: {a}" for u, a in store.turns[-10:]]
parts.append(f"## Current message\nUser: {user_msg}")
return "\n".join(parts)
def check_budget(prompt, token_budget, reserve_for_output=2000):
used = estimate_tokens(prompt)
if used + reserve_for_output > token_budget:
raise ContextBudgetExceeded(
f"~{used} prompt tokens + {reserve_for_output} reserved "
f"exceeds budget of {token_budget}; truncate or summarize first")
def agent_turn(user_msg, store, tool_results=(), token_budget=30000):
# 1-2. LOAD + BUDGET
prompt = build_prompt(store, user_msg)
check_budget(prompt, token_budget)
# 3. ACT
reply = f"[model reply to: {user_msg}]" # stub: the LLM call
# 4. OBSERVE: tool results and other observations enter the record
# here, alongside the reply, before anything is extracted or stored.
store.turns.append((user_msg, reply))
for result in tool_results:
store.turns.append(("tool", result))
# 5-6. EXTRACT + STORE (production often consolidates asynchronously,
# or only when the turn produced something worth keeping).
store.upsert(f"topic: {user_msg[:40]}") # stub: fact extraction
store.truncate_buffer()
return replyA concrete case, with the toy’s limits in mind: the user mentions “we deploy on AWS,” and the extraction step stores infra: team deploys on AWS as a long-term fact. In a production system with a persistent store and semantic retrieval, a later question about database options would surface that fact, and the model would recommend RDS and DynamoDB instead of asking which cloud the team uses. In the sketch above it would not: search() only matches shared words, and the store vanishes when the process ends. The sketch shows the loop’s shape; persistence and semantic search are the components that make it work across sessions.
Common mistakes and limits
Treating the context window as the memory system. A 1M-token window does not mean 1M tokens of usable memory. NVIDIA’s RULER benchmark defines “effective context length” as the longest length at which a model clears a fixed quality bar — and only 4 of 10 tested models held satisfactory performance at 32K despite claiming far larger windows. The advertised window is a ceiling, not a guarantee: usable context depends on the model, the task, and the quality bar you require.
Unbounded memory growth. Every stored fact is a future retrieval candidate and a future cost. A 2026 study found full-history replay beat a memory pipeline on accuracy (by 35.2 points on LoCoMo, 33.4 on LongMemEval) — but the pipeline compressed ~101,600-token conversations to ~2,909 tokens. Accuracy against token cost is the core tradeoff of the whole field; memory is how you buy accuracy back cheaply.
Over-engineering simple tasks. Not every agent needs a knowledge graph. LangChain’s original conversation-buffer memory classes were deprecated at v0.3.1 (they now live in langchain-classic); for short tasks whose history and state requirements fit comfortably inside the window, a plain rolling buffer remains a reasonable choice. Match the memory system to the horizon: buffers for sessions, layered memory for relationships.
Done well, AI agent memory is invisible: the agent simply seems to know what it should know, session after session. Underneath is the unglamorous loop — load, budget, act, observe, extract, store — running in its simplest form on every turn, with consolidation often deferred or asynchronous in production, governed by a token budget, and evaluated against the ways it can silently fail. Build the loop, instrument the failures, and memory stops being the thing your users complain about.
