An AI agent that helps a customer over three days faces the same memory problem as any long conversation, sharpened by session boundaries: it has to remember. AI agent state memory is the set of mechanisms that let an agent keep track of what has happened — the task’s progress, the conversation so far, and what it has learned about the user. This article explains the three kinds of state, how each one survives, and how to put them together into a design you can build.
What AI agent state memory actually is
Three different things hide inside the phrase “agent memory”. Session state is the task’s progress, variables and tool results within one run: which steps are done, what the tools returned, what is still pending. Working context is what the model can see right now, bounded by the context window. Long-term memory is facts, preferences and past episodes stored outside the model and recalled later. A system can be good at one and absent at another: an agent that recalls a customer’s preferences perfectly can still lose the ticket it was working on, because recalling a fact is not the same as resuming a task.
How conversation state survives inside one session
Inside a single session, the default state carrier is the message history: every turn is appended and sent to the model again, and the token cost grows with every turn. When the history approaches the context window, the standard answer is compaction: replace older turns with a running summary and keep going. Anthropic’s compaction does exactly this: when a conversation approaches the context window limit, Claude summarizes earlier messages so it can continue, keeping the gist, not every line. The Messages API exposes compaction as a beta, with a trigger threshold of at least 50,000 tokens, and Claude Code compacts automatically and offers a /compact command. LlamaIndex does the same in its memory classes: ChatSummaryMemoryBuffer summarizes what no longer fits, while ChatMemoryBuffer keeps the last messages within a token limit.
Compaction is one strategy, not the only one: a system can also select, truncate or retrieve portions of history instead of summarizing everything. Every route shares a general risk — excluding needed context can cause failures — but the loss of verbatim detail is specific to summarization: when compaction replaces older turns with a summary, a decisive fact can be the detail that gets dropped. For compaction, the defence is to make the summary carry state, not just topics: what was decided, what is still open, what the next step is. Bigger windows only delay the problem: Anthropic’s published windows run from 200K tokens to 1M on the API for Opus 5 and Sonnet 5.
How memory persists between sessions
Between sessions, memory splits into two jobs. The first is recall: facts and preferences in structured stores, and semantic recall of past conversations and documents in vector stores. Managed memory services and framework memory layers package this up — Mem0, Zep and Letta are the managed options — while LlamaIndex’s Memory class combines a short-term FIFO queue with long-term blocks holding static text, LLM-extracted fact lines, or vector retrieval. ChatGPT’s memory is the product version of the same idea: saved memories (facts you can view, edit and delete) and reference chat history, with a self-updating summary since the June 2026 rollout. It is fact snippets, not conversation continuity, and OpenAI does not publish the saved-memory capacity. A Max Planck Institute study of 2,050 donated memory entries from 80 ChatGPT users found 96% had been created by ChatGPT on its own — stored memory is the system’s interpretation of what mattered, and it can be wrong.
The second job is execution-state persistence, the one most memory articles skip. A durable checkpoint saves structured task state — progress, variables, tool results — under stable session, thread and user identifiers, so a later session can load it and resume without repeating completed work. LangGraph’s checkpointer is the clearest worked example: it saves a snapshot of graph state at each super-step, organized into threads, and the thread_id is the primary key for storing and retrieving checkpoints — without it, the checkpointer cannot save state or resume execution. A checkpoint is a StateSnapshot holding the state values, the next nodes to run, the config (thread_id, checkpoint namespace, checkpoint id), execution metadata and a creation timestamp, exposed through getState and getStateHistory. The wiring is a few lines once the graph exists:
from typing import Annotated
from typing_extensions import TypedDict
from langchain_core.messages import AIMessage
from langgraph.checkpoint.memory import MemorySaver
from langgraph.graph import END, START, StateGraph
from langgraph.graph.message import add_messages
class State(TypedDict):
messages: Annotated[list, add_messages]
def reply(state: State) -> State:
# A real node would call the model here; this one just
# records that it ran.
return {"messages": [AIMessage(content="ok")]}
graph = (
StateGraph(State)
.add_node("reply", reply)
.add_edge(START, "reply")
.add_edge("reply", END)
.compile(checkpointer=MemorySaver())
)
config = {"configurable": {"thread_id": "user-123-ticket-456"}}
graph.invoke({"messages": ["hello"]}, config=config)
# New input on the same thread continues from the saved state. It
# does not resume an unfinished run: reusing a thread_id alone never
# resumes execution. What starts new work, resumes unfinished work or
# replays history is the input and checkpoint_id you pass.
graph.invoke({"messages": ["and then?"]}, config=config)
# The resume pattern, for when work is left unfinished: invoke with
# no new input to load the latest checkpoint and retry what is unfinished.
graph.invoke(None, config=config)MemorySaver is in-memory only, so this example runs as-is for a demo; production needs a durable backend such as LangGraph’s SQLite or Postgres savers. Note what the second invoke is and is not: it submits new input to an existing thread, so the graph continues from the saved state. It does not resume an unfinished run — resuming unfinished work means invoking with no new input, which loads the latest checkpoint and retries what was left unfinished. Which of new work, resume or replay happens is decided by the input and checkpoint_id, not by the thread_id alone. What a stored checkpoint looks like as a StateSnapshot (message entries simplified to plain strings):
{
"values": {"messages": ["hello", "ok"]},
"next": [],
"config": {
"thread_id": "user-123-ticket-456",
"checkpoint_ns": "",
"checkpoint_id": "019a1b2c-3d4e-5f67-89ab-cdef01234567"
},
"metadata": {"source": "loop", "step": 2},
"createdAt": "2026-11-11T20:00:00.000Z"
}The distinction is scope: LangGraph’s checkpointer is per-thread execution state, while its separate Store holds cross-thread data such as facts and preferences. Checkpointing answers “where was this task?”; the store answers “what do we know about this user?” Confusing the two is how an agent recalls a customer’s name while re-running the three tool calls it already completed yesterday. Durability is a setting, not a guarantee: LangGraph’s modes run from exit, which persists only when execution ends, so a crash mid-run leaves no record of the thread, through async to sync.
How agents decide what to remember and what to recall
Writing memory is a choice, and retrieval is a ranking problem, not a dump of everything stored. The classic split is episodic memory (what happened), semantic memory (what is true) and procedural memory (how to do things). What gets written is a salience question. MemGPT’s answer is to let the model itself decide, moving information between main context and external storage with function calls, warning it at 70% of the window in the paper’s example so it can save what matters before the flush at 100%. LlamaIndex’s answer is structural: a short-term queue holds recent messages and long-term blocks hold what was extracted. Either way, what is stored is a selection, and a selection can be wrong. The MemGPT paper’s evaluation shows the scale of the difference: on a deep memory retrieval test, GPT-4 went from 32.1% accuracy without the system to 92.5% with it.
Retrieval is where the ranking happens: relevance to the current turn, recency, and grounding the agent in what was actually recalled rather than what it assumes. The risk of skipping the grounding step is visible in the benchmarks: an independent audit of LoCoMo found about 6.4% of its answer keys were score-corrupting errors, and the judge commonly used to score it accepted roughly 63% of intentionally wrong but topically adjacent answers. A lenient judge can conceal recall failures and inflate measured performance, so a high benchmark score is not proof that recall is sound. That is why recalled memories should reach the model as retrieved evidence with their source attached, not as established fact — and why recall should be validated in production, not only on the benchmark.
A reference architecture for long-running agent memory
Put the layers together and the architecture has five parts: working context, a short-term store, a long-term store, retrieval, and a durable checkpoint store beside them. The checkpoint store is what makes the design survive a session boundary: it holds snapshots of execution state — ticket identifier, completed steps, tool results, pending actions — keyed by stable session, thread and user identifiers, so a new session can ask for the latest checkpoint.
Take the worked example: a support agent handling a multi-day ticket. The customer reports a fault on Monday; the agent runs diagnostics, opens a parts order, and waits for the customer to confirm a visit slot. Each completed step writes a checkpoint: after the diagnostics tool returns, after the parts order is placed, after the visit is proposed. The checkpoint contents are the task state itself — ticket id, completed steps and their results, pending actions — not a summary of the conversation. On Wednesday the customer returns; the agent loads the latest checkpoint and resumes by asking for the visit confirmation. It does not re-run the diagnostics or place a second parts order, because the checkpoint says those steps are done: LangGraph’s pending-writes mechanism makes the same guarantee inside a single run — if a node fails mid-step, the successful nodes’ writes are already durable and are not re-run on resume. Note what the checkpoint does not do in this example: it does not tell the agent the customer’s preferred contact time, because this design keeps the preference in the cross-thread memory store and retrieves it separately. A checkpoint could also carry the preference, if the preference is part of the task state — checkpoints hold whatever state values the task includes. The distinction is between restoring execution state and retrieving cross-thread information, not between kinds of data.
In the agent loop, the working context is rebuilt on every turn from the short-term store. Writes to long-term memory happen when the agent decides something is worth keeping; reads happen at run time, ranked by relevance and recency. Checkpoint writes happen at step boundaries, as each piece of work completes; checkpoint reads happen when a session starts or resumes. Keep those operations on separate paths: a bad summary costs you a conversation, but a lost checkpoint costs you the task.
Where agent memory breaks down
Memory that misleads is worse than no memory. Context loss and silent drift: compaction drops a decisive detail and the agent proceeds confidently from a summary that no longer contains it. Stale, conflicting or hallucinated memories recalled as fact: stored entries go out of date — OpenAI redesigned ChatGPT memory in June 2026 because manually managed memories went stale and contradicted each other. Cost, latency and privacy: storing and retrieving everything costs tokens and milliseconds on every query — Mem0 reports about 6,900 tokens per query and Zep reports 155 ms retrieval — and stored user information needs a deletion strategy: decide what can be removed, by whom, and how, for the application in question.
The numbers quoted for memory systems are easy to over-read. LoCoMo and LongMemEval are the two multi-session benchmarks the industry converges on — LoCoMo runs 10 long conversations of about 300 turns and 9K–26K tokens each, up to 35 sessions, with about 1,540 question-answer pairs; LongMemEval has 500 questions in six categories — but the April 2026 audit found 6.4% of LoCoMo’s answer keys wrong and a judge that accepts roughly 63% of wrong-but-topical answers. With that much noise, small gaps between systems are not meaningful, and vendor-reported scores — Mem0 at 92.5% on LoCoMo and 94.4% on LongMemEval, Zep at 94.7% and 90.2% — are vendor claims on a noisy benchmark, not settled rankings.
Choosing a memory design for your agent
Match the memory layers to session length, user count and recall needs. For a simple conversational assistant, history plus summaries is the starting point — enough when the task is short-lived and nothing must survive a crash or a session boundary. A support agent that resumes multi-day tickets needs durable checkpoints keyed by thread. A personal assistant serving one user for months needs long-term memory with fact extraction and retrieval. Start simple, and add persistent memory only when a session boundary actually hurts — when users repeat themselves, when tasks restart from scratch, when the same question gets answered twice.
Two honest constraints should shape the choice. First, no vendor offers a general persistent-memory API for developers: ChatGPT memory is a product feature and Anthropic offers compaction, not cross-session memory, so durable memory today comes from frameworks such as LangGraph and LlamaIndex or from managed services such as Mem0, Zep and Letta. Second, test memory the way it will be used: multi-session benchmarks such as LoCoMo and LongMemEval, judged on accuracy alongside latency and token cost, and read with the audit caveats above. The checklist is short: separate session state, working context and long-term memory; save checkpoints at step boundaries under stable thread identifiers; retrieve ranked, grounded evidence rather than dumping the store; verify recall before trusting it in production.
