AI agent memory architectures decide what an agent keeps in mind right now and what it stores away for later. The split matters because a language model is stateless: everything an agent “remembers” is tokens the developer puts into its context window, and that window is finite. Working memory is the live context of the current run. Episodic storage is the durable record of past runs that the agent can recall on demand. This article explains how the two layers work, how information moves between them, and how to choose the combination that fits your agent.
What memory means inside an AI agent
The starting point is simple: an agent remembers nothing between sessions unless the design adds a mechanism. The model itself holds no state between calls. Whatever persists, the application put there.
The field has converged on a four-part taxonomy, formalised by the CoALA framework from Princeton researchers in 2023. Working memory is the active scratch pad: the current conversation, retrieved data and intermediate reasoning. Episodic memory is the autobiographical record: specific past events and interactions. Semantic memory is general knowledge accumulated over time, and procedural memory is skill: the instructions, prompts and tool definitions that encode behaviour. This article concentrates on the first two, because that is the split this question asks about. Semantic memory is mostly produced by consolidation — the agent reading its own episodes and distilling the pattern — and procedural memory can itself be learned or stored: LangMem lets an agent rewrite its own system prompt from accumulated experience, and LangMem layers semantic, episodic and procedural memory on LangGraph’s BaseStore.
Anthropic’s line between workflows and agents helps here too. A workflow is LLMs and tools orchestrated through predefined code paths; an agent is a system where the model dynamically directs its own process and tool usage. A workflow can often get by with a fixed context, while an agent that plans its own steps needs somewhere to keep what it has learned.
Short-term working memory: the live context window
Working memory is everything the model can see in a single inference: the system prompt, the conversation history, the results of tool calls, and the reasoning trace it is building. It lives entirely inside the context window, and it vanishes when the run ends unless something writes it out.
Context windows have grown large: OpenAI’s cookbook notes GPT-5 accepts up to 272,000 input tokens and 128,000 output tokens, yet adds that even such a large window can be overwhelmed by uncurated histories, redundant tool results, or noisy retrievals. Size has not removed the budget problem; it has only moved the limit further out.
Two techniques manage the pressure, both documented in OpenAI’s Agents SDK. Trimming drops older turns and keeps the last N: cheap and deterministic, but it forgets abruptly. Summarization compresses prior messages into shorter summaries injected back into the history: it retains long-range context compactly, but adds latency and risks drift.
Neither technique is free, because recall gets worse as the window fills. Anthropic’s engineers call the effect context rot: as token counts grow, recall decreases, because every token attends to every other token and the attention budget spreads thin. Chroma’s July 2025 report tested 18 models with task difficulty held constant and only input length varied, and performance degraded in every experiment. In Chroma’s LongMemEval comparison, the same 306 questions were answered from a focused input of about 300 tokens and from the full conversation of about 113,000 tokens — and the full input performed worse, even though it contained the same answer.
In practice, short-term memory is managed through a session object. The OpenAI Agents SDK’s Session handles it: before each run the runner retrieves the session’s history and prepends it, and after each run all new items are stored automatically. A minimal example, with an API key set in the environment:
import asyncio
from agents import Agent, Runner, SQLiteSession
agent = Agent(
name="Support agent",
instructions="Resolve the ticket, then summarise what you did.",
)
session = SQLiteSession("support-1042")
async def main() -> None:
result = await Runner.run(
agent,
"The customer says the refund never arrived.",
session=session,
)
print(result.final_output)
asyncio.run(main())The session has two parts: the active context the model reasons over right now, and the persisted history the SDK stores and reloads across runs. The active context is still bounded by the window: when history grows long enough, trimming or summarization has to happen. The persisted history can be reloaded, but an uncurated reload replays the whole transcript rather than a selected record — and session history can also be trimmed, summarized or compacted before it is reloaded, so replaying every turn is a choice, not a necessity. That is where a separate episodic store earns its place: it keeps selected, retrievable facts rather than everything that was said.
Long-term episodic storage: remembering across sessions
Episodic storage is the durable record of what happened: a specific interaction, at a specific time, in a specific context, with a specific outcome. Unlike working memory, it is not loaded into the model by default. It sits in an external store, retrieved on demand.
A typical episode is a small structured record. The shape below is illustrative, but it shows the fields that make an episode retrievable later: who it was about, when it happened, what was decided, and how it ended.
{
"episode_id": "ep-1042",
"timestamp": "2026-11-09T10:15:00Z",
"user_id": "user-7",
"summary": "Customer asked for a refund on order 8814; agent confirmed the refund was issued on 4 November.",
"outcome": "resolved"
}Retrieval is usually semantic: the episode is embedded as a vector, and at run time the agent searches for episodes similar to the current situation. Mem0, an open-source memory layer, formalises the pattern: it takes raw messages, extracts the durable facts, deduplicates them against what it already holds, embeds them and saves them, then searches by a combined semantic, keyword, entity and temporal score. Its paper is arXiv:2504.19413.
LangGraph separates the two cleanly: a checkpointer holds history within a single thread, while a BaseStore holds facts that survive across all sessions for a user. Vector stores are the common choice for episodic retrieval, but timestamped logs with metadata filtering work when recall is mostly by recency or explicit keys. The store should match the recall pattern: similarity, recency, or explicit keys.
How AI agent memory architectures move information between layers
Information can cross between the layers in three ways, and each crossing is a design decision rather than a requirement: the write path, the recall path, and consolidation. Which ones an application needs depends on what it must remember, and for how long.
The write path decides what gets promoted from working memory into episodic storage, and when — a policy to settle before the store is built, since storing every turn is too much and storing only on explicit instruction is too little. LangChain distinguishes two timings: hot-path memory updates immediately, before the agent responds, as ChatGPT does; background memory moves the extraction work off the response path, and the memory becomes available once processing completes.
The recall path is the reverse crossing: the agent decides what to retrieve and how much to load back. MemGPT, the 2023 UC Berkeley paper that reframed agent memory as an operating-systems problem, made this explicit. It treats the context window like RAM and external databases like disk, and gives the model function calls to page information in and out. When the model wants a past episode, it emits a call such as this one:
{
"name": "archival_memory_search",
"arguments": {
"query": "customer refund preference",
"page": 0
}
}The results come back into the context window, so working memory now holds a selected slice of the episodic record rather than the whole archive. The paging is self-directed: the model decides what to fetch, instead of the application stuffing the prompt in advance.
Consolidation is the third crossing, and the one most often skipped. Episodes accumulate; left alone they bloat the store and retrieval gets noisier. Consolidation reads episodes and distils them into shorter, more durable form: a session summary, a standing preference, a lesson learned. MemGPT does it under pressure, warning the model at 70 percent of the window to move useful information out, and generating a recursive summary from evicted content at 100 percent. Anthropic’s structured note-taking is the same idea in plainer form: the agent writes notes outside the context window and reads them back when needed. In Anthropic’s Claude Plays Pokemon experiment, the agent maintained strategic notes across thousands of steps and continued seamlessly after context resets.
The crossings have failure modes. Stale recall loads an episode that no longer applies. Over-retrieval drowns the context window — the context-rot problem the episodic layer was supposed to solve. Lost episodes happen when the write path never fires. Anthropic’s multi-agent research system guards against exactly this: its LeadResearcher begins by saving its plan to Memory as a precaution, because if the context window exceeds 200,000 tokens it will be truncated and the plan must survive.
Measured results cut both ways. On the MemGPT paper’s long multi-session conversation test, GPT-4’s accuracy rose from 32.1 percent to 92.5 percent with paging in place — though a follow-up analysis noted the paper’s other benchmark is small and overly retrieval-focused, with a full-conversation baseline scoring about the same. Paging helps a lot when the window cannot hold the history, and matters less when it can.
A worked example: one task across two sessions
Take a support agent handling a refund. In session one, a customer says a refund never arrived. The agent checks the order, finds the refund was issued on 4 November, and tells the customer. As the session closes, the conversation is nearing the context budget, so the agent first writes a short handoff summary of what was decided — order 8814, refund issued 4 November, customer told — and the write path fires: an episode is stored recording the order number, the refund date and the resolution.
In session two, a week later, the same customer asks whether the refund has cleared their bank. The agent starts with a fresh, empty working memory, searches episodic storage for this customer, retrieves the episode from session one, and loads just that record into context. Recall supplies the prior fact — the refund was issued on 4 November — but it cannot establish whether the money has cleared the bank; that remains unknown until checked or confirmed by the customer. The agent says so, checks the order status, and updates the episode when the customer confirms the money arrived.
Each layer contributed something different. Working memory carried the live reasoning of the current session; the episodic store supplied one selected fact from the earlier session, where reloading the persisted session history uncurated would have meant replaying the whole transcript — though that history could equally have been trimmed, summarized or compacted first. Without the write path, session two would have started from zero. Without the recall path, the stored episode would have sat unused. And without consolidation, a year of such episodes would accumulate unchecked — though retrieval is indexed and filtered, so the agent searches the store rather than loading it all.
Choosing the right combination for your agent
Working memory alone is enough when tasks are single-session and short: the conversation fits the window, nothing needs to survive the run, and the session object handles continuity. Most chatbots and single-turn tools live here comfortably.
Episodic storage earns its complexity when the work spans sessions: multi-session tasks, personalisation that depends on remembered preferences, and audit trails where what happened before is itself the deliverable. If a returning user should not have to repeat themselves, you need a store outside the window.
Match the store to the recall pattern: timestamped logs for recency, vector search for similarity, a structured database for exact lookup. Many production systems combine them, and Mem0’s combined semantic, keyword, entity and temporal scoring is one example of that hybrid in a single search call.
Cost and latency belong in the decision too. A synchronous search adds latency on the turns that invoke it: the agent waits for the search before it can reason. But retrieval is often conditional or model-directed — MemGPT’s paging, for example, fires only when the model chooses to fetch — so not every turn pays the cost. LangChain publishes no retrieval-latency figures for LangMem. Measuring memory search in your own environment before putting it in front of users is the practical recommendation here. The trade-off runs the other way as well: secondary reporting of the LOCOMO benchmark puts Mem0 at 26 percent higher than OpenAI’s built-in memory feature, responding 91 percent faster and cutting token usage by about 90 percent against full-context approaches. The numbers are as reported, not from a vendor report: selective retrieval costs a search and saves a window.
Common mistakes and hard limits
The most common mistake is treating the context window as long-term memory. It resets. An agent that keeps everything in context works until the run gets long, then degrades quietly as trimming and summarization discard the parts nobody chose to preserve.
The second is storing everything and retrieving noise. A store with no write policy fills with trivia, and retrieval returns plausible-looking irrelevance that pushes the good material into the middle of the window, where models use it worst.
The third is assuming recalled text is verified fact. An episode is a record of a past event, not a ground-truth database: its contents may be model summaries, user statements or recorded observations, such as the tool-derived refund information in the worked example. Recalled episodes can be wrong, stale, or misattributed, and must be assessed for provenance, freshness and relevance before the agent relies on them.
The fourth is forgetting that episodes are data, with the obligations that follow. Under the GDPR, stored agent memory is personal data when it contains data relating to an identifiable person. Where it does, Article 5 requires storage limitation, so those stores need a purpose, a lawful basis and a retention period; a store with no retention period violates storage limitation by default. Article 17 gives people a right to erasure over the stored records that relate to them. And the erasure has a practical catch: deleting a conversation does not delete the memories made from it, because a saved memory is a separate record. A memory system needs a deletion path that reaches every store holding their data.
Working memory and episodic storage are not competing designs. They are two halves of one system: a small, fast, forgetful surface where reasoning happens, and a large, slow, durable record the surface can draw on. The architecture is the set of rules that moves information between them — what gets written, what gets recalled, what gets distilled — and those rules, more than the size of any window, decide whether an agent actually remembers.
