Ask an AI agent to pick up where yesterday’s conversation left off, and you quickly discover the difference between a clever chatbot and a useful assistant. A model on its own remembers nothing between calls: each request is processed fresh, from the tokens in that request alone. Everything that looks like remembering has to be built around the model, by an application that saves selected information and puts the relevant parts back into a later call. That surrounding system, and the decisions behind it, is exactly what ai agent memory is about.
Memory is what turns a single-turn tool into something that can carry a task forward. It is the difference between an agent that starts from scratch every time and one that knows your preferences, remembers what it already tried, and does not ask you the same question twice. Getting it right is less about storing more and more about deciding what deserves to be remembered, for how long, in what form, and who is allowed to change it.
What agent memory actually is
Agent memory is retained state that an agent can read back and act on. It is not the model itself, and it is not the same thing as the conversation you see on screen. It includes short-term working memory, the current context window, as well as long-term memory: information the surrounding application deliberately keeps outside any single model call, so that a later step, a later turn, or a later session can work with the right facts.
That state can span very different timescales. Within a single conversation, memory is the running record of what has been said, decided, and done. Across sessions, it is what lets an assistant greet a returning user with context intact rather than a blank slate. Across tasks, it is what lets an agent carry lessons from one job into the next, instead of relearning the same ground every time.
It is worth being precise about what memory is not. It is not the model weights: training knowledge is frozen at training time and cannot record that you changed your mind last Tuesday. And it is not the same thing as the context window, though the two are constantly confused. The context window is the short-term form of memory, working space for one inference. Long-term memory is what persists beyond it, and what gets loaded into the window when it matters.
The main kinds of agent memory
A useful taxonomy, supported by the research for this article, divides agent memory into four kinds. The labels matter because each kind is stored, updated, and retrieved differently.
Short-term, or working, memory is the current conversation and task state: the system prompt, the messages so far, the tools available, and the tool outputs produced in this session. It lives inside the context window, it is fast and fully accessible, and it disappears when the session ends, or earlier, when history is trimmed or summarized.
Episodic memory records specific past events and interactions: what happened, when, and with what outcome. A support agent’s note that a particular fix failed on a customer’s machine last week is episodic. Its value lies in the specifics, including the outcome: recording outcomes clearly, including explicit labels where useful, helps an agent learn what worked and what did not.
Semantic memory holds distilled facts, preferences, and knowledge, detached from the moment they were learned: a user’s preferred programming language, a client’s risk tolerance, a project’s key constraints. Where episodic memory says what happened, semantic memory says what is true, as far as the agent knows.
Procedural memory is learned skill: reusable routines, tool-use patterns, and successful action sequences. A coding agent that has learned to run the test suite before committing, or a claims agent that follows an approved workflow, is drawing on procedural memory. It is the closest thing an agent has to habit.
How agents store and retrieve memories
Building memory into an agent means answering four questions in order: what gets captured, where it is stored, how it is retrieved, and how it is used at decision time.
Capture starts with observations, actions, and feedback. An agent may record user messages, its own actions, tool results, and whether the outcome succeeded or failed. Recording everything is the easy default and usually the wrong one: every stored item is a future retrieval candidate, and indiscriminate capture is how memory stores become noisy.
Storage depends on the kind of memory. Conversation history can live in a database or event log. Semantic facts and episodes are often stored with embeddings in a vector store, so they can be found by meaning rather than exact wording. Structured records suit facts that need updating, like a preference that changes. The storage choice is less important than the discipline around it: what is stored, under what scope, and with what expiry.
Retrieval is where memory succeeds or fails. Rather than loading an agent’s entire stored memory into context on every request, which would be expensive and counterproductive, the system searches for the pieces relevant to the current request and pulls in only those. The classic design for this comes from Park and colleagues’ Generative Agents from 2023, which kept a chronological memory stream and retrieved from it using a weighted combination of recency, importance, and relevance. Most later systems vary that idea rather than replace it: recent memories are favoured, memories judged important are favoured, and memories similar to the current situation are favoured.
Finally, use at decision time means the retrieved memories join the working memory for one model call, alongside the current task and instructions. Only that assembled prompt is what the model actually sees. Everything else, the whole memory store, stays outside until it is called for.
As a conversation grows, a second mechanism kicks in: summarization and compaction. When history approaches the context limit, older turns are compressed into a running digest that preserves the goal, the confirmed facts, the decisions, and the open questions. That frees tokens for new reasoning, but it is a lossy compression, and a summary can omit a detail that matters later. The safeguard is to keep the original event log outside the context window, so a summary can be checked or regenerated.
What ai agent memory enables in practice
Continuity is the most visible benefit. An agent with memory can pick up a task where it left off, even after a pause, a crash, or a new session, because its plan, its partial results, and its decisions were checkpointed rather than held only in a vanishing conversation.
Personalization follows from semantic memory. An assistant that knows a user’s preferences, plan tier, or past requests can adapt without being told the same things again. That only works when the facts are accurate and current, which is why a memory that cannot be corrected is a liability dressed as a feature.
Learning from feedback is the payoff of episodic memory. When outcomes are recorded, an agent can avoid repeating a failed approach and repeat a successful one. One design option is to deliberately distil repeated episodes into semantic facts or procedures where that helps; it is a choice to implement and maintain, not an automatic progression, and it sits alongside the evidenced mechanisms of summarization, relevance filtering, correction, and expiry.
And multi-step coherence is what holds long tasks together. An agent working across tools and sessions needs to remember what it has already established, what it has already tried, and what remains open. Without that, each step is a fresh guess; with it, the steps add up to a result.
Memory vs the context window
If there is one confusion to clear up, it is this: the context window is not the whole of memory, only its short-term part. The window is the set of tokens sent to the model for one inference, including instructions, user messages, tool definitions, retrieved documents, and recent tool results. It is the model’s working surface for that call, and nothing more.
The window is bounded, and stuffing everything into it fails in three ways. It gets more expensive as it grows, because providers charge by tokens processed. It gets less reliable, because information buried in the middle of a long context is retrieved less accurately, a phenomenon known as the lost-in-the-middle problem. And it is not available to a fresh session at all: when the session ends, the window is gone unless something outside it deliberately saved what mattered.
The practical rule is to use each mechanism for what it is good at. Summarize for current-task continuity, so the agent keeps moving without replaying every message. Retrieve for durable knowledge, pulling in only the facts and episodes relevant now. And forget on purpose: expire what is stale, deprecate what is provably obsolete, and keep the working set small enough that the fact that matters is not buried.
Letta, formerly MemGPT, makes this division explicit with an operating-system analogy. It treats the context window as RAM and archival storage as disk: a small core memory block stays always in context, and the agent itself manages its memory hierarchy, editing core memory and paging archival memories in and out on its own initiative. The point of the analogy is architectural, not cosmetic. Memory is a first-class part of the agent’s state, not an external add-on bolted on afterwards.
Limits, risks, and common mistakes
Memory is not an unqualified good, and the failure modes are worth naming plainly. The first is staleness. A memory that was true in March can mislead an agent in October: a changed email address, a cancelled project, a switched technology stack. Memories that cannot be updated or expired do not preserve knowledge, they preserve error.
The second is privacy and security. Retaining user data creates obligations: who can read these memories, who can change them, and how a user inspects or deletes them. An agent that remembers without giving the user control over what it remembers increases the risk of inappropriate retention, and makes it harder to detect and correct.
The third is over-retrieval. More context does not always mean better answers. Irrelevant memories clutter the working set, distract the model, and crowd out the facts that matter. A memory system that retrieves generously but ranks poorly is worse than one that retrieves narrowly and well.
The fourth is neglected maintenance. Episodes and facts that are never reviewed, corrected, or expired can accumulate into a store that grows noisier over time. The discipline that prevents this is unglamorous: summarize on a schedule, promote repeated patterns deliberately where useful, and delete or deprecate what has been superseded.
Choosing your ai agent memory approach
Start from the task, not the tooling. Ask what must persist, and for how long. A simple single-session task may need working memory only, when the necessary information fits in the current context. A personalized assistant often benefits from semantic memory added. An agent that should learn from outcomes benefits from episodic memory that records outcomes clearly. An agent executing defined, repeatable workflows often benefits from procedural memory. Adding a kind of memory before you can say what it is for is how systems acquire stores nobody queries.
Match the memory type to the need before picking a store. If the need is facts and preferences, a structured or vector-backed semantic store fits. If the need is what happened and whether it worked, an event log with outcome labels fits. If the need is repeatable behaviour, a versioned procedure or rules store fits. The store follows the need, not the other way round.
Design for correction, expiry, and user control from the start. Every memory should carry enough provenance to be checked, a way to be corrected when wrong, and an expiry or review point when its truth may have changed. Users should be able to see what an agent remembers about them and ask it to forget. These are not polish features; they are what make a memory layer trustworthy enough to keep using.
Finally, measure before expanding. A practical starting point is short-term plus semantic memory, adding episodic and procedural memory only when the improvement can be measured. Memory, done well, is invisible: the agent simply picks up where it left off, remembers what matters, and forgets what does not. That invisibility is the goal, and it is earned by the unglamorous work of capture, retrieval, summarization, and deliberate forgetting.
