An agent that works for hours — researching a topic, writing code, or running a multi-step workflow — will eventually run into the context window limitations that every large language model carries. The window is finite, even at a million tokens, and every message, tool result, and retry accumulates inside it until something has to give. Specialized agent frameworks exist largely to manage that moment: they summarize history, retrieve what matters, page memory in and out, and split work across fresh contexts so the agent keeps going instead of forgetting what it was doing. This article explains the mechanisms frameworks use, how the major frameworks implement them, and how to choose between them.
Why Context Window Limitations Break Long Tasks
Three forces collide in a long task. First, the window is finite. Even the largest current windows — a million tokens on the Claude Sonnet and Opus families, about a million on GPT-5.4, 1,048,576 tokens of input on Gemini 3.1 Pro, and 400,000 on GPT-5 — fill up when a task runs for hours, because every tool call and its result stays in the conversation. Second, more context is not strictly better. Anthropic puts it plainly: “context is a finite resource with diminishing returns, and irrelevant content degrades model focus.” As history grows, the model’s attention spreads thinner and the chance it loses the thread rises. Third, context costs money on every call: the full history is re-sent with each new step, and some providers add long-context surcharges above a threshold — around 270,000 tokens for GPT-5.5 and above 200,000 for Gemini 3.5 Flash.
When the window finally fills, the outcome depends on the framework. Some trim silently, dropping old turns without telling you. Others stop with an error. The failure mode that matters most is the quiet one: the agent keeps working with a degraded view of its own task, and goal drift sets in — decisions made early get lost, constraints stated once get forgotten, and the agent starts repeating work or contradicting itself.
How Context Windows Actually Work in Practice
The context window holds the full message history plus tool results. Nothing outside it exists for the model: if a fact is not in the window, the model cannot use it, however important it was three hours ago. Providers count the window in tokens, and the advertised maximum is not the usable maximum. Part of the window goes to the system prompt, tool definitions, and the model’s own output, and the effective usable space is smaller than the headline number.
The gap between advertised capacity and usable recall is measurable. Anthropic publishes MRCR v2 recall figures showing Claude Opus 4.6 at 78.3% at the 1M mark; at 128,000 tokens, Gemini 3.1 Pro and Claude Opus 4.6 score similarly, both around 84.9%. OpenAI has not published a comparable MRCR v2 figure for GPT-5.4 at 1M tokens. The practical reading is simple: a bigger window buys more room, not more reliable recall across all of it. That is why frameworks do not rely on window size alone — they manage what goes into the window.
Summarization and Compaction: Shrinking History as You Go
The most widely used mechanism is summarization: replace old turns with a condensed record and keep working from the summary. LangGraph’s LangMem library provides this as a prebuilt summarize_messages function and a SummarizationNode. The pattern, in outline, is this — the LangMem guide documents the full call signature:
# Sketch of the LangMem summarization pattern — the LangMem guide
# documents the full call signature:
#
# SummaryState(MessagesState) graph state holds a running summary
# summarize_messages(...) folds older turns into the summary
# max_tokens_before_summary when summarization kicks in
# max_summary_tokens how large the summary may grow
# the model then sees recent messages + the running summaryThe thresholds are the tuning knobs: max_tokens_before_summary decides when summarization kicks in, and max_summary_tokens caps how large the running summary grows. Summarization reduces the accumulated history under those budgets — it does not guarantee the whole request fits. Recent messages, new tool outputs, and other prompt content still count against the window, so they need their own size limits or handling alongside the summary.
Anthropic offers the same idea at the platform level. Its SDK compaction triggers by default at 100,000 tokens: the SDK injects a summary request, Claude generates a structured summary covering the task overview, current state, important discoveries, next steps, and context to preserve, and the entire message history is replaced with it — a history approaching 100,000 tokens collapses back to roughly 2,000–3,000 tokens of summary. Claude Code’s /compact command works the same way: in one documented test run, compaction cut total usage from 68,700 to 51,900 tokens and message tokens from 38,900 to 22,800.
What summarization loses is detail. Exact values, error messages, and the reasoning behind a decision survive only if the summary kept them, and over many cycles the summary can drift — each new summary is a summary of a summary, and small distortions compound. Anthropic’s advice is blunt: a clean session with a better prompt almost always outperforms a long session with accumulated corrections. That is why anything that must survive a long session belongs in a durable file on disk, not only in a chat message: the file is still there to be read after compaction, while anything that existed only in the conversation survives only as far as the summary includes it.
Retrieval Over Memory: Pulling Back Only What Matters
Retrieval takes the opposite approach: keep the full history outside the window and pull back only the relevant parts when needed. Conversation turns are stored in a vector store, and when the agent needs context it queries by semantic similarity and injects the matching turns into the prompt. Nothing is compressed, so exact facts survive — the trade is that the agent only sees what the query retrieves.
Retrieval beats summarization when exact recall matters: a specific error message from two hours ago, a rare detail, a decision made early in the task. It also fits long-horizon recall, where the relevant fact is buried in history the summary long since discarded. LangGraph’s cross-thread Store works this way, holding namespaced facts the agent can search by query, and AutoGen’s older TeachableAgent persisted teachings in a vector database and retrieved individual memos into context as needed.
Its failure modes are retrieval failures. A miss means the needed fact was not retrieved — whether the search returns nothing or returns irrelevant or stale entries — and what happens next depends on the agent’s fallback: it may retry with a different query, ask for clarification, or proceed without the fact. Stale entries are a second hazard: an early decision that was later reversed can be retrieved as if it were still current. And over-retrieval re-inflates the window that retrieval was meant to shrink. Retrieval works best when the store is scoped — per user, per project, per task — and the agent retrieves deliberately rather than on every turn.
Hierarchical and Structured Memory
Hierarchical memory treats the context window the way an operating system treats RAM: small, precious, and managed by paging. The idea comes from the MemGPT paper (arXiv:2310.08560), which proposes “virtual context management, a technique drawing inspiration from hierarchical memory systems in traditional operating systems that provide the appearance of large memory resources through data movement between fast and slow memory.” In MemGPT, the main context — system instructions, working context, and a FIFO queue — is the RAM, and external context — archival and recall storage — is the disk. The LLM itself moves data between tiers through function calls, and when prompt tokens pass a warning count (for example, 70% of the window), the queue manager inserts a memory-pressure warning so the model can save important information before eviction.
The same shape can be used as a design pattern rather than a universal rule: keep a small working set in the prompt and the rest in searchable stores outside it. In MemGPT’s design, the main context holds system instructions, working context, and a FIFO queue of recent messages, while external context holds archival and recall storage; the model moves data between the two through function calls. Other memory layers may organize the same idea differently — the research establishes the MemGPT design, not a single architecture every layer follows.
Structured memory earns its complexity when a task needs knowledge to persist and accumulate across sessions; whether it is the right choice depends on how much must be remembered, how exact the recall must be, and how much coordination the design can carry — not on duration alone.
Task Decomposition and Sub-Agents: Fresh Contexts per Subtask
The architectural escape hatch is to stop trying to fit everything into one window: decompose the long task into subtasks, give each its own fresh context, and have the parent pass only summaries and results between them. No single context holds the whole task — each sub-agent sees its own instructions and results, and the parent holds the plan and the consolidated outcomes.
Multi-agent frameworks are built around this pattern, and managed platforms now do it for you: the OpenAI Agents API, launched in September 2026, keeps sessions alive for hours, summarizes old context before it overflows, and spreads work across up to three helper agents. The cost is coordination: vague handoffs produce vague results, and cross-subtask context can be lost when two subtasks depend on each other’s details. Decomposition also needs a budget — LangGraph.js defaults to a recursionLimit of 25 super-steps, so keep a step counter in state and route to a summary node before the budget runs out.
What the Frameworks Actually Offer
Each framework leans on a different combination of the four mechanisms.
For trimming, LangChain’s trim_messages cuts chat history to a token budget, keeping the system message and starting the trimmed sequence on a human turn:
from langchain_core.messages import (
AIMessage,
HumanMessage,
SystemMessage,
trim_messages,
)
from langchain_openai import ChatOpenAI
messages = [
SystemMessage("you're a good assistant, you always respond with a joke."),
HumanMessage("i wonder why it's called langchain"),
AIMessage(
'Well, I guess they thought "WordRope" and "SentenceString" just '
"didn't have the same ring to it!"
),
HumanMessage("and who is harrison chasing anyways"),
AIMessage(
"Hmmm let me think.\n\nWhy, he's probably chasing after the last "
"cup of coffee in the office!"
),
HumanMessage("what do you call a speechless parrot"),
]
trimmed = trim_messages(
messages,
max_tokens=45,
strategy="last",
token_counter=ChatOpenAI(model="gpt-5.5"),
start_on="human",
include_system=True,
allow_partial=False,
)Trimming affects only what the model sees, not what is stored — the full history remains in the checkpointer. That separation is the key to LangGraph’s design: a checkpointer (InMemorySaver for development, SqliteSaver or PostgresSaver for production) persists full graph state per thread_id, so a failed run resumes from the last checkpoint instead of restarting.
CrewAI takes the opposite approach: memory is a single configuration flag. A crew with memory=True enables three memory layers — short-term memory for the current run, long-term memory across runs, and an entity store for proper nouns:
from crewai import Crew
crew = Crew(
agents=[researcher, writer],
tasks=[research_task, write_task],
memory=True,
)Short-term memory covers the current execution, long-term memory persists across runs in SQLite, and entity memory tracks people, organizations, and concepts. CheckpointConfig saves crew state at each task boundary, so a crew that fails midway resumes from the last checkpoint instead of re-running.
AutoGen defines memory as a protocol rather than a fixed store. The Memory protocol has five methods — add, query, update_context, clear, and close — and an agent’s update_context mutates its internal model context with retrieved information. ListMemory, the provided example implementation, keeps memories in chronological order and appends the most recent ones to the model’s context.
The OpenAI Agents SDK keeps it minimal: sessions. A SQLiteSession (or a Redis-backed store in production) automatically maintains conversation history across runs, and SessionSettings(limit=N) can cap how many of the most recent messages a run fetches:
# Conceptual sketch — a session preserving history between two runs.
# No constructor, runner, or result signature is asserted here:
#
# run 1: ask a question → the session records the exchange
# run 2: ask a follow-up → the session supplies the earlier exchange
# the agent answers from the preserved history, not from a fresh start
#
# Documented behavior: sessions maintain conversation history across runs;
# SessionSettings(limit=N) caps how many of the most recent messages are fetched.One constraint to know: in the same run, a session cannot be combined with conversation_id, previous_response_id, or auto_previous_response_id — you choose the SDK’s client-side sessions or OpenAI’s server-managed continuation, not both.
Anthropic rounds out the picture at the API level with context editing: server-side tool result clearing (clear_tool_uses_20250919, active at 100,000 input tokens by default, keeping the 3 most recent tool uses) and thinking block clearing (clear_thinking_20251015), plus the client-side compaction described above. Clearing is applied server-side before the prompt reaches Claude, and the client keeps the full history locally — the edits change what the model sees on that request, not the history you hold.
Choosing a Strategy and Avoiding Common Mistakes
The choice comes down to what the task demands. How fast does context grow, and how much of it must survive? If the task accumulates history slowly and the agent can work from a condensed record, summarization fits; if it needs exact facts back later, retrieval or a memory store fits better. Does knowledge need to persist across sessions? Then a durable store or a hierarchical design earns its place. How much coordination can the task tolerate? Decomposition adds handoffs and cross-subtask context loss, so it fits when subtasks are genuinely independent. Duration matters as one consideration among these — a longer task usually grows more context and needs more persistence — but it is not a sufficient or necessary condition for any single mechanism. And what is the budget? Summarization spends tokens on summaries, retrieval on embeddings and queries, and a larger window on every single call.
The common mistakes follow from getting those questions wrong. Silent truncation is the first: a framework that trims without telling you will drop the instruction that mattered, and the agent will not know it is gone. Trusting summaries without verification is the second — a summary that dropped a constraint looks exactly like one that kept it, so verify critical facts against the source, not the summary. Unbounded tool-output growth is the third: a single tool returning megabytes of logs can eat the window in one step, so cap or summarize tool outputs before they reach the model. The mistake underlying the others is treating the window as storage. The window is working memory. Anything the agent must not lose — decisions, constraints, the plan — belongs in a durable store or a file, where it survives compaction, trimming, and restarts.
Before a long task goes to production, test the context handling: run long enough to trigger summarization or compaction, feed the agent a constraint early and verify it still holds after the first summary, and deliberately fill the window with a large tool output to confirm the framework trims or clears it instead of failing. If the mechanisms are working, the agent finishes the task with its plan, its constraints, and its results intact — and the context window limitations that would have stopped it never become the story.
