Agent token cost management is the discipline of keeping the token bill under control while an agent framework works through multi-step reasoning. A single prompt to a language model is cheap; a reasoning chain that calls the model twenty times, each time re-reading the conversation history, the tool definitions and every tool result so far — where the framework retains that growing history at comparable uncached rates — is not. The cost drivers are structural, not accidental, and the fixes are engineering decisions: measure per step, prune context, route work to the cheapest model that can do it, cache what repeats, and cap the worst case.
This article follows the money through a real reasoning chain. Each section covers one lever, what it saves according to published practitioner reports and vendor pricing, and what it costs in accuracy or complexity.
Why complex reasoning gets expensive
The first thing to understand is that token cost scales with context length on every API call, not just with new tokens. Each reasoning step sends the model the context actually included in that request — the system prompt, the conversation history it retains, the tool definitions, and the accumulated tool outputs it still carries — and pays input-token price for all of it. Full-history growth applies only where the framework retains that history; where it prunes or selects context, later steps cost less. Tool outputs, retrieved documents and conversation history all count toward input tokens, and every retry, self-correction loop and long chain compounds the bill.
That compounding is the core problem. Take a ten-step task where each step retains the growing history at comparable uncached rates: the tenth step re-reads the context of steps one through nine, so the chain costs more than ten times a single standalone call of the same size — the single-call baseline being one call with no accumulated history. Practitioner research on token usage finds that review and rework loops consume about 59% of tokens on average — not the initial generation — and that input context growth, not prompt size, is usually the main cost driver. A coding agent whose session bootstrap loads 35,000 tokens of global rules, project memory, summaries and tool definitions starts every user prompt expensive before any real work happens.
The practical consequence: carrying context it no longer needs into the next call is a common cost driver for agents — the research identifies input context growth as usually the main driver, not universally the most expensive activity. Everything that follows is a variation on removing that waste.
Agent token cost management starts with measurement
You cannot control what you do not measure, and most teams optimize the wrong thing because they have not measured where their tokens actually go. Before changing a model or a prompt, log tokens and cost per step, per tool call and per task — not just per session or per day.
Per-step attribution matters because it tells you which reasoning step caused the spend. A task whose final answer cost $0.40 might have spent $0.30 of it on a single retrieval loop that returned the same documents three times. Without per-step numbers, you would blame the model instead of the loop. Log the usage fields your provider returns — input tokens, output tokens, cached tokens and, where available, the reasoning-token count — and handle absent fields explicitly rather than assuming every provider exposes the same set. Attribute cost to the step that caused it, not only to the final answer.
Set budgets and alerts per task type, not just globally. A research task and a formatting task have different legitimate costs; a single global cap either strangles the research task or waves the formatting task through. A per-task budget also gives the agent a rule for what to do when the money runs out: summarize and continue, or escalate to a human. That rule is the runaway guard, and it is covered in detail later.
The snippet below is a simplified illustration of the idea, not a production guard: it assumes one uniform price per million tokens for input and output, merges both into a single counter, and checks an estimate before the call — it does not enforce a generation allowance or track actual monetary spend. It uses only the standard library:
class TokenBudget:
"""Per-task token budget. Call check() before each model call."""
def __init__(self, max_tokens_per_task, max_cost_per_task, price_per_million):
self.max_tokens = max_tokens_per_task
self.max_cost = max_cost_per_task
self.price_per_million = price_per_million
self.tokens_used = 0
def check(self, estimated_tokens):
"""Return (allowed, action, message) for the next call."""
projected = self.tokens_used + estimated_tokens
cost = projected * self.price_per_million / 1_000_000
if projected > self.max_tokens or cost > self.max_cost:
return False, "escalate", (
f"Budget limit reached: {self.tokens_used} tokens used, "
f"estimated ${cost:.4f} vs ${self.max_cost:.2f} cap"
)
return True, "proceed", "within budget"
def record(self, input_tokens, output_tokens):
self.tokens_used += input_tokens + output_tokens
budget = TokenBudget(
max_tokens_per_task=30_000,
max_cost_per_task=5.00,
price_per_million=2.00,
)
allowed, action, msg = budget.check(estimated_tokens=4_000)
if not allowed:
print(msg) # summarize and continue, or escalate to a human
Pruning context between reasoning steps
Pruning is the highest-leverage technique in this article, because it attacks the compounding directly. The rule is simple: keep only what the next step needs. Summarize completed sub-task history instead of carrying full transcripts, drop stale tool outputs once their conclusions are recorded, and keep a compact working state — decisions, facts and open items, not raw logs.
Production reports put the saving at 40–60% of tokens on long-running agents when context is summarized aggressively every two to three steps. The mechanism is straightforward: a summary of a completed sub-task might be 200 tokens where the raw transcript was 2,000, and every subsequent step then pays for 200 instead of 2,000.
LangChain’s summarization middleware implements this pattern inside the agent loop. It monitors the conversation and compresses older context automatically when thresholds are reached — triggers can be based on absolute token count, percentage of context capacity, or message count — while retention policies preserve recent messages uncompressed, so the agent never loses the last few turns it is actively working from. The compression itself should run on a cheaper model than the main agent: economical models handle summarization while the powerful model handles core reasoning, which keeps the summarization overhead minimal.
LangChain’s context engineering guidance describes this as one of four strategies — write, select, compress and isolate context — and compression itself covers three techniques: conversation summarization, tool output compression, and state-based compression using LangGraph state management for conditional compression. The practical pattern for a coding agent is small always-loaded memory plus on-demand retrieval, task-specific context, aggressive pruning, and a durable decision log. A session bootstrap that loads 35,000 tokens of memory before the user asks anything is the failure mode this prevents.
Choosing the right model for each sub-task
Not every reasoning step needs the strongest model. Routing means deliberately sending low-stakes sub-tasks to a smaller, cheaper model and reserving the expensive one for genuine reasoning — planning, judgment and final synthesis. Classification, extraction and formatting go to small models; complex reasoning and architecture decisions go to large ones.
The price gap is what makes this worth doing. A task routed to a frontier reasoning model can cost up to 190 times more than the same task handled by a small, efficient model: frontier input runs roughly $5–15 per million tokens against $0.10–0.50 for small or open models. Practitioner reports find that 60–70% of agent sub-tasks can run on mini models, and that model right-sizing usually cuts token spend by 50–70% with no measurable quality drop. Well-implemented routing cascades — where expensive models handle only the roughly 10% of queries that genuinely require their capabilities — achieve about 87% cost reduction.
A concrete example shows the shape of the saving. A legal document analysis agent ran every step, including simple extraction such as pulling party names from a contract header, on the same frontier model, at $1.40 per document. After auditing the call log, the team routed four of seven steps to a mid-tier model and two simpler steps to a small model, keeping only the final legal judgment step on the frontier model. The new cost was $0.34 per document — 75% less, at the same output quality.
Two routing rules keep this honest. First, set the default to the cheaper model and escalate consciously, rather than defaulting to the expensive model and never stepping down — most teams have the polarity backwards. Second, validate the routing decision: cheaper is only cheaper if accuracy holds. If the small model misclassifies, the retry on the frontier model costs more than running the frontier model first. Sub-agent delegation is the same idea applied to structure: give a child agent a cheap model and a tiny context, and have the expensive parent see only distilled results instead of doing the grunt work itself at full price with full context.
Caching, batching and avoiding redundant work
Prompt caching is the simplest large saving available, because it attacks repeated context directly. Caching is prefix-based: the provider caches the exact token sequence up to a breakpoint, and subsequent requests that begin with that identical prefix pay the cache read rate instead of the standard input rate. At Anthropic, OpenAI and Google, cache reads cost roughly 10% of the standard input rate — about a 90% discount.
The numbers are concrete. Anthropic charges 1.25 times the base input price to write a five-minute cache (2 times for a one-hour cache) and 0.1 times the base price to read from it. The break-even depends on the TTL: with a five-minute cache, one write plus one hit costs 1.25 + 0.1 = 1.35 input-price units against 2 uncached, so the first hit already saves; with a one-hour cache, one write plus one hit costs 2.1 against 2, so the saving starts on the second hit — assuming equal eligible prefix sizes. (The research package stated a single second-hit rule; the arithmetic above corrects it.) On OpenAI, cached input on GPT-5.6 Sol drops from $4.00 to $0.40 per million tokens, and caching is automatic for prompts sharing a common prefix of at least 1,024 tokens. The cache is maintained for about five minutes of inactivity at Anthropic, with extended TTL options; the minimum cacheable prefix is around 1,024 tokens.
To use it, mark the stable part of the prompt as cacheable. The snippet below shows the syntax only: it marks the system prompt as cacheable with cache_control, and the usage fields report how many tokens were created in the cache and how many were read from it. The example prompt is far shorter than the roughly 1,024-token minimum, so it would not itself create a cache — a real implementation needs a stable prefix of at least about 1,024 tokens and a second, matching request to demonstrate a cache hit.
from anthropic import Anthropic
client = Anthropic()
response = client.messages.create(
model="claude-3-5-sonnet-20241022",
max_tokens=1024,
system=[
{
"type": "text",
"text": "You are a product support agent for an e-commerce platform.",
"cache_control": {"type": "ephemeral"},
}
],
messages=[
{"role": "user", "content": "How do I track my order?"}
],
)
print(response.usage.cache_creation_input_tokens)
print(response.usage.cache_read_input_tokens)
Three rules make caching actually pay. First, put the static parts of the prompt first and the variable parts last, so the cacheable prefix is maximal — system prompt, tool definitions and few-shot examples up front, the user’s message at the end. Second, respect the TTL: a five-minute window means any gap longer than that between calls silently re-pays the write premium, so agents that idle between human turns can discover their “cached” workload is actually paying the write premium all day. Third, measure the hit rate. Caching only saves money if the cache is actually hit, and a routing layer that does not pin requests to the same upstream provider can silently reduce hit rates, with no error and no warning — so measure the hit rate across routing paths before crediting the savings.
Beyond prompt caching, two more techniques stop you paying twice for the same work. Batch independent sub-tasks instead of running them as separate chains, and memoize deterministic sub-results within a task. OpenAI’s Batch API gives a flat 50% discount on input and output tokens for asynchronous processing within 24 hours, which suits evaluations, bulk summarization and overnight analysis. Semantic caching — storing responses keyed by meaning rather than exact text — adds another layer: one production deployment reported a 34% hit rate on production traffic, saving about $1,600 a month.
Limiting loops, retries and runaway chains
Every technique above reduces the average cost. Caps bound the worst case, and the worst case is what destroys a budget: an unbounded reasoning loop is an unbounded bill. Set hard limits per task — maximum steps, maximum retries, maximum tokens — and enforce them in the loop, not in a dashboard the agent never reads.
Detect loops and repeated failures early. If an agent retries five times without progress, escalate to a human or kill the run; structured prompting and guardrails that inspect tool calls in real time can block bad tool invocations from executing and prevent the avoidable follow-up calls they would trigger — though the tokens already spent generating the bad call are not recovered. Confidence thresholds can serve as a task-specific stopping policy to validate: stop reasoning and answer when the model is confident enough for the task at hand, and tune the threshold against measured accuracy and cost for that task.
The simplified guard in the measurement section illustrates the idea; in production, a hard cap needs actual spend accounting using the applicable model, input, output and cache rates plus a reserved, enforced allowance for the next call, so an underestimated response cannot exceed the cap before it is recorded. When the budget is hit, the agent must summarize and continue or escalate — but work after a cap requires remaining or separately authorized budget, and summarizing and continuing cannot silently bypass the same cap. That rule is what turns a cost spike into a bounded, reviewable event instead of a surprise invoice.
Common mistakes and a practical checklist
The mistakes are the same in almost every cost overrun. Carrying full history into every call and never summarizing. Optimizing the model choice before measuring where tokens actually go — swapping the model when the real problem is a retrieval loop re-sending the same documents. No per-task budget, so one bad task can drain the day’s allowance. And enabling caching without verifying the hit rate, then assuming the discount is being received.
Apply the levers in order, because the order matters. Measure first: log tokens and cost per step and find the real hotspots. Prune second: summarize every two to three steps and keep a compact working state. Route third: move the routine sub-tasks to smaller models and validate that accuracy holds. Cache fourth: put static prompt parts first, mark them cacheable, and verify the hit rate. Cap last: hard limits on steps, retries and tokens per task, with escalation when they are hit.
The percentages in this article are separate, workload-specific reports, not a combined result: summarization trims 40–60% in production reports, model routing cuts 50–70%, and caching discounts repeated input by about 90%. They can overlap — summarization removes tokens that might otherwise have been cached, and routing changes the price of the tokens that remain — so combined savings have to be measured on the resulting workload, not added up from the standalone figures. The goal is not the cheapest possible agent — it is an agent whose cost is predictable, attributable to the step that caused it, and bounded even when the reasoning goes wrong.
