AI agents do not handle concurrent requests the way ordinary APIs do. An ordinary API often takes one request, does work that is roughly bounded for a given endpoint, and returns a response in milliseconds or seconds, though that also depends on the endpoint and workload. An AI agent can turn one user request into a variable, long-running workflow: several large language model calls, a series of tool calls, retries, and repeated reads and writes to state. That run can last seconds, minutes, or hours, and its cost can vary depending on how many steps it takes and how much context it accumulates.
The practical answer is to stop scaling agents like request handlers and start scaling the run. One pattern that fits this problem is to accept requests quickly, place the work on a durable queue, run it on workers that keep no authoritative state in memory, route model calls through a central gateway, and persist state externally so any worker can continue any run from its last persisted checkpoint. Capacity planning must include occupied execution time: tokens per minute = concurrent runs × model steps per run × tokens per step × 60 / seconds per run. Calculate requests per minute and input and output tokens per minute separately, because providers and budgets often limit each dimension on its own. Which resource bottlenecks first depends on the workload.
What concurrency means for agentic AI
Concurrency for agents has three layers that interact. The first is user concurrency: how many people or systems start runs at the same time. The second is run concurrency: how many agent runs are active at once. The third is step concurrency: how many model calls, tool calls, and subagents each run generates in parallel.
A single user request can fan out quickly. A research agent might spawn five subagents. Each subagent might call a model, then a search tool, then a model again. If ten users start similar runs at once, the system is not handling ten requests. It is handling ten runs, perhaps fifty active steps, and a burst of model demand that was invisible in the original traffic count.
Distinguish worker slots, waiting runs, and simultaneous model calls. A run that is actively executing, or awaiting a short-lived model or tool response, may retain an asynchronous job slot without making a model call at that moment. A durably suspended run waiting for human input or an external signal can instead release its worker slot, with its state persisted, and resume later without holding execution capacity. Steps within a run are often sequential, so fifty active runs do not automatically mean fifty simultaneous model calls. Capacity planning should therefore model occupied slots, suspended runs, waiting time, and model-call concurrency as separate quantities. Dividing slots by total run duration measures worker throughput only when runs retain those slots for that duration; it is not a general worker-sizing rule for suspended workflows. For workers that release slots during waits, size on occupied execution time, not wall-clock run duration.
State makes this harder. An agent run carries conversation history, intermediate results, checkpoints, memory, and side effects such as a ticket created, an email sent, or a record updated. If two runs write to the same thread or session at the same time, they can overwrite each other or produce lost updates. Externalising state alone does not prevent that: two workers can read the same version, both apply changes, and the second write can silently discard the first. The control for that case is covered below: serialise work per thread or use versioned optimistic concurrency with conflict handling.
Cost is also variable. An agent that loops, retries a failing tool, or re-sends a growing conversation history on every step is expensive. A retry can also incur additional cost when the original model call already ran and produced a charge before its result was lost.
An architecture pattern that fits the problem
The pattern below has five parts: a gateway, a durable queue, workers, a model gateway, and an external state store. It is a proposed reference pattern, not a claim about what most production systems use. Each part addresses a different concurrency problem, and the right balance depends on the workload.
Accept fast, then queue durably within bounds
Do not require an HTTP connection to stay open for a whole agent run. The gateway should authenticate the caller, apply per-tenant rate limits, check admission, and return a run or job identifier promptly. The actual work goes onto a durable queue. The client then polls, streams tokens where streaming is supported, or receives a webhook when the run finishes. Streaming is an exception to the general rule: a streaming connection can stay open while tokens arrive, provided the run lifecycle itself does not depend on that connection surviving.
A queue makes backpressure explicit and enables retries and dead-letter handling. But a queue absorbs only a finite burst: if the arrival rate stays above the sustainable service rate, no queue can drain the backlog indefinitely. The system therefore needs bounded backlog limits, admission control, deadlines or expiry for queued work, and explicit overload behaviour, such as rejecting new work with a retryable error, shedding lower-priority work, or degrading to a simpler path. The queue should be durable, so jobs survive a deploy, a crash, or a scale-down.
Use workers with persisted authoritative state
Workers need transient execution state while a step runs, such as local variables and in-flight call context. What they must not hold is the only copy of authoritative run state. All recoverable run state should live in an external store, typically a relational database for runs, threads, and checkpoints, plus a fast store for ephemeral data about ongoing runs and streaming. Any worker should be able to resume any run from its last persisted checkpoint.
Run work asynchronously with worker pools
A worker pool can be described as workers multiplied by jobs per worker. The numbers below are an illustrative assumption, not a product default: suppose 5 workers at 10 jobs per worker provide 50 available job slots. Throughput per second is then slots divided by average occupied execution seconds, assuming runs retain a slot continuously for that time, complete at that average pace, and downstream limits do not bind first. If runs suspend durably and release slots during waits, this slots-divided-by-duration calculation does not apply to wall-clock run duration; size on occupied execution time instead. Tune jobs per worker to the workload: raise it for assistants that mostly wait on the model, lower it for CPU bound work. Setting it too high in bursty traffic can overload downstream systems, and blocking calls stall the event loop, so workers should use asynchronous execution and timeouts for tool calls.
Autoscale on the leading signal
When agent workers spend most of their time waiting on model or tool responses rather than doing CPU work, they are input/output bound for that workload, and CPU can be a poor autoscaling signal. When a workload is CPU-heavy instead, CPU may be the right signal. Check which applies from observed waiting versus CPU work, then scale on the leading signal for that workload, such as queue depth, in-flight runs, or provider budget headroom. A container autoscaler such as Kubernetes Horizontal Pod Autoscaling can scale replicas on resource, custom, or external metrics, and queue-driven scaling can scale a pool down when the queue is empty. The exact metrics, thresholds, and stabilisation behaviour should be validated against your own platform rather than copied from another deployment.
Route model calls through a gateway
A central model gateway should track remaining budget per provider and per model, apply token-bucket or leaky-bucket admission per tenant and per model, and handle retries with exponential backoff and jitter on retryable 429 and 5xx responses. It can also apply prompt caching, model routing, and per-run token and step budgets.
Routing matters because provider limits are multi-dimensional. Typical dimensions include requests per minute, input tokens per minute, output tokens per minute, concurrent requests or sessions, and spend caps. Hitting any one dimension fails the call, so the gateway must check each applicable dimension separately, not just requests per second. A spend-cap rejection differs from a transient rate-limit rejection: if the rejection is caused by a spend cap, do not retry blindly. Check billing and budget state first.
State, memory, tool calls, and long-running runs
Short-term state for one run includes the conversation, intermediate results, and checkpoints. It must be recoverable from a durable store, not only from worker memory. Long-term memory includes summaries, preferences, and facts across threads and sessions. Every run may read and write memory, so memory-store throughput and consistency become a concurrency factor in their own right.
Tool calls deserve the same care as model calls. Each tool call is a new failure, latency, rate-limit, and side-effect surface. Tools should run as durable steps with timeouts and heartbeats, execute in a sandbox where code execution is involved, and be rate-limited independently from the language model. A fast model loop can otherwise hammer a slow tool or API.
Long-running runs that wait for humans, tools, or approvals must not depend on holding a worker thread or HTTP connection for the whole wait. Use durable waits, signals, interrupts, and checkpoint-resume for very long histories. Checkpoint frequency is a direct throughput lever: more frequent checkpoints give more recovery points but more write load; less frequent checkpoints reduce write load but mean more rework after a crash. For very long threads, summarise or compact context, prune old threads, and set time-to-live rules, because context growth raises storage cost, read latency, and per-step token cost.
Practical example: sizing for 50 concurrent runs
Consider a support agent that should sustain 50 concurrent runs. The assumptions for this example are explicit: each run averages 20 seconds of occupied execution time and 8 model steps, each step makes one model call, and each step averages 2,000 input tokens and 500 output tokens. The 20 seconds is slot-held, occupied execution time, not wall-clock time including durably suspended waits, and this example assumes continuous slot occupancy throughout each run. The 20-second average is assumed to remain achievable under load; if latency degrades under load, the throughput figures below change.
First, size the worker slots. Using the illustrative assumption of 5 workers at 10 jobs per worker provides 50 available slots. Throughput is then 50 / 20 seconds of occupied execution time, or 2.5 runs per second at steady state under those assumptions, including continuous slot occupancy.
Next, budget the provider, dimension by dimension. One run uses 8 × 2,500 = 20,000 tokens. Tokens per minute = 50 × 8 × 2,500 × 60 / 20 = 3,000,000 combined tokens per minute. Split by direction, that is 2,400,000 input tokens per minute (50 × 8 × 2,000 × 60 / 20) and 600,000 output tokens per minute (50 × 8 × 500 × 60 / 20). At one model request per step, requests per minute = 50 × 8 × 60 / 20 = 1,200. Check each figure against your own quota for that dimension. A combined token total that fits is not sufficient if the input, output, or request dimension exceeds its own limit. If any dimension exceeds its quota, the queue will grow even if your workers are idle.
Then budget the state store. If each step writes a checkpoint, 50 concurrent runs at 8 steps each produce 400 checkpoint writes per run batch, or about 20 writes per second at the assumed pace. That is manageable for many databases, but it becomes the bottleneck if checkpoint size is large, if every subagent writes separately, or if the database also serves reads for streaming and memory. If writes bind first, reduce checkpoint frequency, compact context, or scale the state store before adding more workers.
Finally, decide what breaks first in your own system. There is no fixed bottleneck order that holds for every workload: in one deployment the provider budget may bind first, in another the checkpoint or database throughput, or a tool API limit. A sizing-review checklist is: per-run step and token budget, per-tenant quota, gateway retry and caching policy, queue depth and age alerts, checkpoint write rate, tool timeouts and rate limits, and a kill switch for runaway loops.
What to check for your own deployment
Provider limits change frequently by tier, quota, region, model, and pricing, so treat any figure in an older article as stale until you confirm it. Before sizing, confirm the dimensions for your deployment: requests per minute, input and output tokens per minute, concurrent requests or sessions, new sessions per second, timeouts for synchronous, streaming, and asynchronous work, spend caps, and whether batch and interactive work draw on separate limit pools. Confirm the scope too: limits may apply per subscription, per project, per region, per model, per deployment, or per account.
Use the provider documentation as the place to check those dimensions, for example the OpenAI Batch documentation for latency-tolerant batch work, the Anthropic rate-limit documentation, the Microsoft Foundry Models quotas and limits, the Google Cloud generative AI quotas, and the AWS Bedrock AgentCore quotas. Check your own console for the tier and quota that actually apply to you.
Managed runtimes and self-hosted frameworks differ in how much they handle. A managed runtime may handle some scaling and durability plumbing, but the application still owns its per-tenant budgets, upstream-limit handling, per-thread concurrency control, and side-effect correctness. A self-hosted setup gives control over workers, queues, and state, but you own the scaling logic. Batch-only processing fits when latency does not matter; it does not fit interactive runs.
Failure modes, durability, idempotency, and per-thread concurrency
The most damaging failure is a retry that creates a second effect. A tool may have executed even though the caller saw a timeout. A worker may crash after a charge, email, or ticket was created but before the result was recorded. Without durable state, the run restarts and creates the effect again, or diverges by booking a different option on the second attempt.
Durable execution narrows this risk but does not eliminate the crash window: the gap between an external operation completing and its result being durably persisted. A crash inside that gap means a retry can repeat the effect or charge. Replay guarantees therefore apply only to successfully persisted results. In durable-execution systems, completed steps whose results are recorded in history or checkpoints are reused rather than re-executed, as described in Temporal’s account of building dynamic AI agents. Distinguish workflow replay, which reuses persisted decisions, from activity retries, which re-execute an operation whose result was never recorded.
For external side effects, add application-level idempotency backed by the destination. Use a stable idempotency key per run, step, and tool call, plus destination-enforced deduplication or atomic handling such as first-writer-wins or a conditional write. An upsert or keyed write is only idempotent when its key, its update semantics, and the destination’s guarantees together make repeated execution equivalent to a single execution. A pure read is generally safe to repeat; a blind insert or append is generally not, because a retry adds a second row or entry. Do not assume checkpointing alone guarantees that every completed tool result was recorded.
Concurrent conflicting updates need a separate control from deduplication. Deduplication stops a retried operation from applying twice; it does not resolve two different runs updating the same thread at once. Use one of two controls. The first is per-thread serialisation: route all work for one thread through a single-writer queue, lease, or lock so only one writer acts on that thread at a time. The second is versioned optimistic concurrency: each write carries the version it was based on, the store accepts it only if the version still matches, and on conflict the writer reloads the latest state, merges or re-applies its change, or surfaces the conflict instead of silently overwriting. Choose serialisation when ordering matters; choose optimistic concurrency when conflicts are rare and retries are cheap.
Other failure modes include provider 429 responses mid-run, worker crash mid-run, tool timeout, and checkpoint-store saturation. Mitigations include retrying the affected step rather than restarting the whole run, jitter and backoff, and step, token, and time budgets with kill switches.
Limitations and trade-offs
More workers and higher jobs per worker raise concurrent runs and reduce queue wait, but they also increase provider pressure, checkpoint and database load, and cost. Asynchronous execution with a queue adds resilience and bounded burst absorption, but it adds latency and operational complexity for the queue, dead-letter handling, and observability.
Durable execution improves crash recovery and lets a resumed run reuse persisted step results, but it adds checkpoint overhead and determinism constraints. Cost savings apply only to steps whose results were actually persisted; a step lost in the crash window can still be re-executed and re-billed. Prompt caching can raise effective input-token headroom only if the applicable provider excludes or discounts cache hits for that rate-limit dimension; a cache hit can affect cost or latency without freeing the input-token quota used in the sizing example. Confirm that provider’s cache accounting before reducing a calculated quota requirement. Caching still needs cache management. Batch processing trades turnaround time for throughput over a longer window and is not suitable for interactive runs.
Provider caps differ and are not guaranteed minimums; check your own console before sizing. Timeouts still apply for long runs. Cold starts, cost runaway, and observability gaps for per-run token and step attribution remain open problems even with a good reference architecture.
Conclusion
Scaling AI agents is not a matter of adding replicas until CPU falls. It is a matter of designing for the run: persist authoritative state externally, make every side effect idempotent at the destination, control concurrent updates to the same thread, and budget tokens before replicas.
For a production readiness check, use four rules. First, no authoritative run state lives only in worker memory; transient execution state is fine, but recovery must come from persisted state. Second, every tool call that can change the world has a stable idempotency key and a destination-enforced deduplication or atomicity path, plus a safe retry path. Third, concurrent updates to the same thread are serialised or version-checked, not left to last-writer-wins. Fourth, every tenant and model has a token, step, and cost budget, with alerts on queue depth, queue age, retry rate, and checkpoint write rate.
Teams that get this right do not eliminate failures. They make failures cheap, visible, and recoverable: a step retries without duplicating an effect where destination deduplication holds, a bounded burst queues instead of collapsing, and a crash resumes from the last persisted checkpoint instead of restarting an expensive run.
