When an AI agent sets out to refund an order, book a flight, or file a report, something can go wrong: a tool times out, an API errors, or the model calls the right tool with the wrong arguments. Good AI agent error handling is about what the agent does in the seconds after one happens. This article explains the full recovery loop: detecting errors, retrying, repairing, replanning and falling back, knowing when to ask a human, and the guardrails that bound the damage.
What counts as an error in an agent task
Not every failure looks the same, and the kind decides the response. There are four kinds worth distinguishing. First, the tool itself fails: the API returns an error, the request times out, the credentials have expired, or the service is rate-limiting the agent. Second, the model is wrong: it calls the wrong tool, or the right tool with invalid arguments — a missing field, a malformed ID, a value outside the allowed set. Third, the call succeeds but the result is wrong: stale data, an empty result, or a value that does not match what the task needs. Fourth, the world changed mid-task: the data moved, the service went down, or a permission was revoked.
Each kind needs a different response: retrying a bad argument never helps; retrying a timeout often does. In BFCL V4, the Berkeley Function Calling Leaderboard, multi-turn accuracy runs well below overall accuracy for every leading model: Claude Opus 4.5 scores 77.47% overall but 68.38% on multi-turn calls, Claude Sonnet 4.5 scores 73.24% overall but 61.37% multi-turn, and Gemini 3 Pro Preview scores 72.51% overall but 60.75% multi-turn. The benchmark weights multi-turn at 30% of the overall score and agentic tasks at 40%. Failures happen often enough in multi-turn tool use that the agent needs a procedure for them.
How the agent detects something went wrong
Recovery starts with detection, and there are three channels. The first is error signals from the tool or API itself. Modern APIs return structured errors: a machine-readable code the program can branch on and a human-readable message for the log. OpenAI’s Agents API guidance puts it plainly: “For structured errors, use error.code in application logic and error.message to explain the failure.” A validation error also carries error.param, which identifies the field to correct.
The second channel is the tool result itself. In Anthropic’s tool-use protocol, a tool that ran and returned an error sends its result back with an is_error flag set, plus a short explanation of what failed and what the model can try next:
{
"type": "tool_result",
"tool_use_id": "toolu_01A09q90qw90lq",
"content": "Inventory lookup timed out after 30 seconds; no reservation was submitted. Retry the lookup, or check availability with the cached list instead.",
"is_error": true
}Describe what failed and what to try next, rather than returning a generic “failed”. If a parallel call is deliberately skipped, return an error result for that call so the turn is properly accounted for. The third channel is validation and self-checking: a technically successful call can still be wrong — a lookup that returns the wrong record, a plausible-looking but incorrect value. The agent catches those by checking the result against the expected outcome before building on it. Without detection, the agent happily continues on bad state.
The recovery loop: retry, repair, replan, fallback
Once a failure is detected, the agent works through an ordered loop. Retry the same call when the failure looks transient — a timeout, a rate limit, a brief outage. Repair the call when the call itself was wrong: fix the arguments, or pick a different tool. Replan when the approach itself is wrong: the tool cannot do what the task needs. Fall back when nothing works: a simpler method, a partial result, or an honest report that the step could not be completed. Order depends on the failure type: for a transient failure, replanning first wastes a cheap retry; for a wrong call, retrying it wastes the whole loop.
In practice the model itself drives the loop by reading the error. In LangGraph, the ToolNode’s handle_tool_errors option catches a tool exception and returns the error message as a ToolMessage, so the model sees what went wrong and adjusts. As the LangGraph error-handling guide puts it: “The fix isn’t retrying the exact same call — it’s letting the LLM see what went wrong and try a different approach.” The framework turns the failure into information; the model acts on it.
from langchain_core.tools import tool
from langgraph.prebuilt import ToolNode
@tool
def get_order_status(order_id: str) -> str:
"""Look up an order by its ID."""
if not order_id.startswith("ORD-"):
raise ValueError(f"Unknown order format: {order_id!r}. IDs look like ORD-1042.")
return "ORD-1042: shipped, arriving Thursday."
def format_tool_error(error: Exception) -> str:
return f"Tool failed with: {error}. Review your arguments and try again."
tool_node = ToolNode([get_order_status], handle_tool_errors=format_tool_error)With that wiring, a bad argument produces a ToolMessage the model can read and fix.
Retry strategies that actually work
Retries only help when they are aimed at the right failures. The first step is classification: sort every failure into retryable or terminal. In OpenAI-compatible APIs, the error body carries a machine-readable type to branch on: 429 rate_limit_exceeded and 500 api_error are retryable; 400 invalid_request_error, 401 and 402 insufficient_quota are terminal. Retrying a terminal error just burns quota and time.
{
"error": {
"code": null,
"message": "Insufficient quota. Please upgrade your plan.",
"param": null,
"type": "insufficient_quota"
}
}For the retryable kind, the standard discipline is exponential backoff with jitter and a cap: wait 1 second, then 2, then 4, with jitter so many workers do not retry at once. LangGraph’s published retry classification makes it concrete: a network timeout gets exponential backoff over 3 attempts (1s, 2s, 4s), rate limiting gets a fixed delay over 5 attempts at 30-second intervals, authentication gets no retry, and a service-unavailable error gets linear backoff over 3 attempts (5s, 10s, 15s).
import random
import time
class ApiError(Exception):
"""An HTTP status plus a machine-readable error type."""
def __init__(self, status, error_type):
super().__init__(f"{status} {error_type}")
self.status, self.error_type = status, error_type
RETRYABLE = {429, 500, 502, 503, 504}
def call_with_backoff(fn, max_attempts=4):
"""Retry fn() only when the failure status is retryable."""
for attempt in range(1, max_attempts + 1):
try:
return fn()
except ApiError as exc:
if exc.status not in RETRYABLE or attempt == max_attempts:
raise
delay = min(0.75 * (2 ** (attempt - 1)) + random.uniform(0.2, 0.9), 8)
time.sleep(delay)The example classifies by HTTP status, not message text: 429 or 500 is retried, 400 or 401 raised at once. Two rules sit underneath the code. First, know what the retry is for: a transient failure may succeed after waiting — that is what backoff is for. A wrong call repeated unchanged fails the same way, so change the arguments or the tool. Second, make retries safe: if the call has side effects, store a job ID before the call and reuse it on every retry, so only one final action is ever committed.
When the agent should stop and ask a human
Recovery has a limit, and the limit should be a number, not a feeling. The simplest hard stop is an iteration cap: a maximum number of turns or retries after which the agent stops and hands over. Microsoft’s .NET agent framework caps a request at 40 round trips and 3 consecutive errors, and gives each tool an invalid-argument budget of 2 per request, after which the error tells the model to ask the user instead. LangGraph’s production pattern uses 3 retries per tool node, then a fallback response, with timeouts at three levels: 30 seconds per model call, 60 per node, 5 minutes for the whole graph on customer-facing work.
The triggers for escalation are the same across frameworks: the same failure repeats after the retry budget is spent, the next step would be destructive or irreversible, or the situation is ambiguous in a way the agent cannot resolve — two tools disagree, or the error points at a decision only a human can make. OpenAI’s guidance adds a further stop: “Stop automatic retries if the error changes or the retry limit is reached.” A good handoff says what failed, what was tried, and what state was left behind, so the human can pick up the task.
Guardrails that keep errors from becoming damage
Recovery handles failures after they happen. Guardrails bound how much damage a failure can do in the first place. The first is validation before the call goes out: check the arguments against the tool’s schema and the values against what the task needs. The second is least privilege: every tool gets only the permissions its job requires, and destructive actions sit behind an approval gate that a human must clear. The third is sandboxing and dry runs for risky operations — run them where a mistake is cheap.
The fourth guardrail is the one teams forget: check what already happened before repeating anything. OpenAI’s retry guidance states it directly: “Check completed actions. A failed turn may already have changed files or called external tools. Confirm those effects before asking the agent to repeat work.” The same rule applies to interrupted workers in Anthropic’s guidance: before re-running a non-idempotent action — a payment, a deletion, an email — check a durable call record using the request ID, an idempotency key, or application state. A missing tool result does not prove the tool never ran; re-running it on that assumption is how duplicate charges happen.
A worked example: a failed tool call, recovered end to end
Put the pieces together in one realistic task: a customer asks an agent to refund order 1042. The agent’s first call is get_order_status with order_id “1042”. The tool raises a ValueError — IDs look like ORD-1042 — and the ToolNode returns that error as a ToolMessage. The model reads it, corrects the argument to “ORD-1042”, and the lookup succeeds: shipped, arriving Thursday. That is the repair step.
Next the agent calls initiate_refund, and the API answers 429 rate_limit_exceeded. This is retryable, so the agent waits 1 second and retries with the same stored job ID, so the refund cannot be applied twice. On the second attempt the refund goes through, and the agent reports the refund ID to the customer. Now imagine the second attempt had failed differently — say the refund tool reported that the order was already refunded. That is not a retryable failure and not a repairable one: the state of the world has changed. The right move is the fallback: check the order record, confirm the earlier refund, and tell the customer the refund is already in progress instead of trying again. Each step follows the same rule: detect the kind of failure, then pick the cheapest response that fits it.
Common mistakes in AI agent error handling
The most expensive mistake is the infinite retry loop: an agent that keeps calling a failing tool without making progress, burning tokens and time. The fix is the one already described — a hard iteration cap and a retry budget — built in before the first failure, not after the first runaway bill. The second is trusting tool output blindly: a result without an error is not necessarily correct, and building on an unchecked value is how a small error becomes a wrong refund. Validate results before acting on them.
The third is swallowing errors and reporting success. An agent that catches a failure, says nothing, and tells the user the task is done has turned a recoverable error into a lie the user will discover later, when it is harder to fix. If a step failed and no fallback worked, the honest report is the correct output. The fourth is retrying terminal errors: re-sending a request with invalid credentials, an exhausted quota, or a malformed body, and expecting a different result. Classification comes before retry, every time. And the fifth is re-running non-idempotent actions without checking what already happened. Check the durable record first; if there is no way to check, pause instead of guessing.
Error handling is what separates an agent that works in a demo from one that survives production. Failures can occur in any multi-turn task — on the BFCL benchmark, leading models score 61–68% on multi-turn calls, well below their overall accuracy. What matters is the procedure: detect the kind of failure, classify it as retryable or terminal, work the loop from retry to fallback, and stop at a hard cap with an honest handoff.
