AI agent evaluation is the discipline of measuring whether an agent actually does its job — not whether its demo looked impressive, but whether it completes real tasks reliably, at an acceptable cost, without breaking anything along the way. Without it you are flying blind: the first sign of a regression is a user complaint, and every fix risks breaking something else. This guide builds a measurement system from the ground up: what to measure, how to build the harness that measures it, which benchmarks to trust, and how to turn scores into a better agent.
What AI agent evaluation actually measures
An evaluation is a test for an AI system: give the agent an input, then apply grading logic to its output to measure success. Anthropic puts the value plainly: “Good evaluations help teams ship AI agents more confidently. Without them, it’s easy to get stuck in reactive loops — catching issues only in production, where fixing one failure creates others.”
That definition matters because an agent may operate over multiple turns, call tools, change the state of the world, and adapt based on intermediate results — and evaluating it means measuring the whole trajectory, not just the final message: a support agent that refunds the wrong order but writes a polite summary has failed, however fluent the summary reads. The measurement surface has five parts — task success, quality, cost, latency and safety — and a single score cannot carry all five. Two distinctions keep the system honest: outcome versus process, and demo versus evidence. Anthropic’s strongest engineering advice is to grade the final state of the environment, not what the agent says — check the refund in the database, not the agent’s claim that it issued one. And a good demo is not evidence: a demo is a rehearsed path through a known task, while evaluation scores the same agent on tasks it was not trained or tuned on, by rules written before it ran. Known regression cases still belong in the suite — they catch breakage — but they cannot prove the agent generalises.
Start from the task, not the metric
Metrics follow the work. Before choosing any metric, define what success looks like for each task the agent actually performs, in terms a second person could check. “Resolve the customer’s problem” is not a success criterion; “refund issued only for an eligible order, for the correct amount, with a confirmation sent” is. Anthropic recommends starting with 20 to 50 simple tasks drawn from real failures — the cases your agent already gets wrong.
Each task in the set needs the same shape: an input, the tools the agent may use, and written success criteria, including the things it must not do. A task definition for a refund agent might look like this:
{
"task_id": "refund-late-delivery-001",
"input": "A customer asks for a refund for order 48210, delivered 6 days late.",
"tools": ["get_order", "check_refund_policy", "issue_refund"],
"success_criteria": [
"refund issued only if the order is eligible under the policy",
"refund amount matches the order total",
"customer receives a confirmation message"
],
"should_not": [
"issue a refund for an ineligible order",
"reveal internal policy text to the customer"
]
}With the task set defined, the vanity-metric trap becomes visible: a metric is a vanity metric when it rises while user value does not — average response length, number of tool calls, or a benchmark score chosen because it was already high. If the metric doubled and nothing about the user’s experience changed, it is not measuring the job.
The core metric families for AI agent evaluation

Most agent scorecards draw from the same families. Task success rate is the headline: the share of tasks that meet their success criteria. Tool-call accuracy and planning quality score the sub-skills underneath success: did the agent call the right tool, with valid arguments, in a sensible order? Latency is end-to-end time per task, usually reported as P50 and P95. Token usage and cost per task capture the economics; error, retry and escalation rates show how often the agent stumbles; and user satisfaction, or sampled human review, checks what automated scores cannot.
Two reliability metrics deserve a closer look, because a single success rate hides them. Pass@k is the probability that at least one of k independent tries succeeds — the ceiling of what the agent can do. Pass^k is the probability that all k tries succeed — the consistency a user actually experiences. An agent that succeeds 70% of the time has a pass@3 of 97.3% but a pass^3 of only about 34%:
def pass_at_k(p, k):
"""Probability at least one of k independent tries succeeds."""
return 1.0 - (1.0 - p) ** k
def pass_pow_k(p, k):
"""Probability all k independent tries succeed."""
return p ** k
# An agent that succeeds 70% of the time, over 3 independent tries
print(pass_at_k(0.7, 3)) # 0.973
print(pass_pow_k(0.7, 3)) # 0.343Cost needs the same care. The unit that matters is cost per successful task, not cost per API call. An agent task is a goal — refund the customer, summarise the contract — and agents rarely solve tasks in one call: they plan, call tools, reflect and retry. A support resolution might take eight model calls, and the bill belongs to the resolution, not to any single call. A cheap model that succeeds 70% of the time can cost more per successful task than an expensive model that succeeds 95% of the time, because failures consume spend without adding a successful resolution, and retries or escalations can add further cost. Two facts shape the arithmetic: agents consume five to ten times more tokens than a zero-shot approach, because each step carries the accumulated context of prior steps, and output tokens cost three to five times more than input tokens on most models, so the two must be tracked separately.
How to build an evaluation harness

A harness is the infrastructure that runs evals end to end: it provides instructions and tools, runs tasks concurrently, records every step, grades outputs and aggregates results. A suite is a collection of tasks designed to measure specific capabilities. Treat both as product infrastructure, with owners, versioning and maintenance — in a survey of YC agent builders, a super-majority said evals under-deliver because keeping them up to date becomes impossible.
Grading is where the harness earns its keep, and Anthropic’s guide organises graders into three types. Code-based graders use string checks, fail-to-pass tests and final-state verification: fast, cheap and reproducible, but brittle against valid variations. Model-based graders use rubrics and pairwise comparison: flexible and scalable, but non-deterministic and needing calibration against humans. Human graders — expert review, spot checks, A/B tests — are the gold standard for quality and the slowest and most expensive option. Most useful setups combine all three, choosing the cheapest grader that can grade each dimension reliably.
LLM-as-judge, the model-based grader most teams reach for first, needs the most calibration. The original MT-Bench study found GPT-4 agreed with expert human graders 85% of the time on non-tie votes, about the same as the 81% agreement between two human experts — but counting ties, both figures fall to 66% and 63%. Agreement collapses on hard questions: on JudgeBench, pairs with an objectively correct answer, GPT-4o picked the right one only 50.86% of the time, while reasoning models did far better, with o3-mini at high effort reaching 80.86% and DeepSeek-R1 73.14%. Judges also have measurable biases: GPT-4 gave the same verdict after the two answers were swapped only 65% of the time, though few-shot examples in the prompt raised that to about 77.5%. A panel of three smaller judge models from different families outperformed a single large judge across six datasets and cost over seven times less.
The practical rule follows: validate the judge against human labels on your own data before trusting it, swap answer order, report Cohen’s kappa rather than raw percent agreement — percent agreement can hide score gaps of up to 20 points — and keep human review for hard and high-stakes items. In production, the honest version is sampling: score one to five percent of production traces with a judge model and track the trend.
Finally, run evals in CI so every change is scored before it ships, and split capability evals from regression evals. Capability evals ask whether the agent can do a kind of task yet and should include hard tasks; regression evals ask whether a change broke behaviour that used to work and should pass at nearly 100%. When a new model arrives, a team with evals can test it in days; a team without them faces weeks of manual testing.
Benchmarks worth knowing, and their limits
Public benchmarks are the shared scoreboard, and each measures a different slice of the job. SWE-bench gives agents 2,294 real GitHub issues from twelve Python repositories and counts a task solved only when the patch applies and the repository’s tests pass; its human-validated subset, SWE-bench Verified, has 500 tasks. GAIA is 466 general-assistant tasks across three difficulty levels; the human baseline is about 92%, while the best agents score about 50 to 55%. Tau-bench simulates multi-turn customer conversations — 115 retail and 50 airline — and scores whether the agent follows policy and reaches the right final state, reporting pass^k as its reliability metric.
The limits are as informative as the scores. The Agentic Benchmark Checklist found task-validity flaws in seven of ten agentic benchmarks it audited, and on tau-bench’s airline tasks an agent that returned empty responses scored 38%. In February 2026, the model developer that built SWE-bench Verified with the benchmark’s authors found flawed tests in at least 59.4% of 138 audited problems and stopped reporting the benchmark. Tau-bench itself is now unmaintained; its successor, tau2-bench — now tau3-bench — released version 1.0.0 in March 2026.
The deeper limit is that a benchmark score is not a score on your tasks. METR’s time-horizon research shows the gap: frontier models succeed almost 100% of the time on tasks taking human professionals under four minutes, but under 10% of the time on tasks taking over four hours. A benchmark of short tasks can look excellent while the same agent fails on the long tasks your business actually runs. Use benchmarks to compare models; use your own task set to decide whether to ship.
Evaluate the parts, not just the whole
When the overall score is bad, it does not say why. That is what traces are for. Log every step — reasoning, tool calls, retrieval results — and score the sub-skills separately: tool-call accuracy, planning quality, retrieval quality, and judgement. Evaluation frameworks make this concrete: DeepEval ships metrics named Tool Correctness, Argument Correctness, Task Completion, Step Efficiency and Plan Adherence, so a failure can be localised to the step that caused it.
A failure-mode taxonomy turns those traces into a work list: the wrong tool, bad arguments, a bad plan, bad retrieval, bad judgement, or a policy violation — each class pointing at a different fix. Agents can even find loopholes, as Anthropic’s Opus 4.5 did when it solved a tau2-bench flight-booking problem by exploiting a gap in the policy rather than following it.
Scoring the parts also exposes trade-offs a single score hides: Anthropic’s internal research evaluation found a multi-agent system outperformed a single-agent Claude Opus 4 by 90.2%, while using about fifteen times the tokens of a standard chat interaction. Outcome and cost metrics expose that trade-off, and sub-skill diagnostics help explain the performance difference.
From evaluation scores to a better agent
Measurement only pays off when it changes what gets built. Track scores over releases so regressions are visible the day they land. Compare variants on the same task set with one variable changing at a time. Then run error analysis: classify the failures, count them by class, and fix the biggest class first.
Keep the loop running: re-run the suite after every change, keep the regression suite green, and re-sample production traces weekly, especially after a score moves.
Common AI agent evaluation mistakes
Four mistakes cover most of the evals that look rigorous and measure the wrong thing. The first is evaluating on the training set, or tuning the agent to the test set: the score then measures memory, not capability. Tasks the agent has seen before still have a place — as regression cases that catch breakage — but a generalisation claim needs held-out tasks. The second is optimising a single metric until it stops meaning anything — an agent scored only on tickets closed will close tickets, whether or not it resolved them. The third is ignoring cost and latency, so an agent that takes fifteen times the tokens looks like a pure improvement. The fourth is having no baseline: a score of 72% means nothing without knowing what the previous version scored, or what a trivial approach scores. A do-nothing agent scoring 38% on a benchmark is the warning — without a baseline, even doing nothing can look like progress.
Done well, AI agent evaluation is unglamorous work: defined tasks, written success criteria, a harness that runs them, graders calibrated against humans, and a loop that turns failures into fixes — the difference between an agent you can ship and an agent you can only demo.












