Grading a single model response is a manageable problem: feed an input-output pair to a scorer and check accuracy, relevance, or safety. The discipline of ai agent performance evaluation starts from a harder truth — an agent doesn’t produce one output, it produces a trajectory: a sequence of model outputs, tool calls, tool returns, and intermediate state across many turns, capped by a final outcome. Measure only the reply and you are grading the trailer, not the film.
The same task can succeed twice for entirely different reasons. An agent might complete a refund correctly because it followed the right procedure, or because it stumbled through three wrong tool calls and guessed its way to the right end state. The first is a pass. The second is an incident that hasn’t happened yet:
The agent that reached a fine answer through three wrong tool calls is a latent incident, not a pass.
— James O’Connor, “Evaluating an AI agent is not evaluating an LLM call”
So good agent performance means four things, in order: it completed the goal, it does so repeatably, it doesn’t burn time or money getting there, and it stays within its evidence and authority.
Why AI agent performance evaluation is a different problem than model evaluation
Model evaluation grades one input-output pair. Agent evaluation must grade the entire trajectory — the ordered sequence of reasoning steps, tool calls, observations, and intermediate decisions — plus the final outcome. That difference changes what you record: you need the trace, not just the transcript. The trace is what changed in your systems and which tools were called, not only what the agent said:
Grade the outcome and the steps, not the reply. Agent evaluation checks what changed in your systems and which tools were called, not only what the agent said.
— Codiste, “AI Agent Evaluation Framework: Metrics, Test Sets and CI Gate”
This is why the standard playbook for model evals falls short. Prompt-level benchmarks test whether the model knows the answer; agent evals test whether the system reaches the right end state, through valid steps, without breaking anything along the way.
The metric ladder: what to measure, from outcomes inward
The standard eight-metric ladder works from outcomes inward:
- Task success. Did the agent complete the goal? Grade on the end state — the ticket closed, the record updated, the patch applied — not on how the final message was worded.
- Tool selection accuracy. Did it call the right tools in the right order, including graceful recovery when a tool failed?
- Tool parameter accuracy. Were the arguments it passed valid and correct for the chosen tool?
- Consistency across runs (pass^k). Does it succeed repeatably? This is the metric most teams miss.
- Groundedness. Are its claims supported by retrieved data rather than invented?
- Latency. End-to-end time per task, reported as p50 and p95 — agents take many turns, so averages hide the tail.
- Cost per task. Model tokens plus any tool fees per completed task.
- Handoff/escalation rate. How often the agent routes to a human — a high rate can mean good caution or a broken agent, so read it alongside task success.
Single-run numbers lie: an agent that succeeds 75% of the time succeeds in all three of three tries only about 42% of the time. pass@k measures at-least-one success in k trials (capability); pass^k measures success in all k trials (reliability) — a formulation introduced by Sierra with tau-bench in June 2024. Here is the math as a small helper for an eval harness:
import math
def pass_at_k(n: int, c: int, k: int) -> float:
"""Fraction of tasks where at least one of k runs succeeds.
n = total runs, c = successful runs."""
if c == 0:
return 0.0
if n - c < k:
return 1.0
return 1.0 - math.comb(n - c, k) / math.comb(n, k)
def pass_pow_k(p: float, k: int) -> float:
"""Estimated probability that ALL k runs succeed,
given per-run success rate p."""
return p ** k
# The consistency trap from the research: p = 0.75
print(pass_pow_k(0.75, 3)) # ~0.42 -- reliable less than half the timeReport capability and reliability side by side. A high pass@k with a low pass^k is an agent that can do the job but can’t be trusted to do it.
Build the test set before you measure anything
No test set, no evaluation. A good set starts small: 20 to 50 tasks drawn from real requests and real failures — Anthropic’s recommended starting range. Each task needs defined ground truth (what the correct end state looks like) and environment fixtures (seeded data, mocked services) so the run is repeatable. Include failure-mode cases on purpose: the API timeout, the ambiguous instruction, the malformed input — agents meet these in production, so they must meet them in evals too.
Organize the set into two suites with different jobs:
- Capability suite. Hard tasks with a low pass rate that measures agent quality and rises as the agent improves.
- Regression suite. A near-100% pass rate suite where any drop blocks release. Every production bug becomes a regression test here, so the set grows with real failures.
A manifest makes the split explicit and keeps it honest as the suite grows:
{
"capability_suite": {
"purpose": "measure agent quality on hard, realistic tasks",
"expected_pass_rate": "low -- moves up as the agent improves",
"task_count": 40,
"source": "real requests, edge cases, multi-step workflows"
},
"regression_suite": {
"purpose": "block releases on regressions",
"expected_pass_rate": "near-100% -- any drop blocks the release",
"task_count": 60,
"source": "every production bug, converted into a test"
},
"fixture_policy": "each task carries defined ground truth and seeded environment fixtures",
"run_policy": "every task runs multiple times; consistency metrics are computed, not eyeballed"
}Run the loop: execute, grade, and calibrate your judges
The eval loop has three phases: execute, grade, calibrate.
Execute. Run each task multiple times — one run cannot establish reliability or give you a consistency estimate for a non-deterministic agent — and capture the full trace every time: model outputs, tool calls, tool returns, intermediate state.
Grade. Apply deterministic checks first: end state, tool calls, arguments — cheap, exact, unambiguous. Reserve LLM-as-a-judge, a separate model scoring outputs against criteria, for what determinism can’t grade: reasoning quality, answer usefulness, groundedness. Run judges inline or in batch; use them as triage, flagging outputs for human review rather than making final determinations.
Calibrate. An uncalibrated judge is the most common eval bug in the wild: a high-variance rubric nobody calibrated, treated as gospel. Calibrate concretely: label a human-reviewed golden set of 50–100 examples per rubric, measure judge-human agreement (Cohen’s kappa or percent agreement), and recalibrate at least quarterly — or whenever the judge model, rubric prompt, or product behavior changes. Agreement below roughly 75% means the rubric needs work, not that the agent is bad. Version the rubric so calibration stays reproducible:
{
"rubric": "groundedness",
"scale": "binary",
"pass_criteria": "Every factual claim in the agent's final answer is supported by a tool return in the trace.",
"fail_criteria": "At least one claim cannot be traced to a tool return in the trace.",
"calibration": {
"golden_set_size": 100,
"minimum_judge_human_agreement": 0.75,
"recalibrate": "quarterly, or when the judge model, rubric prompt, or product behavior changes"
}
}Judges carry documented biases: position bias favors whichever answer appears first (Zheng et al., 2023, MT-Bench), verbosity bias rates longer answers higher regardless of accuracy, and self-preference rates the judge’s own model family higher (Panickssery et al., 2024). Mitigations: shuffle answer order, cap response length, and use a different model family as judge. And remember the core problem judges are fighting:
Here’s the core problem: LLMs are trained to generate text, not to score it consistently. Scoring requires applying a fixed rubric the same way every time. LLMs don’t do that.
— DataFramer, “LLM-as-Judge: Why It’s Hard to Get Right”
The tooling landscape: frameworks that already implement this
You don’t need to build metric libraries from scratch — but read licenses carefully, because “open-source” is used loosely here:
- DeepEval (github.com/confident-ai/deepeval) — active open-source, pytest-style, built for CI-integrated agent evals. Works at end-to-end, trajectory-level, and component-level scopes, with metrics like TaskCompletionMetric, StepEfficiencyMetric, ToolCorrectnessMetric, and ArgumentCorrectnessMetric. Runs LLM-as-a-judge and other metrics locally; a managed Confident AI tier exists for collaboration.
- RAGAS (github.com/explodinggradients/ragas) — active open-source, best for RAG evaluation: faithfulness, answer relevancy, and context precision/recall, including reference-free metrics. Reach for it when your agent’s weak point is retrieval, not tool use.
- LangSmith — a managed (non-open-source) platform, active per current docs, native to the LangChain stack. Datasets, examples, targets, evaluators, experiments; offline evals on curated datasets, online evals on sampled production traces, at run, trace, and thread level. For LLM-as-a-judge it recommends binary pass/fail over numeric scales plus regular recalibration — judges drift like the agents they evaluate.
- Braintrust — a commercial eval platform (not open source), experiment-first: datasets, scorers, and GitHub Action gates for CI.
- Arize Phoenix (Arize-ai/phoenix, roughly 10,000 GitHub stars, Elastic License 2.0 — not true open source) — observability-first. OpenTelemetry-native traces with LLM-as-judge evals on collected data. Best for trace exploration and debugging before you know what to eval.
Consider building your own harness when, for example, grading is dominated by domain-specific end-state checks (a custom simulator, a proprietary tool) no framework covers. Even then, consider adopting a framework’s trace format and judge plumbing where they fit — the metric library is the cheap part; trace capture and calibration discipline are the work.
Benchmarks for comparing agents, not just improving one
Internal evals tell you whether your agent is improving. Public benchmarks tell you how it compares — with caveats.
GAIA (Meta FAIR / HuggingFace) tests general agent capabilities: 466 questions across 3 difficulty levels covering web browsing, file processing, calculation, and multi-step reasoning, scored by exact match. The human baseline is 92%; at launch in late 2023, GPT-4 with plugins scored 15%. On Princeton’s HAL verified leaderboard, the HAL Generalist Agent with Claude Sonnet 4.5 reached 74.55% overall (L1 82.07%, L2 72.68%, L3 65.39%) at a verified $178.20 in API cost for the run.
SWE-bench Verified (Princeton NLP / OpenAI) is the coding-agent benchmark: 500 verified instances drawn from 12 Python repos. The agent receives a GitHub issue plus the full repository and must produce a patch — an instance counts as resolved only if all relevant tests pass and no existing tests break.
Read benchmark scores honestly: boards publish different snapshots under different verification standards. HAL reproduces scores with confidence intervals and cost figures; other leaderboards mix self-reported and maintainer-verified entries, and top-system standings were ambiguous across boards in this research. Compare scores only within the same board and verification standard, and treat any single #1 as a snapshot, not a title.
Common mistakes that make evals lie to you
Three traps invalidate more agent evals than any technical limitation:
- Grading only the final message. A polished answer can hide wrong tools, wrong records, or skipped steps. Trajectory scoring of tool selection, argument validity, and graceful recovery catches agent-specific failures that final-answer accuracy waves through:
If your evaluation only reads the last message, it will pass that agent.
— Codiste, “AI Agent Evaluation Framework: Metrics, Test Sets and CI Gate”
- Trusting single-run results. Agents are non-deterministic. One pass tells you almost nothing — without repeated runs you have no consistency signal, and a 75% agent looks identical to a 99% agent on a lucky day.
- Running an uncalibrated judge. A high-variance rubric nobody calibrated, treated as gospel, is the most common eval bug in the wild. Below roughly 75% judge-human agreement, fix the rubric before trusting a single grade.
A fourth trap is quieter: test sets that never include failure modes. A suite of clean, well-specified tasks will green-light an agent that collapses on its first ambiguous instruction or timeout in production. Convert real failures into regression tests and your evals measure the agent your users actually meet.
Evaluate the trajectory, not the reply. Measure consistency, not luck. Calibrate the judge, don’t trust it. Do those three and your evals will tell you the truth about your agent.
