When you test AI agents before deployment, you are not running a bigger version of a unit test suite. An agent takes an instruction, calls tools, reads the results, changes its mind, and tries again — and the same input can produce a different path every time. That variability is why traditional testing methods, while still useful for deterministic components, are insufficient on their own. This guide gives you a complete, repeatable method: a four-layer map of what to check, unit tests for the pieces, scenario-based evaluation for the whole agent, adversarial safety checks, and a release gate with explicit thresholds.
Why Testing AI Agents Is Different From Testing Software
Exact-match assertions are reliable checks for deterministic components tested with controlled inputs and state — the same input, the same code, the same expected output. Generative AI breaks that pattern: as OpenAI puts it, models sometimes produce different output from the same input, which makes traditional software testing methods insufficient for AI architectures on their own. Variable model outputs need additional evaluation methods — repeated trials, scenario-based evaluation, model-based grading — while deterministic checks stay useful for machine-verifiable requirements. An agent adds two more complications on top of that. First, it works over many turns, calling tools and modifying state as it goes, so a small mistake early in a run can propagate and compound into a completely wrong outcome. Second, the quality of the interaction itself is part of what you are testing: an agent that books the right flight but insults the customer has still failed.
This is why “it worked when I tried it” is not a test result. A single successful run proves almost nothing about a system whose outputs vary between runs. A task that passed on one eval run might fail on the next, and each task has its own success rate. Testing an agent means measuring a distribution of outcomes, not confirming one happy path.
What to Test When You Test AI Agents: The Four Layers
Agent testing breaks into four layers, and a deployment gate needs coverage in all of them.
Task success. Does the agent complete representative end-to-end tasks? This is the layer most teams start with, and it is the one that matters most to users. Success is judged against the final state of the environment, not the agent’s words: a flight-booking agent has succeeded when a reservation exists in the booking system, not when it says “your flight is booked.”
Tool use. Check tool use in two parts: the agent must pick the right tool, and it must call it with the correct arguments extracted from the conversation. A wrong argument to a real API is a real side effect, so this layer deserves deterministic, code-based graders wherever the expected result is machine-verifiable.
Safety and guardrails. Does the agent refuse out-of-scope or disallowed actions, resist prompt injection, and fail gracefully when tools are down? Routine quality checks rarely expose misuse, so adversarial cases belong in the suite from the start, not as an afterthought.
Cost, latency and reliability. How long does a run take, how many tokens does it burn, how often does it retry? Once evals exist, you get baselines for latency, token usage, cost per task and error rates on a static bank of tasks almost for free — and those baselines are what let you spot a regression caused by a prompt change that “only” made the agent more thorough.
Unit Testing Individual Agent Components
Test prompts and system instructions with fixed inputs, the way you would test a function: a given instruction and a given user message should produce a response in the expected shape, calling the expected tool. Test tool argument construction by mocking the tools, so you can verify the agent builds correct calls without touching live systems. Test retrieval, memory and state handling separately — for example, that the agent recalls the order ID from three turns ago.
Unit tests are also where deterministic graders shine. Where the expected result is machine-verifiable — a record ID, a status, an amount, a required field, whether a tool was called — a code-based check gives you a fast, unambiguous pass or fail with no judge model involved.
End-to-End Evaluation With Test Scenarios
The core of pre-deployment testing is scenario-based evaluation. Anthropic’s guidance is to start with 20 to 50 simple tasks drawn from real failures — your bug tracker, your support queue, the cases that embarrassed you in demos. That small, well-chosen set delivers most of the early value. Build the tasks from real usage: ordinary requests first, then the awkward ones — incomplete context, contradictory instructions, unavailable tools, ambiguous names, stale records, duplicate events, requests that should be refused.
Each task needs defined inputs and unambiguous success criteria. A good task is one where two domain experts would independently reach the same pass or fail verdict, and where everything the grader checks is clear from the task description. Write a reference solution for each task: a known working output that passes all graders. That proves the task is solvable and verifies your graders are configured correctly.
Here is a worked example — a customer-support agent that processes refunds. The scenario is a JSON test case: the input, the tools the agent may use, the expected final state, and the pass criteria. The fixture below is explicitly illustrative — invented order numbers and amounts standing in for your own data — so the example is reproducible as written.
{
"task_id": "refund-ambiguous-order",
"input": "I was charged twice for order 48210 and I want my money back.",
"tools_available": ["lookup_order", "issue_refund", "escalate_to_human"],
"fixture": {
"note": "Illustrative initial state. Order 48210 has two charges of $49.00 (ch_001 and ch_002); only ch_002 is the duplicate and refundable. Order 48211 has one charge of $29.00 and must stay untouched. No refunds have been issued yet.",
"orders": {
"48210": {"charges": [{"id": "ch_001", "amount": 49.00}, {"id": "ch_002", "amount": 49.00, "duplicate": true}], "refunds_issued": []},
"48211": {"charges": [{"id": "ch_003", "amount": 29.00}], "refunds_issued": []}
}
},
"mocked_tool_responses": {
"lookup_order('48210')": {"charges": ["ch_001", "ch_002"], "amounts": [49.00, 49.00], "duplicate_charge_id": "ch_002"},
"lookup_order('48211')": {"charges": ["ch_003"], "amounts": [29.00]}
},
"expected_outcome": {
"refund_issued": true,
"order_id": "48210",
"refund_amount": 49.00,
"refunded_charge_id": "ch_002",
"total_refunds": 1,
"refunds_for_order_48211": 0
},
"graders": {
"code_based": [
"refund_issued == true",
"refunded_order_id == '48210'",
"refund_amount == 49.00",
"refunded_charge_id == 'ch_002'",
"count(refunds) == 1",
"refunds_for_order('48211') == 0",
"trace_contains('lookup_order', '48210') before trace_contains('issue_refund')"
],
"model_based": "rubric (0-5): tone is calm and professional, confirmation states the refund amount; score >= 4 required",
"human": "sampled review of the full transcript before sign-off"
},
"verdict_rule": "The trial passes only when every code-based check passes AND the model-based rubric scores at least 4 out of 5. Human transcript review is required before sign-off but is not part of the automated verdict."
}Notice what the scenario grades: the outcome in the environment (the refund exists, for the right order, for the right amount, and only once), the trace (did the agent look the order up before refunding?), and the interaction quality (tone, confirmation). The verdict rule is strict on purpose: the trial passes only when every code-based check passes and the rubric scores at least 4 out of 5. Grade what the agent produced, not the exact path it took — except where order matters for safety, as with looking a charge up before refunding it.
Run each task several times. Because model outputs vary between runs, Anthropic recommends reporting two metrics: pass@k, the likelihood the agent gets at least one correct solution in k attempts, and pass^k, the probability that all k trials succeed. As a concrete example, an agent with a 75% per-trial success rate run three times has a pass^3 of about 42%.
Testing Safety, Guardrails and Failure Modes
Safety testing is adversarial by design. Write prompts that try to make the agent do things it must not do: follow instructions hidden in a web page or document it retrieved, reveal system instructions, refund an order it has no authority over, or act on data from an untrusted source. Test both directions — cases where a behaviour should occur and cases where it should not. One-sided evals create one-sided optimisation: test only whether the agent searches when it should, and you may end up with an agent that searches for almost everything.
Also test failure, not just success. Kill a tool mid-run and check the agent reports the problem instead of inventing a result. Feed it a malformed tool response. Give it a task it cannot complete and check that it escalates to a human rather than hallucinating a resolution. A useful bar for any agent is threefold: do no harm to the system state, do what was asked, and do it well. For conversational agents, an illustrative design is to have a second model play the user persona across multi-turn interactions — for example, a simulated customer who changes their mind mid-conversation while the agent handles a refund — and grade the trace for whether the agent held the policy, confirmed the change, and escalated when it should have.
Human review stays in the loop here. Sample transcripts and read them: when a task fails, the transcript tells you whether the agent made a genuine mistake or your grader rejected a valid solution. That reading habit is also how you catch grader bugs — Anthropic reports that Opus 4.5 initially scored 42% on CORE-Bench until researchers found rigid grading, ambiguous task specs and stochastic tasks; after fixing the eval itself, the score jumped to 95%.
Running Tests in CI and Deciding When an Agent Is Ready
An eval suite that only runs by hand will rot. OpenAI’s guidance is to set up continuous evaluation: run the evals on every change — prompt edit, model swap, tool change — monitor for new cases of non-determinism, and grow the eval set over time. The suite is the regression test for everything in your setup that is not code: the system prompt, the tool definitions, the model choice. Re-run the full reference set on every prompt edit, every model swap and every tool change, and treat a regression as a blocker.
Decide the release threshold before you see the first score, or you will move the line to wherever the agent happens to land. A practical gate looks like this:
{
"regression_suite_pass_rate": ">= 98%",
"capability_suite_pass_rate": ">= 80% (pass^3)",
"safety_suite_pass_rate": "100% - every adversarial case must fail safe",
"max_cost_per_task": "within 10% of the previous release baseline",
"max_p95_latency": "within 10% of the previous release baseline",
"human_review": "sampled transcripts read and signed off before launch"
}The numbers above are a starting template, not a standard — your thresholds should reflect your business risk and service targets. A 100% pass rate requirement on the capability suite is almost always wrong, because it punishes the eval for catching genuine ambiguity; a floor that blocks deployment, with investigation required below it, is the honest version.
After the gate passes, roll out in stages. Move from offline tests to shadow mode — the agent runs alongside the existing process without affecting users — then a limited rollout with explicit rollback criteria. Production monitoring then takes over: distribution drift, unanticipated real-world failures and user feedback are the signals evals cannot give you. The most effective teams combine all of it: automated evals for fast iteration, production monitoring for ground truth, and periodic human review for calibration.
Common Mistakes When Testing AI Agents
The traps are consistent enough to list. Testing only happy-path scenarios is the first: the failures that hurt in production are the ambiguous request, the missing field, the tool that returns an error — and none of them appear in a demo script. Grading with the same model that generated the output is the second: the judge shares the generator’s blind spots, so calibrate any LLM-as-judge against human experts, give it a way to answer “unknown” when it lacks information, and re-check the calibration when you change the judge model.
Treating a single successful run as proof the agent works is the third. Non-determinism means one pass is noise; only repeated trials produce a number you can trust. And skipping cost and latency measurement until production is the fourth: an agent that works beautifully at ten times your budget is not ready, and the time to learn that is before launch, when the baseline is cheap to capture.
Finally, do not let the eval suite go stale. An eval at 100% tracks regressions but provides no signal for improvement — as capability evals saturate, graduate them into the regression suite and write harder capability tasks. Treat the suite as a living artifact with clear ownership, the same way you treat unit tests. The teams that invest early find the opposite of what they feared: development accelerates as failures become test cases, test cases prevent regressions, and metrics replace guesswork.
