Test AI agent behavior by checking what the agent did, not just what it said. Build a small set of realistic tasks, run the agent on them in a sandbox, and grade the outcome, the tool calls, the state changes, and the safety of the run. Use deterministic checks for routing and parsing, evals with code and model-based graders for judgment calls, repeated runs to handle non-determinism, and human review for ambiguous cases. Run those evals offline before release to catch regressions, and monitor live traffic after release to find new failure modes.
That approach works because agents are non-deterministic systems that act. The same input can produce different wording, different tool-call orders, and different paths to the same result. A test that asserts exact wording will be flaky. A test that asserts the right outcome, the right tools, and no harmful side effects can be stable enough to support release gating, but only when measured variability is low, graders are validated, and thresholds are calibrated. A single pass does not prove reliability.
Why agent behavior is hard to test
Traditional software tests assume the same input produces the same output. AI agents break that assumption in three ways.
First, the model itself is non-deterministic. Even with a low temperature, wording, ordering, and tool choice can vary between runs.
Second, agents act over multiple steps. A failure can appear in the final answer, in an intermediate tool call, in a bad argument to a tool, or in a missing step. Checking only the final answer can miss intermediate failures.
Third, agents touch the world. They read files, query databases, call APIs, send messages, and change state. A test that runs against live systems can create real side effects, leak data, or incur cost. That is why Anthropic recommends extensive testing in sandboxed environments, with appropriate guardrails, and why sandboxing is not optional for agents that can act.
The practical consequence is that agent testing is evaluation, not just unit testing. As LangSmith puts it, LLM outputs are non-deterministic, so evals break down what good looks like and measure it. You define good behavior with examples, run the agent against those examples, and score the results with graders that tolerate variation in wording while remaining strict about outcomes and safety.
What to test
A good agent test suite should cover all five dimensions. Covering a dimension is separate from deciding whether a check blocks CI or a release. Only appropriately validated checks with calibrated thresholds should block CI or releases. Other checks may produce warnings.
Outcomes
Did the agent achieve the task? For a support agent, that means the ticket was resolved correctly, the refund was issued for the right order and amount, or the user got an accurate answer. Outcome checks should be based on state and data, not on whether the reply contains a particular phrase. For example, check the order status in the test database, not whether the reply says the word refund.
Trajectory
Did the agent take a sensible path? Trajectory is the sequence of messages and tool calls the agent produced. LangChain AgentEvals describes evals as measuring an agent by assessing its execution trajectory. Trajectory checks catch agents that reach the right answer by the wrong route, such as skipping a policy lookup, calling a privileged tool before authorization, or reading data it should not have accessed.
Useful trajectory checks include:
- Required tools: the agent called the tools it must call, such as looking up an order before issuing a refund.
- Forbidden tools: the agent did not call tools outside its scope, such as deleting records when it was only asked to read them.
- Arguments: tool arguments were valid, in range, and tied to the task, not copied from an unrelated example.
- Order: steps that must happen in order did happen in order, such as checking authorization before taking an action.
Tools and state
Tools are where agents cause real effects. Test tool selection, argument formatting, error handling, and idempotency. What happens when a tool returns an error, an empty result, or malformed data? Does the agent retry sensibly, stop safely, or loop forever? A robust agent should degrade gracefully, not invent a result when a tool fails.
Also test state changes. After the run, inspect the database, files, tickets, or messages that the agent was allowed to change. The final state is often a more reliable signal than the final message.
Safety
Safety checks are not an extra layer. They are part of correctness for any agent that can act. Test prompt injection, jailbreak attempts, and requests that would cause harm if followed. Test that the agent refuses or escalates when it should, and that it does not leak sensitive data, follow instructions from untrusted tool output, or bypass limits.
Behavioral evaluations are useful here. Anthropic Bloom generates targeted behavioral evaluations from a researcher-specified behavior, using a four-stage pipeline of understanding, ideation, rollout, and judgment, and scores each rollout for how strongly the behavior appears. The idea transfers: define the behavior you want to measure, generate scenarios designed to elicit it, run them, and score the transcripts with a rubric.
Cost and latency
Agents can be correct but too slow or too expensive to ship. Track tokens, tool calls, turns, and wall-clock time per task. Set budgets and fail the eval if the agent exceeds them.
How to build a small, maintainable eval set
Start small. A large, brittle eval set is worse than a small, trusted one. The goal is a set that engineers can explain and that runs fast enough to support release gating when the checklist conditions are met.
- Define good behavior with examples. As an illustrative starting point, write 5 to 10 examples of what good looks like for each critical component. For an agent, that means examples of correct tool selection, proper argument formatting, and a sensible trajectory, not just a good final answer. Adjust the number based on risk, cost, and observed variability.
- Draw cases from real traffic and known failures. Include typical cases, edge cases, and adversarial cases. Typical cases provide evidence about the agent on the sampled common-path scenarios. Edge cases provide evidence about the sampled ambiguous scenarios. Adversarial cases provide evidence about the sampled injection, manipulation, and misuse scenarios. Results apply to the sampled scenarios, not as proof for all unseen cases.
- Make each case self-contained. Each example should include the input, the starting state, the expected outcome or trajectory, and the rubric for grading. Use fixtures, fakes, or a sandbox database instead of live data.
- Tag blocking cases. Mark the cases where a failure of an appropriately validated check with a calibrated threshold should block a release, such as safety violations, data leaks, or destructive actions. Other checks may produce warnings without blocking.
- Grow the set from production. When online monitoring or human review finds a new failure, add a case that reproduces it.
As an explicitly hypothetical example, a starting set might be 20 to 50 cases drawn from real traffic and known failures, with a mix of typical, edge, and adversarial inputs. This range is not presented as demonstrated to be adequate or as applicable across teams. Choose the size based on risk, cost, and measured variability. Do not chase a large number for its own sake. A set of 30 cases you trust beats 300 cases no one understands.
Graders: how to score agent behavior
A grader is the function that turns an agent run into a score. Different graders suit different checks. Use more than one type.
Code and rule-based graders
Code graders are deterministic, rule-based functions. They work well for checks where the answer is unambiguous: the response is not empty, the JSON parses, the order ID matches, the tool was called, or the state changed correctly. Code graders are fast, cheap, and repeatable. Use them for every check that does not require judgment.
For trajectory matching, AgentEvals offers strict, unordered, subset, and superset modes for deterministic checks.
LLM-as-judge graders
LLM-as-judge graders use a model to score an output against a rubric. They are useful when the check requires judgment, such as whether a summary is accurate or whether a trajectory is sensible. They can be reference-free, checking quality without an expected answer, or reference-based, comparing the output to a reference.
LLM judges need the same care as any other model component. Write a clear rubric with examples of good and bad outputs. Calibrate the judge against human labels. Sample judge decisions and review them, especially when a judge score blocks a release.
OpenAI Evals defines an eval with a data source config plus testing criteria. Grader types include label-model, string-check, text-similarity, Python, and score-model graders, each with a name, inputs, and pass criteria.
Human review
Human review is the most flexible grader and the least scalable. Use it for ambiguous cases, for calibrating automated graders, and for finding failure modes no grader yet covers. Use a rubric, an annotation queue, and a sample of runs, not every run.
Pairwise comparison
Pairwise graders compare two outputs and judge which is better. Pairwise evaluation works well when absolute scoring is hard but comparison is easy. Use pairwise comparison for A/B tests between agent versions, prompt changes, or model swaps.
Running evals offline and online
Evals serve two different purposes, and you need both.
Offline evals run before deployment on curated datasets. They are for benchmarking, regression testing, unit testing of components, and backtesting new versions against historical data. Offline evals often use reference outputs, but not always. When a reference output is available, they can check correctness against expected answers. Many offline checks are reference-free, such as format validation, safety checks, and quality heuristics. They should run in CI, gate releases only when the conditions in the checklist are met, and produce a report that engineers can act on.
Online evals run after deployment on live traffic. They monitor quality, detect anomalies, and collect production feedback. Online evals usually have no reference outputs, so they focus on quality patterns, safety checks, and real-world behavior. When an online eval flags a run, that run should become a candidate for a new offline case.
The loop is the point. Offline evals validate fixes. Online evals surface new failures. Human review turns those failures into new offline cases.
Worked example: testing a support agent
Consider a support agent that can look up orders, issue refunds, and update tickets. A useful eval set might include:
- A damaged-order refund: the agent must look up the order, confirm the item and amount, issue the refund for the correct order, and update the ticket.
- A refund for an order that does not exist: the agent must not invent an order, must not issue a refund, and must tell the user it could not find the order.
- A request to refund twice the purchase price: the agent must not over-refund, must check the order total, and must escalate or refuse.
- A prompt-injection attempt in a ticket: the agent must not follow instructions embedded in the ticket text, must not reveal internal data, and must handle the request safely.
- A vague request: the agent must ask a clarifying question instead of guessing.
For the damaged-order case, a code grader can check that the agent called lookup_order before issue_refund, that the refund amount matches the order total, and that the ticket status changed to resolved. An LLM-as-judge grader can check whether the reply is clear, accurate, and helpful.
For the injection case, the grader is different. The key check is not whether the reply is helpful, but whether the agent followed an instruction it should have ignored. That check must be strict.
As an illustrative example, run each case three times and average the scores. Three runs is a design choice that trades cost against a rough sense of variability, not a proven guarantee of stability. Choose the repeat count based on measured variability, cost, and risk. Do not average away a safety failure. A safety check that passes two out of three times is not passing.
Common failure modes
Agent tests tend to fail in predictable ways. Knowing the patterns helps you write better evals.
Flaky assertions on wording. The test asserts a specific phrase, but the agent paraphrases correctly. Fix the test, not the agent. Assert outcomes and state, not exact wording.
Missing trajectory checks. The test checks only the final answer, so an agent that skipped a required step still passes.
Reward hacking the harness. The agent learns to satisfy the grader rather than the task, for example by gaming a metric or routing around a limit. Broaden the checks, add a human review sample, and tighten the sandbox.
Silent state changes. The agent changes state but reports failure, or reports success without changing state. Always inspect the final state, not just the message.
Tool failures treated as success. The agent invents a result when a tool returns an error or empty data. Test tool errors explicitly. The correct behavior is usually to stop, retry sensibly, or escalate, not to hallucinate.
Judge drift. An LLM-as-judge grader gradually becomes more lenient or more strict as prompts or models change. Calibrate judges against human labels regularly.
Conclusion: a practical testing checklist
Testing AI agent behavior is not about proving the agent is perfect. It is about gathering evidence that it is safe enough and reliable enough for the intended use, based on measured variability and validated graders, and improving over time. Use this checklist as a starting point.
- Define good behavior with, as an illustrative starting point, 5 to 10 examples per critical component, drawn from real traffic and known failures, adjusted to risk and measured variability.
- Test outcomes, trajectory, tools, safety, and cost, not just the final answer. Cover all five dimensions, but only appropriately validated checks with calibrated thresholds should block CI or releases; other checks may produce warnings.
- Use code graders for deterministic checks, LLM-as-judge for judgment calls, and human review for ambiguous cases.
- Run each case more than once, average scores, and treat failures of appropriately validated safety checks with calibrated thresholds as blocking even if they are intermittent. Gate releases only when measured variability is low and graders are validated. Other checks may produce warnings.
- Test in a sandbox with limits on tools, data, side effects, and iterations.
- Run offline evals before release to catch regressions, and online evals after release to find new failures. Use offline results to gate releases only when the eval set, graders, and thresholds have been validated.
- Turn every production failure into a new offline case, so the eval set grows with the agent.
Agents will always surprise you. A good test suite does not eliminate surprise. It makes surprise visible, measurable, and fixable before it reaches users.
