AI Agents, Evergreen

Testing AI Agents: A Practical Method for Behaviour You Cannot Pin Down

Cartoon robot holding a clipboard with one green checkmark, beside a curtain hiding a red cross, a question mark, and looping arrows, under the headline What one run hides

Testing AI agents is not a harder version of testing ordinary software. It is a different job. Ordinary tests usually assert that a given input produces one exact output. An agent can produce many different valid outputs for the same input, act over several steps, call tools, keep state, and change its plan when a tool result arrives. Testing AI agents therefore means checking behaviour against expectations: did it call the right tools, with valid arguments, in a sensible order, and did the world end up in the right state? The exact wording of the final answer is usually the least important thing to assert.

As Anthropic puts it, an evaluation is a test for an AI system: give the system an input, then apply grading logic to its output to measure success. This guide gives a practical method for agents, whose grading logic must cope with variation, multi-step behaviour, and outcomes in the environment the agent changed.

Why Testing AI Agents Is Different from Testing Ordinary Software

The first difference is non-determinism. LangSmith’s evaluation documentation states it plainly: language model outputs are non-deterministic, which makes response quality hard to assess. The same prompt can yield different valid responses, and a task that passed once might fail next time. A single run is therefore weak evidence. Anthropic treats each attempt at a task as a trial and recommends multiple trials to produce more consistent results, reporting how often the agent succeeds rather than whether it succeeded once.

The second difference is that behaviour emerges from a chain. An agent’s output is produced by the prompt plus the conversation history, plus whatever retrieval placed in context, plus the tools available and the results they returned. Anthropic distinguishes the transcript — the complete record of a trial, including tool calls and intermediate results — from the outcome, the final state in the environment. A flight-booking agent might say the flight is booked, but the outcome is whether a reservation exists. Tests should check both, and weight the outcome most heavily. Third, the model is only one part of what is tested: evaluating an agent evaluates harness and model together. A failure may be a bad prompt, a misleading tool description, or a missing guardrail.

What to Test: The Layers of an AI Agent

Cartoon robot beside a stacked tower of test layers: prompt document, tool, memory database, planning gears, and final output

Before writing tests, map the system. LangSmith’s guidance is to break the system into critical components — language model calls, retrieval steps, tool invocations, and output formatting — and set quality criteria for each. For an agent, the testable layers are the prompt and instructions, tool definitions and tool results, memory and retrieved context, planning and reasoning steps, and the final output. Each layer fails differently, so each needs different tests.

Start at the bottom with cheap tests. Prompt, tool, and memory tests catch most regressions early and run fast. End-to-end tests then check ordering, tool choice, and state across steps, which is where multi-step behaviour breaks. LangSmith describes a different distinction: offline evaluation, which runs curated datasets before deployment to benchmark, regress, and unit-test, versus online evaluation, which monitors live traffic without reference outputs to detect quality and safety problems. Test scope is separate from evaluation setting: both component tests and end-to-end scenario tests can run offline on curated datasets, while online evaluation monitors production traces. Both settings have a place, but offline tests are the repeatable core. Decide what not to test: model internals and general knowledge are the provider’s responsibility. Pin down your instructions, tools, retrieval, guardrails, and orchestration.

Unit and Component Tests for Prompts, Tools, and Memory

Prompt tests work best when the rest of the system is frozen. Give the prompt a fixed input and a recorded model response, and assert on what your code does with it: does it parse the tool call, route correctly, or handle a refusal? When you do run a live model, run it several times and score the pass rate.

Tool tests are ordinary software tests with an agent twist. Test valid arguments, invalid arguments, missing arguments, and error returns, then add the agent-specific cases: bad data, empty data, data that contradicts an earlier result, and timeouts. The agent must cope without inventing a replacement, calling the wrong tool next, or looping. Those behaviours are checkable exactly. Grade the tool calls, not the prose.

Memory and retrieval tests need the same discipline: check what is stored, what is recalled, and what happens when context is stale or conflicting, injecting retrieved context verbatim rather than re-retrieving it live. The fragment below is a tool-call assertion; it assumes a recorded list of tool calls from a run your own harness produced.

# Fragment: assert on tool calls, not on wording.
# `calls` is the recorded list of tool calls from one trial.
def test_refund_uses_lookup_before_refund(calls):
    names = [call["name"] for call in calls]
    assert "lookup_order" in names
    assert "issue_refund" in names
    assert names.index("lookup_order") < names.index("issue_refund")
    refund = [c for c in calls if c["name"] == "issue_refund"][0]
    assert refund["arguments"]["order_id"] == "ORD-123"

Integration and End-to-End Tests for Multi-Step Behaviour

End-to-end tests walk a realistic task from start to finish with mocked or sandboxed tools — for example, a refund request through lookup, eligibility checks, and the refund action — and end by checking the environment: was the refund recorded, was the customer told the right amount, and was nothing else changed? Record tool calls, order, and state changes, so a failure can be replayed step by step.

Repeatability is mostly an environment problem. Fix tool responses, seed data, and record model behaviour. Pin the model version, inject retrieved context verbatim, replay the full conversation history, and fix temperature, seeds, and tool definitions as part of the fixture. Those controls reduce controllable variation, while provider-side and execution effects can remain, so variation is reduced, not removed. OpenAI’s seed documentation is explicit that the system makes a best effort to sample deterministically and that determinism is not guaranteed, with the system fingerprint signalling backend changes. Temperature 0 does not fix it alone, because sampling is only one source of variance alongside retrieval ordering, tool-result ordering, and provider-side changes. Measure a rate over many trials and gate releases on a scored pass rate, not a single green build. Anthropic adds two rules: start each trial from a clean, isolated environment, and grade the outcome more than the path, giving partial credit for multi-component tasks.

Evaluation: Scoring Agent Outputs Without a Single Right Answer

Many agent outputs can be correct in several wordings, so exact-match assertions fail even when the agent did the job. The solution is a golden dataset: curated example tasks with accepted outcomes, built from manually curated examples, historical traces, user feedback on failed runs, and heuristics that flag long-latency or error cases. LangSmith suggests starting by defining what good looks like with five to ten examples for each critical component, such as correct tool selection and proper argument formatting, and building datasets of ten to twenty high-quality examples covering common scenarios and edge cases. Anthropic’s roadmap is similarly modest: twenty to fifty simple tasks drawn from real failures is a great start.

Scoring can draw on three kinds of graders, selected for the task, and retaining human calibration when model graders are used. Code-based graders are deterministic: string matching, schema validation, tool-call verification, and pass-or-fail unit tests. Model-based graders — an LLM as judge — apply a rubric in natural language where wording varies but meaning can be judged. Human graders review a sample, calibrate the model graders, and settle cases automation cannot. Anthropic recommends deterministic graders where possible, LLM graders where necessary, and human graders judiciously, with LLM graders calibrated against human experts. LangSmith offers the same approaches, plus decision-model and pairwise evaluators, distinguishing reference-free evaluators from reference-based ones, which compare against a reference and only work offline.

Choose metrics that fit the task: report task completion and correctness of tool use as pass rates, and report cost per task or trial in monetary or token units and latency in time units, with appropriate summaries, alongside quality. For variation across trials, Anthropic defines two complementary metrics. Pass at k is the likelihood that at least one of k attempts succeeds. Pass to the power of k is the probability that all k trials succeed; with a 75 percent per-trial rate over three trials, that is about 42 percent. Use pass at k where one success matters, and pass to the power of k where users expect reliable behaviour every time. Frameworks include LangSmith (datasets, experiments, and evaluators for offline and online evaluation), the framework-agnostic options Anthropic lists (Harbor, Braintrust, LangSmith, Langfuse, Arize), and Promptfoo, an open-source evaluation and red-teaming framework. What matters is a versioned dataset, calibrated graders, repeatable trials, and comparable results.

Testing Failure Modes, Guardrails, and Safety

Cartoon robot scientist prodding a trapped document, a broken sparking tool, and a runaway loop, with a shield and gauge standing guard

Happy-path tests miss the failures that matter most. Probe ambiguous instructions, contradictory tool data, tool failures and timeouts, and looping or runaway behaviour. Return contradictory data from two tools and check that the agent notices the conflict instead of acting on the first result. Fail a tool mid-task and check that it degrades gracefully — reports the failure, retries within a limit, or escalates — rather than inventing a result or retrying forever.

Guardrail tests deserve their own suite. Prompt injection is the central case: direct injection from the user, and indirect injection hidden in retrieved content such as a review, web page, email, or tool result. The OWASP Top 10 for LLM Applications places prompt injection first, as LLM01, in its 2026 edition, and its Top 10 for Agentic Applications lists agent goal hijack (ASI01) and tool misuse (ASI02) as agent-specific counterparts. Plant adversarial instructions in retrieved context and check whether the model follows them, attempt to extract disallowed data, and try to get the agent to take a disallowed action. Promptfoo’s red-teaming documentation describes this approach, including an indirect-prompt-injection plugin that plants adversarial instructions in retrieved context and checks whether the model follows them. Do not assume a filter prevents prompt injection: test the deployed controls against direct and indirect attacks, and verify the system around the model — least-privilege tools, scoped permissions, approval gates, and output checks.

Set limits the tests can verify. Step budgets cap turns, spending or call budgets cap model and tool calls, and timeouts cap step duration. Each limit needs a test that exceeds it on purpose and checks that the agent stops, reports, or escalates. A budget that has never been tested is a hope, not a control. Test sensitive data the same way, with canary data the agent must not expose, checking outputs and tool arguments.

Running Agent Tests Repeatedly: CI, Regression, and Tracing

Run the suite in continuous integration with recorded or mocked model responses, so deterministic parts are checked on every change, and re-run against live models on a schedule, so drift in model behaviour, prompts, or tools is caught before users report it. Report the two kinds of run separately: a failure in a controlled replay suite warrants investigating changes in code, fixtures, configuration, dependencies, and environment, using traces and version comparisons to establish the cause, while a live failure may mean the model, data, or environment changed.

For every bug found in production, add a failing test that reproduces it before fixing it, using the real conversation, the exact retrieved context, and the same tool definitions. Anthropic makes the same point for eval suites: production failures become test cases, capability evals with high pass rates graduate into regression suites, and regression evals should have a nearly 100 percent pass rate, so a decline signals breakage. Watch for saturation: add harder tasks when scores hit 100 percent.

Tracing makes all of this debuggable. A trace is the complete record of a run: inputs, outputs, intermediate steps such as tool calls and retrieval, and metadata including latency. LangSmith lets evaluators score runs offline against datasets or online against live traffic, with production feedback feeding back into offline datasets. Read transcripts when tests fail: Anthropic’s guide insists failures should seem fair, and reading the transcript shows whether the agent made a genuine mistake or the grader rejected a valid solution. Keep the suite healthy the way you keep unit tests healthy: owned, reviewed, pruned when tasks go stale, and extended whenever production teaches something new. The goal is not a green build. It is a method that tells you, with evidence you can replay, whether the agent’s behaviour is good enough to ship, and what changed when it stops being good enough.