Agent evaluation methods on Amazon Bedrock: choosing metrics that match the task

Introduction to agent evaluation methods

Amazon Bedrock AgentCore Evaluations offers agent evaluation methods through AWS-built evaluators, managed metrics from DeepEval and AutoEval, and custom evaluators. Its practical value depends on a decision that comes before choosing a metric: defining which behaviors demonstrate that an agent has achieved its business purpose.

An agent that runs successfully has not necessarily completed its user’s task. It may have dropped an earlier request, selected the wrong tool, or disclosed personal information while producing an otherwise useful answer. Evaluation needs to distinguish those outcomes instead of treating a plausible response or a high completion score as sufficient evidence of success.

In its agent evaluation playbook, AWS recommends starting with a business outcome, translating that outcome into observable behaviors, and then selecting evaluators at the appropriate level. For a customer support agent, the outcome might be resolving an issue on first contact. The behaviors supporting it include completing the stated goal, maintaining coherence across turns, respecting compliance boundaries, and invoking backend tools correctly.

That approach makes evaluation a way to examine whether the agent meets its requirements. It also gives teams a basis for deciding which measurements deserve attention. A research agent producing a complete answer grounded in retrieved documents needs a different assessment from a moderation assistant expected to flag the right content without flagging acceptable material.

The following framework and product capabilities come from AWS’s account of AgentCore Evaluations. The supplied evidence does not include independent comparative results establishing how reliably its evaluators score particular workloads. The central question for production teams is therefore both practical and specific: which evaluation methods expose the failures that would make their own agent unsuccessful?

Challenges in evaluating AI agents

The first challenge is that different dimensions of quality can diverge. AWS describes a support agent that resolves tickets while exposing personal information in its replies. High completion in that case does not establish acceptable performance. Completion and privacy answer different questions, and both affect whether the business outcome has been reached.

A similar problem arises when a single score combines several requirements. A support response may need to solve an issue, follow refund and data policies, and remain professional. If one judge blends those criteria into one number, the result does not reveal which requirement caused a failure. An improvement in tone could obscure a deterioration in policy compliance.

The second challenge is choosing the right scope of evidence. AWS distinguishes trace-level measurements, which cover per-turn or per-tool-call behavior, from session-level measurements, which assess the conversation as a whole. Tool selection can be examined at the point of invocation. Whether the agent satisfied a user’s goal may require the entire exchange.

This matters particularly for conversations containing multiple requests or clarifications. A relevant answer to the latest message can still leave an earlier request unresolved. AWS identifies DeepEval’s ConversationCompleteness as a metric for that failure and KnowledgeRetention as a way to assess whether information supplied earlier in the conversation has been retained. These are related concerns, but they examine different behaviors.

A third challenge is that some evaluations require information unavailable in live traffic. An expected answer or an expected sequence of tool calls can support a controlled assessment. Production conversations do not come with those references. AWS places evaluators requiring ground truth in batch evaluation, making the availability of reference evidence a constraint on metric selection.

The resulting editorial implication is straightforward: teams need to choose the evidence and the unit of analysis alongside the metric. A catalog entry with an appealing name cannot establish that its score measures the intended requirement under the conditions in which it will run.

Overview of Amazon Bedrock’s AgentCore evaluators

AWS organizes the AgentCore evaluator catalog into built-in evaluators, third-party managed evaluators, and custom evaluators. These provide different starting points for assessment and different degrees of control over the judge model and evaluation criteria.

Built-in evaluators are authored and quality-tested by AWS. The playbook describes them as benchmarked for reliability, although the supplied material does not provide benchmark values or methods for assessing that claim. They cover response quality, task completion, tools, skills, and safety.

Other built-in response metrics include Conciseness, Coherence, InstructionFollowing, and Refusal. Three trajectory checks—TrajectoryExactOrderMatch, TrajectoryInOrderMatch, and TrajectoryAnyOrderMatch—compare the actual tool path with an expected path. AWS recommends these when the route to the answer matters as well as the answer itself. Their dependence on an expected trajectory also makes reference data relevant to deployment choices.

Managed third-party evaluators draw on DeepEval and AutoEval. AWS says AgentCore hosts their code, chooses the judge model, and runs inference on its own capacity in managed mode. For teams already using those libraries in development, the stated benefit is running the same metrics in production without maintaining a separate evaluation service.

DeepEval’s catalog includes TaskCompletion, GoalAccuracy, and ConversationCompleteness for task and goal assessment; TurnRelevancy, KnowledgeRetention, and Summarization for conversational quality; ToolUse for tool behavior; and Bias, Toxicity, and PIILeakage for safety. These expand the choices available for conversations where task completion, retained context, and safe output need separate attention.

AutoEval offers more specialized checks. AWS describes Security as flagging malicious output, including attempts to inject instructions or reveal system prompts. Possible addresses agents that incorrectly declare a valid task impossible. Humor measures a characteristic relevant to consumer-facing agents whose personality is part of the product. Their inclusion illustrates why useful evaluation depends on the application: a behavior can be essential for one agent and peripheral for another.

AgentCore supports three execution modes. On-demand evaluation accepts a session through client.evaluate() for development or debugging. Online evaluation attaches to a live agent runtime and samples real traffic at a configured rate. Batch evaluation applies a defined evaluator stack to a fixed set of sessions for regression testing before releases.

AWS states that all three modes use the same evaluator IDs and that built-in and third-party evaluators can be mixed in an online configuration. That continuity supports carrying suitable metrics from development into production. It does not remove input requirements: an evaluator needing a reference answer or expected path still belongs in batch evaluation under the playbook’s guidance.

Importance of customizing evaluators

Customization becomes useful when a catalog metric does not match the behavior that needs assessment, or when teams need control over the judge model, prompt, or rubric. AWS describes derived evaluators that start from a built-in or third-party evaluator and allow those elements to be changed. After creation, the derived evaluator is used like other evaluators.

The playbook also describes writing a custom LLM-as-a-judge when no catalog entry fits. Here, a language model evaluates agent behavior against supplied instructions and scoring criteria. The value is the ability to express requirements specific to the application. The quality of the resulting assessment still needs to be considered against those requirements; creating a custom evaluator alone does not demonstrate that its judgments are reliable.

Multi-dimension evaluators address a narrower problem: several criteria can be judged from the same evidence but can fail independently. Each dimension receives its own name, instructions, and numeric or categorical scale. The judge reads the trace once and scores every dimension in that call. AgentCore returns a separate result for each dimension, with no composite score.

For the support-response criteria described earlier, this preserves the distinction between resolving the issue, following policy, and maintaining professionalism. AWS’s argument is that teams can then gate a release on each criterion. A tone improvement would not compensate for a compliance regression because the results remain separate.

There are limits to that grouping. Dimensions within one evaluator share a single evaluation level and judge model. AWS recommends separate evaluators when criteria require different levels or models, or when an established built-in or third-party metric already covers the behavior. Grouping criteria is useful only when the shared evidence and execution conditions make sense for all of them.

Reference requirements create another reason for separation. Criteria that need {expected_response} or {assertions} should be kept apart from reference-free criteria when the latter need to run on live traffic. Otherwise, a requirement attached to one criterion can restrict where the combined assessment is usable.

Customization also has a governance role in AWS‘s examples. The playbook recommends derived evaluators for trajectory and skill metrics when inference must remain in the team’s account, and generally recommends them for domain-content agents so judge calls remain under the relevant data governance. That is guidance about evaluation deployment, rather than evidence that a custom rubric automatically improves accuracy.

Best practices for effective evaluation

AWS’s starting recommendation is to select the two or three behaviors that most clearly indicate whether the business outcome was achieved. Add another evaluator when a failure appears that the existing set does not capture. This makes each addition answer a concrete question about the agent’s behavior.

For customer support, the playbook starts with Builtin.GoalSuccessRate, ThirdParty.DeepEval.ConversationCompleteness, and ThirdParty.DeepEval.PIILeakage. Together, these examine goal completion, unresolved requests, and personal-information disclosure. AWS suggests adding checks such as Toxicity or Security when the relevant failure modes warrant them.

For multi-step research, its starting stack is Builtin.TrajectoryAnyOrderMatch, ThirdParty.DeepEval.GoalAccuracy, and ThirdParty.DeepEval.KnowledgeRetention. This combines attention to the expected workflow with goal accuracy and retained context. The trajectory metric needs ground truth, so the full stack cannot simply be transferred unchanged into live evaluation.

For domain content in areas such as medicine, law, or finance, AWS starts with Builtin.Correctness, ThirdParty.DeepEval.PIILeakage, and ThirdParty.AutoEval.Security. Correctness is marked as requiring ground truth in that example. These stacks are AWS’s proposed starting points for the stated outcomes, rather than independent evidence that a particular combination is sufficient for every deployment in those categories.

Before adopting a metric, examine its definition, scoring scale, judge prompt, evaluation level, and required inputs. AWS identifies the AgentCore prompt template library as the place where these details are documented. It also identifies ListEvaluators as the API for retrieving the built-in, DeepEval, AutoEval, and account-specific custom catalog. The useful selection question is whether the evaluator’s actual criteria match the observable behavior that matters.

Then use the execution modes for their documented purposes. On-demand evaluation supports examining a particular session during development. Batch evaluation provides a fixed set of sessions against which to assess prompt changes or model upgrades. Online sampling supplies a production health signal from real interactions. Each contributes a different view, while shared evaluator IDs provide continuity where the input requirements permit it.

The strongest implication of AWS’s playbook is that evaluation design begins with an operational definition of success. AgentCore supplies ways to measure individual responses, tool decisions, conversations, and custom criteria. The production task is to connect those measurements to business requirements and preserve separate visibility into failures that cannot safely compensate for one another.