AI agent self-evaluation is the step where an agent checks its own output before the user ever sees it. A language model generates its first answer in one pass, and that pass can be wrong in ways the model cannot see while generating: a wrong figure, a missed requirement, a step that contradicts the plan. Self-evaluation is the pause between finishing and returning, in which the agent inspects what it produced and fixes the faults it finds. Done well, it catches errors a single pass cannot. Done badly, it is a second opinion from the same mind that made the mistake — and it can make a right answer wrong.
Why an agent must check its own work
A model cannot reliably tell right from wrong by generating alone. Generation is prediction: the model produces the most plausible continuation. Plausible is not the same as correct, and nothing in the generation step checks the output against the request, the facts, or the constraints. So the first answer is a draft, whatever the user asked for. Returning it unchecked risks a wrong refund, a wrong figure in a report, a plan that contradicts itself. Checking costs another model call, more tokens, more latency. The design question is not whether to check, but what the check can actually catch — because a check that shares the generator’s blind spots adds cost without adding safety.
This differs from handling errors during a run, when a tool times out or returns malformed data: that is error handling, with its own loop of detect, classify, decide and act. Self-evaluation is about the finished output: the work is done, nothing failed, and the question is whether the result is good enough to return.
What AI agent self-evaluation actually means
Evaluating an output means checking it against four things: the original request, the constraints the task carries, the facts it relies on, and the format it must take. Completeness asks whether every part of the request was answered. Correctness asks whether the claims are right. Grounding asks whether each fact traces to a source the agent actually saw, rather than to something the model invented. Format asks whether the output matches the shape the next step expects — valid JSON, the right fields, no missing keys.
Some checks the agent can do itself. It can re-read the request and compare it with the output, point by point, and re-read a tool result it is about to quote to confirm the number matches. What it cannot do on its own is verify a fact it has no source for. A model that invented a figure cannot reliably catch the invention by thinking harder; the check needs an appropriate reference to test the claim against — the original request, a tool result, a retrieved document, or a second model with different information. A source already used in drafting can still verify a requirement or a transcription; only unsupported facts need new evidence. That distinction — between what the agent can verify against inputs it already has and what needs new evidence — is the whole design problem of self-evaluation.
Self-critique: the agent as its own reviewer
The simplest form of self-evaluation is prompted critique: the agent is asked, in a separate call with a different prompt, to review its own output against an explicit rubric or checklist. The separation matters. A critic prompt that just says “check this” produces vague approval; a critic prompt that lists the requirements and asks for each one to be confirmed or denied produces findings the generator can act on. Stronger versions separate the roles further: a different model for the critic, a lower temperature so its judgment is stable, or a critic that sees only the output and the requirements, not the generator’s reasoning — so it is less likely to be talked into the generator’s mistakes.
Self-critique can catch some errors and can miss others, especially shared factual misconceptions, without guaranteeing either outcome. It can catch missed requirements, format errors and internal contradictions, because those are visible in the text. It may miss factual errors the model itself believes, because the critic shares the generator’s knowledge and misconceptions. A replication study of the Reflexion framework found exactly this failure mode: when the same model generates the actions, evaluates itself and writes the reflections, the reflections tend to repeat earlier misconceptions rather than introducing new reasoning, and confirmation bias limits the corrective feedback. That is the central weakness of self-critique, and everything else in this article is a way of working around it.
Verification in code, not just in prose
A programmatic check has one advantage a second model call does not: it does not share the model’s beliefs. Structured outputs handle the shape of an answer: OpenAI’s Structured Outputs, launched in August 2024, constrain the model at decode time so its response must match a supplied JSON Schema — the model literally cannot emit a token that breaks the schema, and OpenAI reports that gpt-4o-2024-08-06 reached 100% schema compliance with the feature enabled, against under 40% for older prompting approaches. But OpenAI’s own guide states the limit plainly: “Structured Outputs can still contain mistakes.” The shape is guaranteed; the values are not. A schema can force the model to return a verdict from a fixed list; it cannot force the verdict to be right.
So the second layer is verification in code: before a consequential action, the agent re-reads the tool result it is about to act on, checks preconditions, and compares the output against the plan. A schema for an evaluation verdict can force the critic’s output into a shape the program can act on:
{
"type": "object",
"properties": {
"verdict": {
"type": "string",
"enum": ["pass", "revise", "escalate"]
},
"issues": {
"type": "array",
"items": {
"type": "object",
"properties": {
"type": {
"type": "string",
"enum": ["missing_requirement", "wrong_fact", "format_error", "unsupported_claim"]
},
"detail": { "type": "string" }
},
"required": ["type", "detail"],
"additionalProperties": false
}
},
"confidence": { "type": "number" }
},
"required": ["verdict", "issues", "confidence"],
"additionalProperties": false
}The schema is the first layer, not the whole solution. A verdict of “pass” from a schema-valid response is still a model’s opinion. The program’s job is to make that opinion checkable: run the tests, query the database, fetch the source, and let the result — not the model’s confidence — decide.
Reflection and correction loops
Finding a fault is half the job; the other half is fixing it and checking the fix. The reflection loop is the pattern that does both. In Reflexion (Shinn and colleagues, NeurIPS 2023), the agent converts feedback on a failed attempt into a verbal reflection — a short text describing what went wrong and what to change — stores it in an episodic memory buffer, and conditions the next attempt on it. No weights are updated; the learning is entirely in the text the agent carries forward. The results were substantial: Reflexion reached 91% pass@1 on the HumanEval coding benchmark, against 80% for the GPT-4 baseline, and improved AlfWorld decision-making tasks by 22% absolute over 12 iterative steps. A reflection record is a small, structured note, not a paragraph of prose:
{
"task_id": "summarize-sources-014",
"attempt": 2,
"failed_check": "The summary cited a figure the source list does not contain.",
"lesson": "Verify every number against the source list before drafting the summary.",
"next_change": "Check figures against sources first, then write."
}Self-Refine, from Madaan and colleagues at NeurIPS 2023, uses the same loop with a single model playing all three roles: the model generates an output, gives itself feedback on it, and refines it, repeating for up to four iterations. Across seven tasks spanning dialogue, code, math and generation, the refined outputs were preferred over one-step generation, improving by about 20% absolute on average — though the gains vary by task, and an independent re-analysis found near-zero gain on math. The loop itself is short enough to write out, and Anthropic’s evaluator-optimizer workflow is the same shape described as a design pattern: one LLM call generates a response while another provides evaluation and feedback in a loop, used when the evaluation criteria are clear and iterative refinement provides measurable value, as in literary translation or multi-round search:
# Fragment: the loop logic, not a runnable program.
# call_model() is the agent framework's model call; run_checks() is a
# programmatic check against ground truth, a test suite, or a retrieved
# source, returning (passed, failures) — a pass flag and the list of
# failed checks with their diagnostics. A draft is returned as verified
# only after passing both.
def evaluate_and_revise(task, max_rounds=3):
draft = call_model(f"Complete this task:\n{task}")
for _ in range(max_rounds):
passed, failures = run_checks(draft)
if not passed:
failures_text = "\n".join(f"- {f}" for f in failures)
draft = call_model(
"The programmatic checks failed on this draft. "
"Fix only the failures listed below; change nothing else:\n"
f"Failures:\n{failures_text}\n"
f"Task: {task}\nDraft: {draft}"
)
continue
critique = call_model(
"Critique this draft against the task requirements. "
"Reply with exactly PASS if it meets every requirement, "
"or list every concrete problem:\n"
f"Task: {task}\nDraft: {draft}"
)
if critique.strip() == "PASS":
return draft, "verified"
draft = call_model(
"Revise the draft using this critique. Change only what the "
f"critique names:\nCritique: {critique}\nDraft: {draft}"
)
return draft, "unverified — route to a human, do not return as checked"Self-consistency needs no loop at all. Wang and colleagues at Google sample a diverse set of reasoning paths for the same question and return the answer most of them agree on. With PaLM-540B it raised GSM8K accuracy from 56.5% to 74.4%, SVAMP from 79.0% to 86.6%, and AQuA from 35.8% to 48.3%, with no extra training and no auxiliary model. The insight is statistical: correct reasoning paths, even when they take different routes, tend to converge on the same answer, while wrong paths scatter. At an equal budget of six responses on GSM8K, self-consistency reached 85.3% against 83.2% for multi-agent debate — higher accuracy at the same budget, with no cost comparison established.
When to stop correcting
A correction loop without a stopping rule is a new failure mode. Every round costs time, tokens and latency, and — more importantly — every round is a chance to make a right answer wrong. Huang and colleagues measured this directly (ICLR 2024): when a model revises using only its own judgment, with no external feedback, accuracy falls. GPT-4’s GSM8K accuracy dropped from 95.5% to 91.5% after one intrinsic revision round and to 89.0% after two; HotpotQA fell from 49.0% to 43.0% after two rounds. Their explanation is that the gains reported by reflection methods came from oracle labels — ground-truth signals telling the loop when to stop — and the gains vanish when those labels are unavailable. In one reported condition, models challenged on a correct answer abandoned it up to 86% of the time. A review prompt reliably changes an output; it does not reliably move it toward the truth.
The practical stopping rules follow from that finding, and they are application policy — chosen limits for a particular system, not measured research conclusions; the research leaves safe iteration counts open. Stop with success only when the output passes checks that can be verified independently of the model: the programmatic checks pass and the evaluator’s verdict is confirmed against them. Stop at a fixed maximum number of rounds — three in the loop above, four in Self-Refine — regardless of what the model thinks. Escalate to a human when a correction changes the answer without new evidence, or when the loop ends without a verified pass. A critic’s PASS on its own is not a success condition: the exit must rest on an externally checked result, because a model that cannot reliably judge its own output cannot be the sole judge of when to stop judging it.
Limits and common mistakes
The first limit is the shared blind spot. Self-critique, reflection and self-consistency all draw on the same model, and a model may fail to correct a misconception it holds with confidence. The defence is separation: a different model for the critic, or an external check — a test suite, a retrieved source, a database — that does not share the generator’s beliefs.
The second is verification that only re-reads the same output. Asking a model whether its own answer looks right is not verification; it is a second reading of the same evidence. Verification means checking the output against an appropriate reference or an independently checkable result — the original request, a tool result, or a retrieved document — which can include sources already used in drafting. New evidence is needed for facts the output asserts without support; existing authoritative inputs can still verify requirements and transcription.
The third is treating evaluation as a guarantee. A schema guarantees shape, not truth, and a passing evaluator verdict is a model’s opinion — an 86% abandonment rate on challenged correct answers warns against treating it as a verdict on reality. Self-evaluation is a filter: it catches the errors it can see, at a known cost, and the errors it cannot see are why the filter is not the last line of defence. For consequential actions — payments, deletions, messages sent on a user’s behalf — the last line is still a human, or a programmatic check against ground truth. Self-evaluation makes an agent’s outputs more reliable. It does not make them right.
