LLM reasoning and planning is what turns a chatbot into an agent. A chatbot answers a prompt and stops; an AI agent takes a goal, works out a sequence of steps, acts on the world through tools, watches what happens, and adjusts its plan until the job is done or it needs a human. This guide explains how that happens, step by step, and how to judge whether an agent’s plan is sound.
What the LLM Actually Does Inside an Agent
Google Cloud defines AI agents as intelligent systems that demonstrate reasoning, planning, and memory. In practical terms, they interpret goals, plan multiple steps ahead, and work independently across systems under supervision. The large language model sits at the centre of that system as its reasoning engine, but reasoning is only one of its jobs.
Inside a single run, the same model typically does three different jobs. First, it interprets: it reads the goal, the tools available, and everything observed so far. Second, it decides: it selects the next step, whether that is a tool call, a question to the user, or a final answer. Third, it generates: it produces the text of a message, the arguments for a tool call, or a revised plan. A model can be good at generating fluent text while still choosing the wrong next step.
A plain chat completion is not an agent, because it has no loop, no tools, and no persistent goal. It receives one prompt and returns one response. An agent adds the machinery around the model: a goal that persists across steps, tools that let it affect something outside its own text, observations that feed back into its next decision, and a stopping rule. Anthropic’s framing, from its December 2024 guide Building Effective Agents, draws the line at who decides the next step: in a workflow, code follows predefined paths, while in an agent, the model dynamically directs its own process and tool usage. Systems can also combine the two, with workflows around agents and agents that call workflows as tools.
From Goal to Plan: How Agents Break Work Down
Planning starts before any action, with goal parsing. The agent, or the system built around it, has to establish what counts as success, what the constraints are, and what is out of scope. A goal such as “book a trip” is not yet a plan: it needs a destination, dates, a budget, and a rule for what happens when no option fits. The clearer the success criteria, the more checkable the plan becomes.
Next comes task decomposition: breaking the goal into ordered subtasks with dependencies between them. Searching for options has to come before comparing them; comparing has to come before booking. Some plans are fixed in advance, like a checklist written in code. Others are generated on the fly, with the model deciding the next subtask after seeing the previous result. Fixed plans are predictable and easy to test; generated plans handle problems where the number of steps cannot be known in advance, where a model directing its own next steps fits better than a fixed path.
A useful habit when judging a plan is to ask whether each step is verifiable. A step that says “find the best option” cannot be checked; a step that says “retrieve the three cheapest options that meet the budget and dates, then compare them on price and travel time” can be. Vague, uncheckable steps and unverified assumptions are risks to check for in any plan.
The Reason-Act-Observe Loop: LLM Reasoning and Planning in Action
The core execution pattern is the reason-act-observe loop, formalised as ReAct by Yao and colleagues in 2022. In ReAct, the model generates reasoning traces and task-specific actions in an interleaved manner: it reasons about the current state, acts through a tool or an interface, observes the result, and reasons again. Reasoning traces help the model induce, track, and update action plans as well as handle exceptions, while actions let it interface with external sources, such as knowledge bases or environments, to gather additional information.
That interleaving is the point. Reasoning alone, as in chain-of-thought prompting, can drift: an early error propagates through every later step, and nothing checks the reasoning against the world. Acting alone can be efficient but blind. The evidence bears this out. On question answering and fact verification, interacting with a simple Wikipedia API let ReAct overcome the hallucination and error propagation prevalent in chain-of-thought reasoning. On interactive decision-making benchmarks, it outperformed imitation and reinforcement learning methods by an absolute success rate of 34 percent on ALFWorld and 10 percent on WebShop, using only one or two in-context examples. On ALFWorld specifically, ReAct reached 71 percent, against 45 percent for acting without reasoning and 37 percent for the BUTLER baseline.
The loop also needs a stopping rule. It stops when the success criteria are met, when a step cap is reached, or when the situation calls for a human. Without a stopping rule, a loop has no reason to quit, and an agent can keep retrying a failing step indefinitely.
The simplified fragment below shows the shape of the loop. It is an illustration of the structure, not a working agent: a real implementation would call a model API, parse tool calls, and handle errors at each step.
# Simplified illustration of the reason-act-observe loop structure.
# Not a working agent: no model call, no real tools.
def run_agent_loop(goal, tools, max_steps=10):
history = [f"Goal: {goal}"]
for step in range(max_steps):
thought = f"Step {step + 1}: what is still missing for this goal?"
history.append(thought)
action = choose_next_action(history, tools)
if action is None:
return "Done: " + "; ".join(history)
observation = tools[action["tool"]](**action["arguments"])
history.append(f"Observed: {observation}")
return "Stopped: step cap reached, human review needed"
Memory, Context and Retrieval Keep Plans Grounded
An agent’s plan is only as good as the information it is built on. A model reasoning from training data alone is reasoning from memory of the world as it was when it was trained, not from the facts of the task in front of it. That is why Anthropic describes the building block of agentic systems as the augmented LLM: a model enhanced with retrieval, tools, and memory, rather than a bare model on its own.
Short-term context is what the agent carries within a run: the conversation so far, the trajectory of thoughts, actions, and observations, and the current state of the task. Long-term memory and retrieved documents supply what lies beyond that run: previous decisions, preferences, company documents, and current data. Retrieval matters because it grounds the next step in something checkable. Retrieved facts still need checking for relevance and correct use, and recalled facts need independent checking against an external source before they support an action.
Grounding has limits. Retrieving the right document does not guarantee the agent uses it: production analyses describe cases where the correct document was retrieved but ignored, lost among too much context. Retrieval reduces hallucination; it does not abolish it. So retrieval should be paired with verification: check the step against the retrieved source before acting on it, especially when the action is hard to undo.
When Plans Change: Replanning and Error Recovery
Real plans fail, and replanning is where agent quality actually shows. A tool returns an error, a search returns nothing, or an observation contradicts an assumption the plan depended on. The agent’s job then is to detect the failure, work out which part of the plan it invalidates, and revise the remaining steps rather than restarting from scratch or blindly retrying the same step.
Detection comes from the environment, not from the model’s confidence. Tool errors, validation failures, and observations that do not match expectations are ground truth the agent can use to assess its progress, in Anthropic’s phrase. A search that returns zero results is information: the query may be wrong, the item may not exist, or the source may be unavailable.
Self-correction has limits. Repeated retries that make no progress are looping; a bounded retry justified by the observed error is recovery, and for side-effect tools the prior outcome must be checked before rerunning the action. An agent that silently revises its goal to match whatever it managed to do is not replanning; it is redefining success. The honest stopping points are: the plan succeeded, the plan is blocked with no alternative step, or the uncertainty is high enough that a human should decide. Building those stopping points in advance, as step caps and escalation rules, is what separates a system that recovers from one that spins.
Where LLM Reasoning Breaks Down
Agents break in patterns that single model calls do not. The first is hallucinated tool arguments: the model invents parameters that feel correct, such as guessing a field name from training data instead of checking the tool’s schema, and the tool then returns a plausible-looking but wrong result. The second is looping: without a termination condition, an agent rephrases a failing search or re-executes a failed plan again and again, burning time and cost while making no progress. Repeated identical or near-identical calls are the signature, and production guides suggest investigating any single request that passes around ten iterations.
The third pattern is goal drift over long trajectories. As context grows, the original goal and its constraints get buried under pages of intermediate reasoning, and the agent starts optimising for the most recent step rather than the original objective. The fourth is compounding error: Anthropic’s warning that one bad step taints the rest, because every later step builds on the earlier one’s output.
Guardrails follow directly from those patterns. Validate every tool call against a strict schema before execution. Require confirmation for destructive or irreversible actions, and prefer read-only tool variants by default. Set step caps and progress checks so a loop cannot run indefinitely. Test in sandboxed environments, and keep a human gate where the cost of being wrong is high. None of these make the model smarter; they make its mistakes cheaper and easier to catch.
Tool definitions are part of that guardrail story. In OpenAI’s function tools, the developer defines each function with a name, a description, and a JSON Schema for its arguments; the agent requests a call, application code runs it and returns a result, and the harness continues the turn. The fragment below shows the shape of one such definition.
{
"name": "search_documents",
"description": "Search the approved document store for a query string.",
"parameters": {
"type": "object",
"properties": {
"query": {"type": "string"},
"max_results": {"type": "integer"}
},
"required": ["query"]
}
}
A Worked Example and a Checklist for Judging a Plan
Consider an illustrative task: arrange a short business trip within a budget. The agent parses the goal into constraints (destination, dates, budget, approval rule) and success criteria (a booked itinerary, or a clear statement that none fits). It decomposes the work: search, compare on price and timing, book, confirm. In the loop, it reasons that it needs options before it can compare, calls a search tool, observes the results, reasons that one option breaks the budget, replans around the rest, and stops either with a booking or with an escalation when no option fits. At each step, the observation checks the reasoning: if the search returns nothing, the plan changes; if the booking tool errors, the agent does not claim success.
Use this checklist on any agent plan, including one you built yourself:
- Is the goal stated with success criteria and constraints, not just a vague request?
- Are the steps ordered, with dependencies explicit and each step verifiable?
- Have the facts behind each step been checked against relevant external evidence — recalled facts independently verified, and retrieved facts checked for relevance and correct use?
- Are failure paths handled: what happens when a tool errors, a search returns nothing, or an observation contradicts the plan?
- Is there a stopping rule: a step cap, a success check, and a human gate for irreversible actions?
- Can you trace the final answer back through the trajectory to the observations that support it?
An agent is not just a chatbot with extra steps. It is a language model inside a loop with tools, memory, and rules for stopping, and its reasoning is only as good as the observations that check it. Judge the loop, not the fluency.
