AI agent risks are the reason the most useful deployment advice about agents is also the most boring: start supervised, scope the tools tightly, and put a human in front of every irreversible action. An agent is not a smarter chatbot. It is software that takes a goal, plans its own steps, calls tools, and acts on the results — and that autonomy is exactly what turns an ordinary model mistake into a real-world action. This article maps the main risks and limitations, grouped by where they originate, pairs each risk with a mitigation that works, and ends with a checklist for deciding whether an agent fits your task at all.
The stakes are not theoretical. In McKinsey’s November 2025 State of AI survey of 1,993 participants across 105 nations, 51% of organisations using AI reported at least one negative consequence, with inaccuracy the most common at 30%, followed by cybersecurity concerns at 10% and regulatory compliance at 8%. Agents do not create the inaccuracy problem, but they amplify it: a wrong answer from a chatbot can be checked and corrected, while the same wrong answer inside an agent can trigger a payment, a message, or a deletion before anyone notices.
What an AI agent is — and why autonomy changes the risk profile
Anthropic draws a clean architectural line that makes the whole risk picture easier to hold. Workflows are systems where large language models and tools are orchestrated through predefined code paths: your code stays in charge. Agents are systems where the model dynamically directs its own processes and tool usage. In Anthropic’s own words, agents are typically just language models using tools based on environmental feedback in a loop — the model calls a tool, looks at the result, decides the next step, and repeats until the task is done or a stopping condition is reached.
That loop is where the risk profile changes. A chatbot’s wrong answer stops at the screen, where a person can check it before acting on it. An agent’s wrong answer can execute directly: it can book, send, delete, or publish something before anyone checks it. The agent also gets its sense of reality from tool results — what Anthropic calls ground truth from the environment at each step — so a misleading tool result does not just confuse the model, it steers the next action. Two kinds of risk follow: inherent model limits, which no amount of careful deployment fully removes, and deployment choices, which you can constrain and mitigate, though not eliminate. You cannot prompt your way out of a model that hallucinates, but you can limit what it is allowed to act on.
Reliability risks: when the agent gets it wrong
Reliability is the most common failure class, and it starts with the same weakness every language model has: it can produce answers that are confident, complete, and wrong. From a chatbot, that is an embarrassment. From an agent, it is an error with consequences, because the agent treats its own output as the input to its next step. A wrong fact retrieved early in a multi-step run can shape every step that follows.
Planning errors can compound the same way: an agent that misjudges how to break a goal into subtasks can execute the wrong plan thoroughly, and its progress reports can look convincing throughout, because each step genuinely completed. This is the silent-failure mode: the agent can report success it did not achieve, formatted as a tidy summary. McKinsey’s 2025 survey put a number on the underlying problem: inaccuracy was rated a relevant risk by 74% of respondents, up from 60% in 2024. That survey covers AI use generally, not agents specifically — but it shows how common the underlying inaccuracy problem is, and in an agent the human check that would catch it is often the step the system was built to skip.
Hypothetical example: an agent asked to reconcile monthly expenses pulls the wrong month’s bank statement, treats it as ground truth, and carries the error into a summary, a budget adjustment, and a message to a supplier — three actions, one initial mistake. A checkpoint asking a human to confirm the statement’s date before the agent acts would have caught it at step one.
Brittleness is the quieter cousin of outright error. Small changes in an input, a tool’s response format, or a web page’s layout can derail a run that worked yesterday, and the agent may not distinguish a meaningful change from a cosmetic one, so it improvises — sometimes well, sometimes by inventing a workaround that looks plausible and is wrong.
Security risks: prompt injection, tool misuse and data exposure
The security risks of agents are genuinely different from those of a chatbot, and they start with a fact that surprises most teams: the model cannot reliably tell instructions apart from data. Indirect prompt injection hides instructions in content the agent reads — a web page, an email, a ticket, a tool result — rather than in the user’s prompt. For an agent it is the most direct route to a hijacked goal, because the injected text arrives through a trusted tool channel and can steer the next tool call. The clearest public demonstration is EchoLeak: in mid-2025, a zero-click indirect prompt injection (CVE-2025-32711, CVSS 9.3) in Microsoft 365 Copilot let a single crafted email cause Copilot to extract data from OneDrive and SharePoint and exfiltrate it, with no user interaction required. The attack needed no malware and no user click: the injected instructions arrived through a channel the agent trusted, and the agent’s own legitimate tools did the rest — the pattern OWASP calls tool misuse, where a valid tool is driven to an illegitimate end. Google’s threat intelligence team found the malicious category of prompt injection in scanned web pages grew 32% between November 2025 and February 2026.
OWASP’s GenAI Security Project gave this threat landscape its own catalogue in December 2025, releasing the Top 10 for Agentic Applications: ten agent-specific risks from ASI01 Agent Goal Hijack through ASI10 Rogue Agents, developed with more than 100 industry experts. Three entries matter most. ASI01 Agent Goal Hijack is the foundational threat — an attacker redirects the agent’s objective through content it reads, so it pursues the attacker’s goal while believing it still serves the user. ASI02 Tool Misuse and Exploitation is the operational consequence: a legitimate agent uses legitimate tools in destructive ways, such as chaining API calls in sequences nobody anticipated. ASI03 Identity and Privilege Abuse covers the credentials the agent carries, and ASI06 Memory and Context Poisoning covers compromised information planted in persistent memory so it poisons later sessions.
Underneath these sits OWASP’s older excessive-agency entry (LLM06), which names the root causes plainly: excessive functionality, excessive permissions, and excessive autonomy. Tool misuse is what happens when those three meet a manipulated input. The academic literature agrees: the arXiv survey “Trustworthy Agentic AI” organises agent failure modes across five dimensions — safety and robustness, alignment and human oversight, transparency and auditability, privacy and data governance, and regulatory compliance — and names indirect prompt injection, backdoor triggers, goal misgeneralisation, memory contamination, and cross-session data leakage among them. NIST’s Center for AI Standards and Innovation puts it bluntly: agent systems are capable of taking autonomous actions that impact real-world systems or environments, and may be susceptible to hijacking, backdoor attacks, and other exploits.
Operational limitations: context, cost and scale
Even with no attacker and no error, agents hit practical ceilings. The first is context. An agent’s working memory is its context window, and long or complex tasks fill it with tool results, retries, and corrections. As the context grows, the earliest instructions can get crowded out, and the agent may lose track of constraints set at the start of the run. This is a limit on the model’s working memory. It is different from the automatic guardrails around the run — input, output, and tool checks that validate behaviour at defined points, and human review that pauses the run for approval — which are enforced by the surrounding system rather than held in the model’s context.
The second ceiling is cost and latency. Anthropic states the trade-off directly: agentic systems often trade latency and cost for better task performance. Each step of an agent run consumes model calls and tool calls, so cost and latency add up across a run.
The third ceiling is observability. When an agent fails, reproducing the failure means replaying the full trace of model calls and tool results, and the same input may not produce the same failure twice. Debugging an agent is closer to debugging a colleague’s judgement than debugging code: you can see what it did, but not quite why — and the explanation it gives afterwards is itself a model output.
Governance and accountability gaps
When an autonomous system acts, somebody is still responsible — but the chain of responsibility is harder to trace than in any system most organisations have run before. Attribution is the first gap: a bad outcome may involve a model decision, a tool result, a human approval given too quickly, and a guardrail that did not fire. Audit logs must therefore record who approved what, what was reviewed, and what the result was, and those logs must remain independently reviewable and protected against rewriting by the agent itself.
Compliance is the second gap. When agents handle personal or regulated data, the organisation inherits the exposure: a privacy breach caused by an agent’s tool call is still a privacy breach. The third gap is the delegation question itself. Some decisions should never be delegated — hiring and firing, credit decisions, medical treatment, legal commitments — and the line has to be drawn in policy, not left to the agent’s judgement.
Where AI agents should not be used yet
Do not hand an agent an irreversible consequential action without a human approval step — payments, deletions, messages sent in your name, changes to production systems. Do not deploy an agent on top of data you would not trust a person to act on unsupervised. And where the steps can be written down in advance, consider a deterministic workflow first, and compare the actual requirements, cost, and latency before choosing.
Practical mitigations for AI agent risks
Every risk class above has a concrete control, and the controls are mostly unglamorous. A core one is risk-tiered human approval: low-risk work such as drafting a summary can run automatically, medium-risk work such as a drafted customer reply can be sampled and audited, while sending it needs approval, and high-risk, consequential or irreversible actions such as a payment or a policy decision should require real-time human approval. The OpenAI Agents SDK makes this a first-class concept: guardrails are automatic checks on input, output, or tool behaviour, while human review pauses the run so a person can approve or reject a sensitive action.
A risk-tiered policy is worth writing down before the first deployment. As a starting point:
{
"approval_policy": {
"automatic": ["draft_summary", "search_internal_docs", "classify_ticket"],
"sampled_audit": ["draft_customer_reply", "update_crm_record"],
"human_approval_required": ["issue_refund", "send_external_message", "delete_record", "change_production_config"]
},
"tool_scoping": {
"principle": "least_privilege",
"rule": "each agent gets its own identity and only the permissions its tools need"
}
}Least-privilege tool scoping is the second control, and it limits the blast radius of every other failure. An agent’s credentials should only allow what its tools need, and each agent should have its own identity so its actions can be traced. Sandboxed testing is the third: Anthropic recommends extensive testing in sandboxed environments with appropriate guardrails, and stopping conditions such as a maximum number of iterations to keep a run from looping. The fourth is tracing — log every step, every tool call, and every result, because you will need the trace the first time something surprising happens. Finally, deploy in stages: start supervised, with humans approving consequential actions, and loosen controls only as reliability is proven over real traffic.
A checklist for deciding whether an agent fits your task
Before you build an agent, run the task through these questions. If the answer to the first three is not comfortable, the task is not ready for an agent — and the honest answer may be a workflow, a supervised agent, or no agent at all.
- Is the outcome reversible? If the agent’s worst mistake cannot be undone, the task needs human approval at that step, full stop.
- How much does an error cost, and how visible is it? Cheap, obvious errors are easier to automate — but only once the other checks on this list pass. Expensive, silent ones are not.
- Can you trust the data and tools involved? Unverified inputs or poisonable tool outputs mean the agent acts on bad information at machine speed.
- Can a human approve at the consequential points? If the workflow cannot pause for a person before an irreversible action — a payment, a deletion, or an external message — do not automate that step.
- Could a workflow do it instead? If the steps are predictable, a fixed workflow is a strong candidate — compare the actual requirements, cost, and latency before choosing, and use the agent only for the open-ended part.
- Do you have tracing and an audit trail? If you cannot replay a run and see what the agent did and approved, you are flying blind.
- Have you tested it in a sandbox? Test with guardrails on, then test with someone trying to break it.
AI agents are genuinely useful for open-ended work where the steps cannot be known in advance — and genuinely dangerous when they are given irreversible actions, broad credentials, and nobody watching. The risks are not a reason to avoid agents. They are a reason to scope them tightly, approve the consequential steps, and keep a human in the loop until the system has earned the trust it is asking for.
