Choosing an ai agent harness shapes what an enterprise agent can reliably do. IBM Research has now published a controlled comparison that shows how much that choice can matter: with the same open-weight model running the same tasks, changing only the harness moved task completion from 30% to 81%. The experiment does not crown a universal winner, and IBM is explicit about that. What it does show is that the harness is not plumbing around the model. It plans the work, chooses the tools, checks the results, and, in enterprise use, enforces the rules. Understanding those jobs is the key to choosing well.
What IBM actually tested: one model, three harnesses
There are nearly 200 agent harnesses to choose from, according to community lists such as awesome-agent-harness and best-of-Agent-Harnesses that IBM cites. The question IBM set out to answer was not which harness is best in the abstract, but which harness fits a given workload with a given model. To isolate the harness, the team kept the model fixed and changed only the harness.
The three harnesses were CUGA, IBM’s ConfigUrable Generalist Agent; Hermes from Nous Research; and Deep Agents from LangChain. Every harness used the same open-weight model, GLM-5.3-flash, on the same 90 tasks in 30 scenarios from AppWorld, with one attempt per task. All runs used the Harbor evaluation framework, and IBM says its setup is public in its Harbor fork, so others can reproduce the comparison or submit a better configuration for any harness.
AppWorld is a benchmark of multi-step tasks across 9 apps and 457 APIs, covering activities such as email, payments, and file storage. IBM is careful about what that means: the apps are consumer apps, so AppWorld is a proxy. It tests a subset of the capabilities enterprise automation also needs, such as logging in, reading API documentation, paging through results, and changing state across several apps. It does not test enterprise systems directly, and it does not test policy enforcement at all, a limit that matters for the second half of this story.
The setups were not identical, and IBM does not hide that either. CUGA ran in its AppWorld configuration, with AppWorld instructions, task decomposition, verification before execution, and automatic reflection. Hermes ran in Harbor’s default setup, with AppWorld tools over MCP. Deep Agents ran AppWorld tools over MCP plus LangChain’s built-in middleware: tool selection, context summarization, a to-do list, retries, and call limits.
The results: the harness moved completion from 30% to 81%
On task completion, CUGA finished 73 of 90 tasks (81.1%), Hermes finished 42 of 90 (46.7%), and Deep Agents finished 27 of 90 (30.0%). That 51-point spread, from the same model on the same tasks, is the headline, and it echoes an analysis cited from best-of-Agent-Harnesses in which the same GLM-5.2 model went from 23% to 52% on SWE-bench Pro when only the harness changed.
The more useful number, IBM argues, is scenario consistency. AppWorld groups tasks into scenarios with three variants each, and a scenario counts as passed only when all three variants pass. That measure, called SGC, rewards consistency rather than a lucky run. On SGC, CUGA passed 20 of 30 scenarios, Hermes passed 8, and Deep Agents passed 2. The gap widens on consistency, which is exactly what an enterprise buyer should care about: a harness that sometimes completes a workflow is less useful than one that completes it reliably across variations.
IBM’s own framing matters here. The team writes that the result is not a verdict on Hermes or Deep Agents. Hermes is built for always-on personal assistants, and Deep Agents for long, multi-step tasks such as research. Neither was built for multi-app API work like AppWorld. The conclusion is narrower and more useful than a ranking: the harness has to fit the job. A harness optimised for a personal assistant living in chat apps, or for a long research task, should not be expected to excel at driving dozens of APIs across several apps, and vice versa.
Where CUGA’s points come from
Because the model was the same in every run, the differences come from the harness. Every CUGA agent runs the same compact loop: prepare, call the model, execute, and repeat. The same loop scales from one agent with its own tools to a supervisor coordinating several agents. Within that loop, several features did the work in the AppWorld setup.
First, planning. Task decomposition splits a task into steps, one app at a time. Focused tools mean each step sees the tools of its app and a few shared ones, not the full tool set at once. A tool shortlister, find_tools, searches for any other tool a step needs.
Second, acting in code. In CodeAct, the model writes Python that calls the app APIs, and that code runs in a restricted sandbox with a time limit. That structure suits API work: the agent’s actions are explicit, inspectable code rather than free-form tool calls, and the sandbox bounds what a bad step can do.
Third, checking before and after acting. Verification before execution reviews each code block that changes data before it runs. Automatic reflection reviews the results and lets the agent correct its plan. A pagination audit reviews how the agent pages through lists, a small, concrete failure mode that is easy to miss and expensive to get wrong. IBM’s summary is that with these features, the model makes fewer decisions, and the harness catches more mistakes before they reach the result.
Every step the harness owns is an enforcement point
The benchmark measures task completion, but IBM’s enterprise argument is about something the benchmark does not measure: rules. Enterprise agents act on business systems such as CRM and payments, and they must follow rules that hold whatever the model decides. When business rules such as payment approval or keeping customer personal data inside the organisation are placed in the system prompt, IBM’s objection is blunt: a prompt is an instruction, not enforcement. The model can misread it, a user can talk the agent out of it, and an injected message can change it in the middle of a task.
CUGA policies move these rules out of the prompt and into the harness. There are five policy types, acting at four points in a run. At user intent, an Intent Guard blocks or answers matching requests before the agent acts, for example blocking a request to delete all customer records. At agent guidance, a Playbook gives the agent a standard procedure, such as following a refund procedure step by step. At tool execution, two enforcement mechanisms apply: Tool Approval is a hard stop, where the run pauses for a person’s approval and shows the proposed code, for example asking a person before a payment tool runs; and Tool Guide with ToolGuard is a hard check, where guard code runs before the tool and a blocked call returns a policy violation, so the tool is never called, for example blocking a payment over $1,000. At the final response, an Output Formatter shapes the answer to an approved template.
The distinction between the two tool-execution mechanisms is the point. Guidance steers the model, but approvals and ToolGuards are enforced by the runtime. Put rules such as payment approval and personal data handling in the second group, IBM argues, where they hold whatever the model decides. The two counts in IBM’s own materials describe the same system from different angles: the blog post groups the policies into four stages of a run, with Tool Approval and Tool Guide sharing the tool-execution stage, while the paper counts five structural checkpoints, counting Tool Guide and Tool Approvals separately. The five policy types are Intent Guard, Playbook, Tool Guide, Tool Approvals, and Output Formatter.
The Policy Playground makes the difference concrete. The same message, "Pay everything I owe," goes to two live agents, one without policies and one with them. The agent without policies paid $90, including a denied claim in collections. The agent with policies held the payment for approval, then paid only the $10 that was due. The benchmark runs did not use policies, so this example is a demonstration of the policy layer, not a measured result from the comparison. Even so, it illustrates the enterprise question that task completion alone cannot answer: when the agent is wrong, what stops it?
When to pick CUGA — and when to pick something else
CUGA is for automating business work across APIs, apps, and web user interfaces, under rules you can enforce. For that job, IBM points to three further strengths, all reported here with attribution to IBM rather than as independent findings. With hosted models, CUGA held first place on WebArena from February to September 2025 and on AppWorld from July 2025 to February 2026. Deployment can stay inside the customer’s own boundary: CUGA is Apache 2.0 licensed, works with any OpenAI-compatible endpoint, and runs locally, on Kubernetes in a private or sovereign cloud, or even in an air-gapped sovereign cluster, with OIDC sign-in and separate chat and management roles. IBM Sovereign Core builds on CUGA to run agents fully inside the customer’s boundary. Cost is measurable through budgets that limit spending and run receipts. And configuration, not code, sets tools, budgets, and policies.
IBM is equally clear about when not to pick CUGA. For a terminal coding agent, it names opencode, Codex, or Claude Code. For a personal, always-on assistant that lives in chat apps, it names OpenClaw or Hermes. For a blank-canvas graph framework, it names LangGraph directly, noting that CUGA is built on it.
How to choose an AI agent harness in practice, and the limits of this comparison
Start with the job, not the leaderboard. What apps and tools must the agent drive, and how many? What rules must hold regardless of what the model decides, and which of those rules need a hard stop or a hard check rather than a prompt? Where must the system run, and does it need to run inside your own boundary, including a sovereign or air-gapped environment? A harness built for a personal assistant will not necessarily answer those questions well, however good it is at its own job.
Then ask how the harness reduces decisions and catches mistakes. Does it decompose work and narrow the tools visible at each step? Does it verify data-changing actions before they run, and reflect on results after? Does it audit the failure modes of your workload, such as paging, retries, and partial results? And does it enforce rules at runtime, with approvals and guards that hold even when the model is misled, or does it merely instruct the model to behave? Treat those questions as considerations for evaluating workload fit and reliability, alongside benchmark scores.
Finally, keep the limits of this comparison in view. It is a 90-task sample on one benchmark, so the results may not generalise to other tasks. The setups were not identical, which is inherent to comparing harnesses as they are configured, but it means the result is a comparison of configurations, not of abstract designs. AppWorld’s consumer apps are a proxy for enterprise systems, not a substitute for testing on your own systems. And the runs did not use policies, so policy enforcement itself was not measured in the comparison; the policy argument rests on the design and the Playground demonstration, not on the 81% figure. IBM has invited better configurations for Hermes and Deep Agents through its public Harbor fork, with a promise to rerun them on the same tasks and publish the results whichever harness wins. Until such reruns appear, the honest reading is this: the harness matters, a lot, and the right harness is the one whose planning, checking, and enforcement fit the work you actually need done.
