The small local model test
A user on r/openclaw, posting as GapNew4766, set out to answer a practical question: which AI agents actually work well on a small local model? The test he reported used Qwen3.5-4B and ten agents, each run with its own isolated settings folder, headless, with auto-confirmation turned on and telemetry turned off where possible. It was not a grand benchmark of artificial intelligence. It was a workbench test, and that is exactly why it is interesting.
Each agent was given the same five basic tasks, with ten minutes per task, a fresh copy of the fixture every time, and an automated check script to judge the result. The tasks were deliberately ordinary. Read a database port from a YAML file and write it to a file. Write a stats script, run it, and save the output. Fix a bug in a slugify function so the tests pass, with hidden examples designed to catch overfitting. Count ERROR lines across three log files and find the most frequent error. Rename a function across three files without changing what the program does. These are basic tasks, and the ranking does not establish performance on longer, harder or interactive work.
The ranking
Pi, version 1.3.0, finished first. Its total time across the five tasks was 1 minute 36 seconds, with individual times of 10, 14, 24, 19 and 29 seconds, and it completed all five tasks. The post describes Pi as follows: the model barely has to think about any tool formats.
Goose, version 1.55.0, came second at 2 minutes 51 seconds, also completing all five tasks, with times of 16, 18, 26, 45 and 66 seconds. The post calls it solid and consistent, and says it completed five tasks without fumbling. Crush, version 0.98.1, came third at 3 minutes 31 seconds, described as a clean run, slightly more verbose in reasoning but getting everything done without a hiccup.
Those three results are the firm ground in this story. They are reported times for reported tasks, judged by a check script, on one model, in one tester’s setup. Everything beyond them is interpretation, and the discussion around the post has been unusually good at saying so.
The author’s guess
In a reply, the author guessed Pi’s advantage was four tools and a short system prompt, and that on a 4B model every extra token of instructions shows up in the timing. That guess was framed by the author as a guess, not a measured result. That guess is attractive because it fits the numbers. A small model has less room to spare, and the author’s guess is that anything the agent makes it read, parse and keep in mind before it even starts the task adds to that cost. Whether that is what happened here remains unmeasured.
It is also, by the author’s own framing, still a guess. The test did not measure token counts or tool-call counts, and it did not compare the effect of tool-format friction with the effect of an agent’s feature list. A separate profile reporting Pi’s default tools of read, write, edit and bash is consistent with the guess, but that architectural detail is not evidence explaining benchmark performance, and consistency is not proof. The honest version of the story is not that tool-format friction explains the ranking. It is that the ranking raises that question, and the test as reported cannot yet answer it.
What the ranking may be hiding
Commenters pressed on exactly that gap, and their questions are the most useful part of the thread. One asked whether the runs were single runs per task, and suggested repeating each agent and task pairing and reporting pass rates alongside median times, so readers could tell whether the ranking is stable or whether a single lucky or unlucky run moved an agent up or down the list.
Another suggested separating two very different kinds of failure in the next round: an agent that could not start with a backend, and an agent that started, ran, and failed a task. A startup error is useful compatibility data, the commenter argued, but it does not tell you how that agent would have performed on the five tasks. Counting it the same way as a task failure would make the ranking harder to read, not easier.
A follow-up writeup of the same 50-run test, ten agents by five tasks, reported that six agents passed all five tasks: Pi, Goose, Crush, Cline, Atomic Agent and Opencode. It also reported that tool calls were not the main failure mode, and listed other failure modes. The failure modes reported were exact-match edits, overfitting to visible test data, miscounting, and vague instructions followed literally. Those are other failure modes reported in the follow-up writeup. That finding is about tool-call failures, not about the broader prompt-overhead guess, which remains untested, and it does not by itself exclude any contribution from tool formats.
The hardware question underneath
The other thread running through the comments is hardware, and here the author was refreshingly blunt. Asked what machine the test ran on, he said readers do not need anything close to an expensive new setup. The 5B model at Q4, he said, is a file of about 2.7 gigabytes, and it runs fine on an 8 or 16 gigabyte Mac, or on any graphics card with 6 to 8 gigabytes of video memory. The M5 machine used for the test, he added, was overkill.
For a 16 gigabyte machine, his advice was specific: stick to models in the 4B to 9B range at Q4, and keep the context around 16,000 to 32,000 tokens, because a big context is what kills speed on smaller Macs. On the author’s advice, a small local setup works best when the model, the quantisation and the context window are all kept within what the machine can actually do. That is advice, not a measured result, and the test does not by itself establish what happens when any one of them grows.
What happens next
Questions for a next round, as listed in the research, include repeat runs with pass rates and median times, token and tool-call logging, and a sub-2B comparison using the same fixtures and checks. Those questions, taken together, would turn a lively ranking into a test that could actually settle the question it raised.
Until then, the right way to read this test is narrow and useful. On Qwen3.5-4B, on five basic tasks, with a fresh fixture and an automated check, Pi finished first, Goose second and Crush third. Whether that order reflects tool-format friction, or something else entirely, remains an open question, and answering it would need the token and tool-call measurements the research lists as still missing. That is not a weakness in the story. It is the story.
