Local Agent Harnesses Scale From 0.75B to 4.7B, New Study Finds Mingbird Leads
A Grafy Research Lab paper maps five locally‑run agent harnesses across four open‑weight models, showing the Mingbird harness outperforms others at every size.
Source: Grafy Research LabA new pre‑print from the Grafy Research Lab evaluates how five open‑weight agent harnesses behave when paired with four locally deployable language models ranging from 0.75 B to 4.7 B parameters, plus a 35.5 B reference model. The authors call the systematic comparison a “ladder” because every combination of model size and harness forms a cell in a 5 × 5 grid, all scored by a deterministic checker.
What is an agent harness and why does it matter?
An agent harness is a thin software layer that wraps a language model, turning it into a task‑oriented system. The harness is responsible for constructing the prompt, exposing external tools (such as a calculator or web retriever), and deciding when the agent has completed a request. By separating the harness from the underlying model, researchers can isolate whether performance gains come from better prompting logic or from larger model capacity.
How did the study set up its ladder of experiments?
The researchers fixed four open‑weight models – released under permissive licenses – at 0.75 B, 1.3 B, 2.7 B, and 4.7 B parameters. They also included a 35.5 B “anchor” model to provide an upper bound. For each model they ran five publicly available harnesses that differ in how they manage prompts and tool use. The experiments covered 18 artifact‑producing tasks (e.g., code generation, data extraction, summarization), yielding a total of 360 evaluated cells. A single deterministic checker scored every output, ensuring consistent comparison across the entire grid.
Which harness performed best and what were the key findings?
According to the paper, the completion‑controlling harness named Mingbird achieved the highest mean score in every model size, from the smallest 0.75 B model up to the 35.5 B anchor. Four pre‑registered paired contrasts (comparing two harnesses at the same model size) survived Holm‑corrected statistical testing, with two of those significant differences occurring at the extremes – the 0.75 B and 4.7 B models. The results suggest that sophisticated prompt‑management logic can compensate for limited model capacity, especially when the harness can reliably decide when a task is finished.
What does this mean for developers building local AI agents?
The ladder experiment demonstrates a practical trade‑off: smaller models paired with a strong harness can approach the performance of much larger models paired with a weaker harness. For developers constrained by hardware or cost, investing effort in a high‑quality harness – such as Mingbird’s completion‑control mechanisms – may yield more bang for the buck than simply scaling up model size. Moreover, the deterministic checker used in the study offers a template for reproducible benchmarking of future harnesses.
How can Grafy users experiment with these ideas?
Grafy provides several tools that map directly onto the study’s workflow. The Reader app lets you upload the 56‑page PDF, search for specific task definitions, and ask follow‑up questions to the model – ideal for digging into the methodology. To prototype a harness, you can combine Chat (which lets you run multiple open‑weight models side‑by‑side) with Graph, which records prompt variations, tool calls, and decision points in a branching visual history. This mirrors the paper’s approach of keeping the harness logic separate from the model and provides a low‑friction way to iterate on prompt engineering.
Where does this research fit into the broader AI ecosystem?
The work adds empirical weight to a growing consensus that agent‑oriented systems need more than raw parameter count. Prior studies have highlighted the importance of tool use and self‑reflection; this ladder experiment quantifies those benefits across a spectrum of model sizes. As the community moves toward on‑device and edge deployment, the ability to get strong performance from sub‑gigabyte models will be crucial. Harnesses like Mingbird could become a de‑facto standard layer for local agents, similar to how tokenizers and inference wrappers are now commonplace.
What are the next steps for researchers and practitioners?
Future work could expand the ladder to include even larger open‑weight models, add more diverse tasks (such as multimodal generation), or test additional harness designs that incorporate retrieval or planning modules. Practitioners might also explore automated harness synthesis – using meta‑learning to generate prompt‑management policies tailored to a specific model’s strengths. The deterministic checker framework introduced in the paper offers a reusable benchmark for such experiments.
Source: Grafy Research Lab, Harness Leverage Across Model Size: A Ladder of Five Local Agent Harnesses from 0.75B to 4.7B, arXiv abs/2611.21407 (2026‑11‑17).
Frequently asked questions
Does the Mingbird harness work with any language model?
Mingbird was evaluated on four open‑weight models and a 35.5 B anchor, and it outperformed the other four harnesses in every case. While the study shows strong results across that range, the authors note that harness performance can still depend on model architecture and tokenization, so testing on a new model is recommended.
Can I reproduce the ladder experiment using Grafy tools?
Yes. Upload the paper to Reader to extract task definitions, then use Chat to run the same models side‑by‑side. With Graph, you can log each prompt, tool call, and completion decision, effectively recreating the 360‑cell matrix inside Grafy.
Why does a better harness sometimes outrank a larger model?
A harness adds structured reasoning steps, explicit tool use, and a clear termination condition. These additions reduce the model’s need to “guess” the next action, which can be especially helpful for smaller models that have limited internal knowledge. The study’s deterministic checker shows that such systematic guidance can raise the overall quality of outputs beyond what raw parameter count alone provides.


