Fastino: agent harness swings benchmark scores widely
A Fastino engineering guide argues that the software around a frozen model — tools, context, control flow — can shift agent benchmark results by tens of points, making the harness itself an optimization target.
Fastino has published a practical guide to "harness engineering". The guide defines the harness as the software that assembles context, exposes tools, enforces invariants, orchestrates control flow and preserves state around a language model. Its central claim is that the harness, not just the model weights, determines how much of a model's capability shows up in practice.
The guide backs this up with benchmark spreads in which the weights stay fixed:
- Berkeley Function Calling Leaderboard: Claude Opus 4.5's V4 overall score rises from 33.5% to 77.5%, a 44-point gap, when functions move from prompt-text descriptions to native schemas. Single-turn accuracy is roughly equal in both setups. Almost all of the gap comes from the multi-turn, web-search and memory categories.
- SWE-bench Verified: the same GPT-4o snapshot scores 23.2% with the model-driven SWE-agent loop and 38.8% with the fixed Agentless-1.5 pipeline. The authors say part of that gain comes from extra inference, since Agentless samples about 40 candidate patches per issue.
- Nine off-the-shelf harnesses: Nemotron 3.5 Super VL resolved between 30 and 76 of the same 100 SWE-bench Verified tasks, depending on the harness. Cursor CLI and Devin were left out because they can't target a self-hosted model.
The first of four stated design principles is to "minimize the decision surface". That means giving the model fewer, higher-level actions and running deterministic steps such as patching, linting and formatting in code instead of through the model. As support, the guide cites SWE-agent's Agent-Computer Interface, which scored 18.0% against 11.0% for a shell-only agent on SWE-bench Lite with GPT-4 Turbo.
The excerpt available covers only this first principle. The other three principles, and the guide's methods for benchmarking and searching across harness designs, were not included in the material reviewed.