State of the Art in Agent Harness Design: What Actually Moves Task Success

Executive summary

The strongest evidence from 2023–2026 is that the harness is not merely packaging around the model. Several non-model choices produce double-digit changes in task success with the model held fixed: tool-interface design, recovery semantics, context preservation, tool routing, and the match between orchestration architecture and task structure. Conversely, some choices that receive substantial engineering attention—framework DSL, protocol envelope, dashboard UI, or whether a logically sequential loop is implemented with synchronous versus asynchronous application code—have little or no controlled evidence of improving task success by themselves.

The most important finding is that architecture should follow the dependency structure of the task. In a 2026 controlled study of 260 configurations across six benchmarks, centralized multi-agent orchestration improved Finance-Agent from 0.349 to 0.631, a +80.8% relative gain, yet reduced PlanCraft from 0.568 to 0.282, a −50.3% relative loss; an independent-agent architecture was even worse on PlanCraft at −70.0%. Agent count itself had no statistically significant direct effect after controlling for other variables, while tool diversity and coordination-task interactions did. [1][2] Anthropic reports a compatible but less controlled production result: its breadth-first research system with an orchestrator and parallel subagents beat a single-agent configuration by 90.2% on an internal research evaluation, at the cost of roughly 15× chat token consumption. [3]

Tool-interface design has some of the cleanest causal evidence. With GPT-4 Turbo fixed on SWE-bench Lite, SWE-agent's purpose-built agent-computer interface resolved 18.0% of tasks versus 11.0% for the shell-only agent. Within the interface, summarized search scored 18.0% versus 12.0% for an inefficient iterative search; a linting edit guardrail scored 18.0% versus 15.0% without linting; and a 100-line file window scored 18.0% versus 14.3% for 30 lines and 12.7% for whole-file viewing. 4(https://github.com/SWE-agent/SWE-agent)5(https://github.com/SWE-agent/SWE-agent) CodeAct independently found that changing the action representation from JSON/text to executable code raised GPT-4-1106-preview success on a multi-tool benchmark from 53.7%/52.4% to 74.4%, while reducing average interaction turns from about 7.6–7.7 to 5.5. [6]7(https://github.com/xingyaoww/code-act) These results say that **affordances, feedback, o

Agent Harness Effectiveness

What we found

Scope: this holds for software-engineering and stateful agent tasks under a fixed model. On frontier models, competent harnesses plateau on correctness and differ only on cost; on weak models the cross-harness spread opens up.

The five ideas most worth stealing:

1. The interface is the lever, the loop is not. SWE-agent's central ablation shows the agent-computer interface (ACI) moves SWE-bench Lite from 2.67% (RAG) to 18.00% under the same GPT-4 Turbo, while removing the autonomous loop entirely (Agentless) gains — 32.00% vs the best open-source agent's 30.67%, at $0.70/issue. The harness's value is "actions the agent can execute and feedback it can read," not "the agent gets to act autonomously." Measured, not asserted.papers.nips.ccarxiv.org

2. Watch the run and intervene — that's a measured ~10-point lever. Online process-monitoring (trajectory graph + automatic intervention) raises issue-resolution 6.9–23.5% across models at <10ms overhead, and a cleaner rule-based follow-up (LivePlan) independently reproduces +9.9pp avg / +15.2pp max. Telling the agent what to do (plan content) is noise; watching what it does and stepping in is measured-positive on the pathological tail. But note the denominator is ~10% "lucky passes," so part of the gain is converting luck into verified success, not new skill.arxiv.orgarxiv.orgmicrosoft.com

3. Components don't stack — the "all-in" harness loses. A full 2⁵ factorial (32,000+ evals) finds the agent bundling every scaffold component underperforms the best subset in every setting: Tool Use alone beats All-In by 32% on HotpotQA, and 56.3% of triples violate submodularity with 90% of violations being sign-flips (adding a component makes it worse). Tool Use carries ~70% of the value; Planning is significantly negative. Do not "add until marginal turns negative" — that greedy rule is empiri