# State of the Art in Agent Harness Design: What Actually Moves Task Success

## Executive summary

The strongest evidence from 2023–2026 is that **the harness is not merely packaging around the model**. Several non-model choices produce double-digit changes in task success with the model held fixed: tool-interface design, recovery semantics, context preservation, tool routing, and the match between orchestration architecture and task structure. Conversely, some choices that receive substantial engineering attention—framework DSL, protocol envelope, dashboard UI, or whether a logically sequential loop is implemented with synchronous versus asynchronous application code—have little or no controlled evidence of improving task success by themselves.

The most important finding is that **architecture should follow the dependency structure of the task**. In a 2026 controlled study of 260 configurations across six benchmarks, centralized multi-agent orchestration improved Finance-Agent from 0.349 to 0.631, a **+80.8% relative gain**, yet reduced PlanCraft from 0.568 to 0.282, a **−50.3% relative loss**; an independent-agent architecture was even worse on PlanCraft at −70.0%. Agent count itself had no statistically significant direct effect after controlling for other variables, while tool diversity and coordination-task interactions did.  [1][2] Anthropic reports a compatible but less controlled production result: its breadth-first research system with an orchestrator and parallel subagents beat a single-agent configuration by **90.2%** on an internal research evaluation, at the cost of roughly **15× chat token consumption**.  [3]

**Tool-interface design has some of the cleanest causal evidence.** With GPT-4 Turbo fixed on SWE-bench Lite, SWE-agent's purpose-built agent-computer interface resolved 18.0% of tasks versus 11.0% for the shell-only agent. Within the interface, summarized search scored 18.0% versus 12.0% for an inefficient iterative search; a linting edit guardrail scored 18.0% versus 15.0% without linting; and a 100-line file window scored 18.0% versus 14.3% for 30 lines and 12.7% for whole-file viewing.  [[4]](https://github.com/SWE-agent/SWE-agent)[[5]](https://github.com/SWE-agent/SWE-agent) CodeAct independently found that changing the action representation from JSON/text to executable code raised GPT-4-1106-preview success on a multi-tool benchmark from 53.7%/52.4% to **74.4%**, while reducing average interaction turns from about 7.6–7.7 to 5.5.  [6][[7]](https://github.com/xingyaoww/code-act) These results say that **affordances, feedback, output scope, and composability matter**; they do not establish that a particular brand of tool protocol matters.

**Planning and parallel execution are valuable when they remove avoidable serialization, not merely because a system has a “planner.”** ReWOO reported a **5× token-efficiency improvement and +4% accuracy on HotpotQA** by decoupling planning from tool observations.  [8][9] LLMCompiler's planner/DAG executor reported up to **3.7× lower latency, 6.7× cost savings, and ~9% accuracy improvement** over ReAct-style sequential baselines.  [[10]](https://github.com/SqueezeAILab/LLMCompiler) More recent AsyncFC isolates asynchronous execution below the model and reports preserved task accuracy while overlapping decoding and independent function execution; its authors explicitly note that strictly sequential tasks or negligible tool latency provide little opportunity for speedup.  [11][12] Thus “async” is mostly a **latency architecture**, unless it changes which work can be attempted or how much work fits inside the task budget.

**Context compression is not semantics-preserving.** Governance Decay, evaluating 1,323 agent episodes across seven models, found that compaction raised governance-constraint violations from **0% to 30% on average**, reaching as high as 59%; when a constraint survived the summary, violation was 0%, whereas a dropped constraint was violated 38% of the time. Pinning roughly 47 tokens of constraints outside the lossy summary restored violations to **0% at under 0.5% token overhead**.  [13] A separate systems study shows synchronous compaction itself can consume **51.3–62.4% of end-to-end wall time** at a 16K threshold for two tested open models, while parallel block compaction delivered roughly **1.37–2.13× compaction throughput improvements** in tested settings.  [14][15][16] The practical conclusion is unusually strong: **hard requirements, permissions, user constraints, completion criteria, and identifiers should not live solely inside a lossy summary.**

**Long-term memory is useful, but “add a vector database” is not a generally valid prescription.** A broad 2026 evaluation of memory substrates found that, on ALFWorld-unseen with Qwen3-32B, no memory achieved 22.4% task success while the best refinement-style memory reached **32.1% (+9.7 percentage points)**. Yet on BigCodeBench-Hard, one flat memory improved 17.6% to 19.6%, whereas a more expensive structural memory fell to 16.2% and incurred roughly sixfold latency in the authors' analysis.  [17][18][19] A separate 400-turn cost study found no memory design dominated both accuracy and cost; depending on workload, the break-even point against full-history prompting ranged from immediate to beyond the entire 400-turn experiment.  [20][21][22] **Memory policy is workload- and model-dependent; storage technology is secondary.**

**Retries work best when failure produces new information.** Reflexion increased HumanEval pass@1 from GPT-4's 80.1% baseline to **91.0%**, and improved ALFWorld by 22 percentage points. More revealingly, on a difficult HumanEval-Rust subset, reflection without grounded generated tests reduced success from 0.60 to 0.52; tests alone gave 0.60; tests plus reflection reached 0.68.  [[23]](https://github.com/noahshinn024/reflexion) A subsequent retrial study found that simple retries can be more cost-effective than elaborate reflection/tree-search methods under matched inference budgets.  [24] The operational lesson is: **do not blindly replay a failed trajectory; change evidence, state, search path, or stochastic sample.**

The most striking recent reliability result is **stateful rewind rather than restart**. AgentRewind increased success on its 82-task long-horizon engineering benchmark from **62.2% to 87.8%** with GPT-5.4/mini-SWE-agent, from 58.5% to 81.7% with a function-calling harness, and from 67.1% to 82.9% with a CodeAgent harness. Removing environment rewind collapsed the 87.8% result to 43.9%, showing that restoring chat context without restoring mutated state is insufficient.  [25] Crab independently found **100% recovery correctness** from sandbox crashes versus only 8–13% for chat-only recovery and 28–42% for chat-plus-filesystem recovery, while remaining within 1.9% of no-fault execution time.  [26] Both are August/April 2026 preprints and should be treated as promising rather than settled replication-level evidence.

### Bottom-line verdict

| Design choice | Does it measurably affect success? | Best evidence | Effect on latency/cost | Verdict |
|---|---:|---|---|---|
| Reactive tool loop vs non-interactive generation | **Yes** | ReAct: +34 pp ALFWorld, +10 pp WebShop over prior imitation/RL approaches.  [27] | More interaction/tool calls; exact comparable cost **unspecified** | Functional |
| Planner/DAG vs fully sequential ReAct | **Yes, task-dependent** | ReWOO +4% HotpotQA with 5× token efficiency; LLMCompiler up to +~9% accuracy.  [8][[10]](https://github.com/SqueezeAILab/LLMCompiler) | Up to 3.7× latency and 6.7× cost improvement for LLMCompiler.  [[10]](https://github.com/SqueezeAILab/LLMCompiler) | Functional |
| Async vs sync execution | Usually latency, not accuracy | AsyncFC preserves accuracy while overlapping work; benefits disappear for sequential/low-latency tools.  [11][12] | Potentially substantial when calls are independent; aggregate comparison in source excerpt **unspecified** | Functional for latency; often stylistic for sequential work |
| Hierarchical vs flat multi-agent | **Very strongly task-dependent** | +80.8% Finance-Agent, −50.3% PlanCraft for centralized orchestration.  [1] | Coordination overhead 58–515% depending architecture.  [2] | Functional, but no universal winner |
| Number of agents by itself | **No independent effect detected** | Direct agent-count coefficient β=.040, p=.487 in controlled study.  [2] | Usually increases tokens/messages | Often stylistic unless it enables useful parallel decomposition |
| Purpose-built tool affordances | **Yes** | SWE-agent 18.0% vs shell 11.0%; multiple direct ablations.  [[4]](https://github.com/SWE-agent/SWE-agent)[[5]](https://github.com/SWE-agent/SWE-agent) | SWE-agent Lite average solved-task API cost $1.67 vs shell $1.46 in that historical setup.  [[4]](https://github.com/SWE-agent/SWE-agent) | Functional |
| Code action vs JSON/text | **Yes in complex multi-tool tasks; model-dependent** | GPT-4: 74.4% vs 52.4–53.7%; 5.5 vs 7.6–7.7 turns.  [[7]](https://github.com/xingyaoww/code-act) | Fewer turns; dollar cost **unspecified** | Functional when composition/control flow matter |
| Tool routing/retrieval | **Yes at scale** | ToolRet-trained retrievers lifted downstream agent pass rates by 10–20%.  [28] | Retrieval adds overhead but prevents huge catalogs entering context | Functional |
| MCP vs native function-calling protocol | **No controlled success effect found** | Large MCP benchmarks evaluate retrieval/composition rather than an equal-semantics protocol ablation.  [29] | Protocol overhead comparison **unspecified** | Primarily interoperability/style until shown otherwise |
| Sandboxing / least privilege | **Yes for robustness/security** | Prismata: attack success 85.5%→0.7%; user-task completion under attack 4.5%→23.0%.  [30] | Runtime cost **unspecified** in retrieved primary result | Functional |
| Summarization/compaction policy | **Yes, potentially catastrophic** | Constraint violations 0→30%; pinning restores 0%.  [13] | Can consume >50% wall time at aggressive thresholds.  [15] | Functional |
| RAG/persistent memory | **Yes, but non-monotonic** | ALFWorld +9.7 pp best case; some memory variants hurt code tasks.  [17][18] | Can range from cheap to ~orders-of-magnitude slower depending substrate.  [18][19] | Functional, workload-specific |
| Retry without new evidence | Sometimes | Simple retrials can outperform more elaborate reasoning under fixed cost.  [24] | Linear-ish inference cost until success/budget exhaustion | Useful but not sufficient |
| Retry with verifier feedback | **Yes** | Reflexion 80.1%→91.0% HumanEval; grounded-feedback ablation shows synergy.  [[23]](https://github.com/noahshinn024/reflexion) | Additional inference/tests | Strongly functional |
| Context + environment rollback | **Yes, large recent effects** | AgentRewind +15.8 to +25.6 pp across three harnesses.  [25] | Crab within 1.9% of no-fault; rollback tool up to −29% wall time.  [26] | Strongly functional |
| Human clarification / HITL | **Yes when task information is missing** | Workspace-Bench human-agent collaboration 80.7%, significantly above autonomous execution.  [31] | Human latency/cost not normalized in study | Functional at ambiguity/irreversibility boundaries |
| Monitoring/observability | Detection mechanisms: **yes**; dashboard style: unproven | Telemetry monitor content-failure detection 0.28→0.59; completion checker caught 7/7 silent aborts.  [32] | Microseconds/step reported for telemetry monitors.  [32] | Signals functional; presentation largely stylistic |

## Evidence framework and architecture map

The report uses a deliberately strict distinction between **harness causality** and “a newer system got a higher benchmark score.” The most persuasive results hold the base model, task set, and evaluation fixed while changing a non-model component—for example SWE-agent's interface ablations, AgentRewind's execution-strategy comparisons, and the controlled multi-agent scaling study. Cross-paper leaderboard comparisons are much weaker because models, token budgets, prompts, benchmark versions, infrastructure, and contamination risk can all change simultaneously.  [33][25][2]

This distinction matters for seemingly contradictory results. SWE-agent showed a large advantage for purpose-built search/edit primitives with GPT-4 Turbo, yet later mini-SWE-agent demonstrated that a roughly 100-line, bash-oriented harness can exceed 74% on SWE-bench Verified with much stronger later-generation models.  [[4]](https://github.com/SWE-agent/SWE-agent)[34] That does **not** invalidate the fixed-model SWE-agent ablation; it indicates that the size of a harness advantage depends on model capability. It is therefore unsafe to conclude either “specialized tools are always necessary” or “bash is always enough.”

A useful contemporary harness can be represented as five separable planes: control, execution, context, recovery, and governance. The exact framework classes or graph DSL are less important than whether those semantic responsibilities exist.

```mermaid
flowchart TD
    U[User task] --> C[Controller]

    C --> L[LLM decision]
    L --> D{Next operation}

    D -->|Direct tool| R[Tool router / policy gate]
    D -->|Plan or delegate| P[Planner / orchestrator]

    P --> W1[Worker or DAG node]
    P --> W2[Worker or DAG node]
    W1 --> R
    W2 --> R

    R --> X[Sandbox / executor]
    X --> O[Typed observation<br/>result + error + state delta]

    O --> V{Verifier / monitor}
    V -->|continue| K[Working context]
    V -->|retry with evidence| K
    V -->|rewind| Q[Checkpoint restore<br/>context + environment]
    V -->|high risk / ambiguous| H[Human]
    Q --> K
    H --> K

    K --> L

    K <--> M[External memory / artifacts]
    G[Pinned constraints<br/>permissions + acceptance criteria] --> L
```

The evidence favors making these planes independently replaceable because their benefits arise through different mechanisms: LLMCompiler improves scheduling; SWE-agent changes the interface between model and machine; memory systems change retention/retrieval; AgentRewind changes state transitions after failure; and least-privilege defenses constrain which effects an otherwise capable agent may externalize.  [[10]](https://github.com/SqueezeAILab/LLMCompiler)[[5]](https://github.com/SWE-agent/SWE-agent)[17][25][30]

### Synchronous and asynchronous execution are not the same architectural question as reactive and planned control

These dimensions are often conflated.

```mermaid
flowchart LR
    subgraph S["Reactive synchronous loop"]
        S1[Observe] --> S2[Reason / choose]
        S2 --> S3[One tool call]
        S3 --> S4[Wait]
        S4 --> S1
    end

    subgraph A["Planned asynchronous DAG"]
        A1[Plan dependencies] --> A2{Ready nodes}
        A2 --> A3[Tool A]
        A2 --> A4[Tool B]
        A2 --> A5[Tool C]
        A3 --> A6[Join / update state]
        A4 --> A6
        A5 --> A6
        A6 --> A2
    end
```

A reactive agent can use asynchronous tool execution, and a planner can still execute synchronously. The measurable question for async execution is mostly whether tool calls have enough **independent critical-path latency** to overlap; the measurable question for planning is whether early dependency reasoning prevents wasted calls or creates brittle commitments. ReWOO, LLMCompiler, and AsyncFC collectively support this distinction.  [8][[10]](https://github.com/SqueezeAILab/LLMCompiler)[12]

## Control-loop architectures

### Reactive loops are a strong default because observations remain inside the control loop

ReAct's foundational contribution was to interleave reasoning and environment actions rather than generate a complete answer or action sequence before observing the consequences. On ALFWorld and WebShop, the paper reported absolute improvements of **34 and 10 percentage points**, respectively, over prior imitation/reinforcement-learning approaches while using only one or two in-context examples.  [27] This is evidence for **closed-loop interaction**, not evidence that ReAct's exact textual “Thought/Action/Observation” syntax is uniquely optimal.

A reactive loop is particularly appropriate when tools are stochastic, stateful, or information-seeking: the output of one call genuinely determines what should be done next. Its main weakness is serialization. When three independent queries can be launched now, a strict observe-one-act-one-observe loop spends wall time waiting for information that did not depend on the preceding call. LLMCompiler and production multi-agent research systems specifically target that inefficiency.  [[10]](https://github.com/SqueezeAILab/LLMCompiler)[3]

### Plan-then-execute pays when dependencies can be discovered in advance

ReWOO separates a planner's reasoning from tool observations and lets a worker fill in intermediate variables. It reported a **5× improvement in token efficiency together with +4% accuracy on HotpotQA**, as well as improved robustness to tool failures.  [8][9] This is important because it shows that exposing every raw tool observation to the main reasoning stream is not inherently beneficial; unnecessary observation tokens can increase both cost and context pressure.

LLMCompiler pushes this further by compiling a plan into a dependency graph and dispatching ready tasks concurrently. Against sequential ReAct-style execution, it reported **up to 3.7× latency speedup, up to 6.7× cost savings, and up to roughly 9% accuracy improvement**, depending on benchmark.  [[10]](https://github.com/SqueezeAILab/LLMCompiler) Its open implementation is available as [SqueezeAILab/LLMCompiler](https://github.com/SqueezeAILab/LLMCompiler). The repository exposes benchmark commands and supports both hosted and vLLM-backed execution, making it one of the more straightforward planning/scheduling claims to reproduce.  [[35]](https://github.com/SqueezeAILab/LLMCompiler)

Those gains should not be generalized into “planner beats reactive.” A plan generated before enough evidence exists can create a stale dependency structure. The best production pattern is therefore often **plan enough to expose parallelism and critical dependencies, execute, then re-plan at observation boundaries** rather than either extreme of one-action-at-a-time reactivity or an immutable end-to-end plan. This synthesis is consistent with the benefits of ReWOO/LLMCompiler and the highly task-dependent architecture results in the multi-agent study.  [8][[10]](https://github.com/SqueezeAILab/LLMCompiler)[1]

### Hierarchy is an optimization for decomposability, not a universal sophistication upgrade

The Google-led 2026 “Science of Scaling Agent Systems” study is unusually useful because it compares single-agent, independent, decentralized, centralized, and hybrid configurations while controlling tools, prompts, and reasoning budget across a broad suite. Outcomes ranged from **+80.8% to −70.0% relative to the single-agent baseline**.  [36][1]

| Benchmark | Single agent | Independent | Decentralized | Centralized | Hybrid | What it says |
|---|---:|---:|---:|---:|---:|---|
| Finance-Agent | 0.349 | — | 0.609 (+74.5%) | **0.631 (+80.8%)** | 0.604 (+73.1%) | Breadth/decomposition rewards coordination.  [1] |
| Workbench | 0.629 | — | **0.664 (+5.6%)** | ~−1.2% rel. | ~−1.2% rel. | Peer coordination can help modestly.  [1] |
| BrowseComp-Plus | 0.318 | — | **0.347 (+9.2%)** | +0.2% rel. | — | Independent search directions are useful.  [1] |
| PlanCraft | **0.568** | 0.170 (−70.0%) | 0.332 (−41.5%) | 0.282 (−50.3%) | 0.346 (−39.1%) | Communication overhead damages tightly coupled planning.  [1] |
| SWE-bench Verified | **0.522** | 0.444 (−14.9%) | 0.494 (−5.4%) | 0.506 (−3.1%) | 0.511 (−2.1%) | Multi-agent decomposition did not improve coding here.  [1] |
| Terminal-Bench | 0.344 | 0.350 (+1.7%) | — | 0.278 (−19.2%) | — | Added hierarchy can be pure overhead.  [1] |

The same study quantifies why. Relative to a single agent averaging 7.2 turns, decentralized, centralized, and hybrid systems averaged 26.1, 27.7, and 44.3 turns; estimated coordination overhead rose to 263%, 285%, and 515%, respectively. Success per 1,000 tokens fell from 67.7 for the single agent to 23.9, 21.5, and 13.6. Crucially, **agent count itself was not statistically significant** (β=.040, p=.487), whereas tool diversity was positively associated with performance and interactions between architecture and workload characteristics were important.  [2]

This makes “how many agents?” the wrong first design question. The useful questions are: **How many independent information frontiers exist? How expensive is communication? Must workers share rapidly changing state? Can outputs be verified and merged independently?** Anthropic's production experience agrees: its multi-agent research architecture works best for breadth-first parallel research and is a poor fit for tasks with many inter-agent dependencies; the company reports the architecture uses roughly 15× the tokens of an ordinary chat.  [3]

### Async execution belongs below the semantic controller whenever possible

AsyncFC is notable because it treats asynchronous function execution as an **execution-layer transformation**, not a model capability: tool execution can overlap with model decoding and with other independent functions without changing the model itself. The primary paper reports reduced end-to-end time while preserving task accuracy, and specifically identifies sequential dependency chains and negligible tool latency as cases with little available speedup.  [11][12]

That suggests a practical separation of concerns: the controller should declare or infer dependencies; the runtime should schedule any dependency-safe actions concurrently. “Async agent” should not mean letting arbitrary actions race against each other. It should mean **semantics-preserving concurrency over a dependency graph**.

## Tool interfaces and execution surfaces

### The interface can be worth more than another round of reasoning

SWE-agent remains one of the clearest fixed-model harness experiments. With GPT-4 Turbo on SWE-bench Lite, the full interface resolved 18.0% versus 11.0% for a shell-only agent. Its full SWE-bench result was 12.47% over 2,294 instances, while the historical RAG baseline with the same model reached only 1.31%; the latter comparison changes both interaction mode and interface and is therefore less causally clean than the Lite ablations.  [33][[4]](https://github.com/SWE-agent/SWE-agent)

The ablations are more informative than the headline score:

| Interface decision, GPT-4 Turbo on SWE-bench Lite | Resolve rate | Change vs full 18.0% | Interpretation |
|---|---:|---:|---|
| Purpose-built editor + linting | **18.0%** | — | Baseline ACI.  [[4]](https://github.com/SWE-agent/SWE-agent) |
| Editor without linting | 15.0% | −3.0 pp | Immediate deterministic error rejection helps recovery.  [[4]](https://github.com/SWE-agent/SWE-agent)[[5]](https://github.com/SWE-agent/SWE-agent) |
| No editor primitive | 10.3% | −7.7 pp | Whole-file/shell editing creates substantial interaction friction.  [[4]](https://github.com/SWE-agent/SWE-agent) |
| Summarized search | **18.0%** | — | Concise result presentation.  [[4]](https://github.com/SWE-agent/SWE-agent) |
| Iterative one-result-at-a-time search | 12.0% | −6.0 pp | Extra interaction can be actively harmful.  [[4]](https://github.com/SWE-agent/SWE-agent)[[5]](https://github.com/SWE-agent/SWE-agent) |
| No special search | 15.7% | −2.3 pp | A bad specialized tool can be worse than none.  [[4]](https://github.com/SWE-agent/SWE-agent) |
| 100-line file window | **18.0%** | — | Useful locality.  [[4]](https://github.com/SWE-agent/SWE-agent) |
| 30-line window | 14.3% | −3.7 pp | Too little context.  [[4]](https://github.com/SWE-agent/SWE-agent) |
| Whole file | 12.7% | −5.3 pp | Too much context.  [[4]](https://github.com/SWE-agent/SWE-agent)[[5]](https://github.com/SWE-agent/SWE-agent) |
| Last five observations | **18.0%** | — | Bounded interaction history.  [[4]](https://github.com/SWE-agent/SWE-agent) |
| Full interaction history | 15.0% | −3.0 pp | More context was not better.  [[4]](https://github.com/SWE-agent/SWE-agent) |

This is a remarkably general design lesson: **tool surfaces should expose the smallest semantically complete operation, return immediate machine-checkable feedback, and bound result volume**. Merely adding more commands can lower success if the model must page through them inefficiently.  [[5]](https://github.com/SWE-agent/SWE-agent)

The historical result should not be overread. mini-SWE-agent later showed that much stronger models can attain greater than 74% on SWE-bench Verified with a minimal linear-history, bash-centric harness.  [34] The defensible conclusion is therefore not “specialized tools beat shell,” but **interface friction is model-relative: weaker or less tool-native models benefit more from engineered affordances; stronger models can absorb more of the interface complexity themselves.**

### Executable code is a powerful tool-composition surface

CodeAct evaluates the action representation itself on M³ToolEval, an 82-task benchmark requiring multiple calls to multiple tools across domains such as browsing, finance, travel planning, science, and information processing. Across 17 tested models, code actions were most often the best-performing representation.  [6][[7]](https://github.com/xingyaoww/code-act)

For GPT-4-1106-preview:

| Action representation | Success | Mean turns |
|---|---:|---:|
| Executable CodeAct | **74.4%** | **5.5** |
| JSON | 52.4% | 7.6 |
| Text action syntax | 53.7% | 7.7 |

The **+20.7 percentage-point** improvement over the next-best success result came together with 2.1 fewer turns. The authors attribute the advantage to native control/data flow—loops, conditionals, composition—and automatic runtime feedback such as exceptions.  [6][37][[7]](https://github.com/xingyaoww/code-act) The implementation is available at [xingyaoww/code-act](https://github.com/xingyaoww/code-act).  [[38]](https://github.com/xingyaoww/code-act)

The tradeoff is equally important: executable code has a much larger authority surface than typed individual functions. A good design therefore combines CodeAct-like composition with **process isolation, capability-scoped credentials, filesystem/network policy, execution budgets, and explicit commit boundaries** rather than granting an unrestricted interpreter.

### Tool selection becomes a first-class problem when catalogs get large

ToolRet contains 7,600 retrieval tasks over approximately 43,000 tools. General information-retrieval systems performed surprisingly poorly on this specialized retrieval problem; the paper reports that improving retrievers on a 200,000-instance tool-retrieval training set subsequently improved GPT-3.5 and ToolLlama downstream pass rates by **10–20%**.  [28][39]

This means “put every tool schema in the system prompt” is not a scale strategy. There are two distinct failure modes:

1. retrieving the wrong tool means the downstream agent never gets a chance to act correctly; and
2. exposing too many semantically similar tools increases prompt size and selection competition. ToolRet directly demonstrates the first problem; newer skill-routing work studies the second.  [28]

A 2026 SkillRouter preprint reports 74.0% Hit@1 for a compact 1.2B retrieve/rerank pipeline, versus 68.0% for a much larger 16B base pipeline; its serving benchmark reports **5.8× lower median routing latency**, and downstream Claude Sonnet/Opus 4.6 agents gained an average **3.22 percentage points** from the routing improvement. The authors correctly caution that the main human-authored benchmark contains only 75 core queries and that downstream significance testing was not performed.  [40][41]

Thus **routing quality is functional; router implementation style is not**. Dense retrieval versus a trie, a skill index versus an API registry, or MCP discovery versus an internal catalog matters only insofar as it changes recall, false positives, context size, latency, or authority.

### Sandboxing is mainly about robustness and blast radius, not benign benchmark score

AgentDojo demonstrates why tool-using agents require a threat model that includes tool results themselves: its environment contains 97 realistic tasks and 629 prompt-injection security cases, and individual attack categories can be highly successful even when benign utility is already below 100%.  [[42]](https://github.com/ethz-spylab/agentdojo)

More recent least-privilege designs show large causal robustness effects. Prismata, which confines both what a web agent can observe and what actions content is permitted to influence, reports reducing average attack success from **85.5% to 0.7%**, while raising successful completion of the intended user task under attack from **4.5% to 23.0%**.  [30] Progent similarly reports reducing AgentDojo attack success from 39.9% to 1.0% using programmable privilege controls.  [43]

For ordinary non-adversarial task completion, the marginal success benefit of containerization by itself is **unspecified** in these studies. Its measured value appears in failure containment, reproducible state, security, and recovery rather than making the underlying reasoning model smarter.

## Context and memory management

### Treat context as different classes of state, not one ever-growing transcript

The evidence supports separating at least four classes:

**Pinned control state** consists of user requirements, permissions, invariants, acceptance tests, and irreversible-action rules. These should bypass lossy compaction because Governance Decay directly shows that disappearance from a summary predicts subsequent violations.  [13]

**Working state** consists of the small set of current observations, hypotheses, active plan nodes, and immediate tool outputs. SWE-agent's 18.0% result using a bounded recent-observation context versus 15.0% with full history is direct evidence that unbounded transcript retention can hurt task performance.  [[4]](https://github.com/SWE-agent/SWE-agent)

**Artifact state** consists of files, test outputs, database records, fetched documents, structured plans, and checkpoints. These are better retained by reference than repeatedly copied into the prompt; LLMCompiler's worker graph, external-memory systems, and sandbox recovery work all rely on this separation.  [[10]](https://github.com/SqueezeAILab/LLMCompiler)[26]

**Long-term semantic/episodic memory** contains selectively retrieved material from earlier episodes or distant parts of the current episode. The best storage and retrieval policy depends strongly on task, model, and history scale rather than admitting a single best substrate.  [17][19]

A robust state model therefore looks more like this:

```mermaid
flowchart LR
    A[Pinned invariants<br/>never summarized] --> P[Prompt assembly]
    W[Recent working set<br/>bounded] --> P
    R[Retrieved long-term memory<br/>query-selected] --> P
    F[Artifact references<br/>files / tests / DB rows] --> P

    T[Old transcript] --> C[Lossy compaction]
    C --> R

    P --> L[Agent decision]
    L --> X[Tool execution]
    X --> W
    X --> F
```

### Summarization is a lossy cache, not authoritative state

Governance Decay provides perhaps the clearest demonstration. Across seven models and 1,323 episodes, compaction changed constraint-violation rates from **0% before compaction to 30% afterward**, with some configurations reaching 59%. A constraint present in the compacted representation was never violated in the reported experiment; when omitted, the violation rate was 38%. Soft organizational policies decayed much more severely than hard safety-style norms.  [13]

The paper's “Constraint Pinning” intervention is particularly actionable: roughly **47 tokens of pinned constraints** outside the summary reduced observed violations back to **0%**, at under 0.5% token overhead for the production-scale context regime studied.  [13] This is strong evidence for an architectural rule: **summarize narrative; pin invariants.**

Summarization also has systems cost. Parallel Context Compaction studied context inputs from about 2K to 96K tokens and found that simply prompting models to be “concise” versus “detailed” did little to reliably control summary volume; input grew by approximately 48× while outputs grew only about 3×, with increasing instability at long context lengths.  [14][15] At a 16K compaction threshold, synchronous compaction accounted for 51.3% of end-to-end time for gpt-oss-20B and 62.4% for Llama-3.1-8B in the reported configurations; raising the threshold reduced that fraction, while parallel block processing delivered roughly 1.37–2.13× throughput improvements in selected experiments.  [15][16]

Therefore, a prompt instruction such as “keep the summary short and retain everything important” should not be treated as a hard storage policy. **Length bounds, pinned fields, retention classes, and overflow handling belong in code.**

### Memory retrieval can raise task success, but more structure is not monotonically better

The August 2026 “Harness the Memory” preprint evaluates eleven memory substrates across three model backbones and four benchmarks, measuring both quality and systems cost. On ALFWorld-unseen with Qwen3-32B-AWQ, no memory achieved 22.4% task success; a refinement-based memory reached **32.1%, a +9.7-point increase**. Other memories reached 21.6–29.9%, so merely having memory was not sufficient.  [17][18]

On BigCodeBench-Hard with the same 32B family, no memory scored 17.6% pass@1. A relatively simple flat memory reached **19.6%**, whereas one structural/graph substrate scored **16.2%**, below no-memory, while imposing roughly sixfold latency in the authors' comparative analysis. With weaker models, some of those same structural memories helped considerably, demonstrating a strong backbone interaction.  [18]

At larger history sizes, the study also found different substrates exhibit different scaling curves: refinement and structural approaches could improve conflict resolution as memory grew, but graph rebuild/entity-extraction costs became steep at hundreds of thousands of tokens.  [19] Because this is a very recent unaccepted preprint and its code was not yet publicly released at the version reviewed, these results are best treated as directional current evidence rather than settled benchmark consensus.  [17]

### Persistent state wins only after its ingest/retrieval overhead amortizes

“Total Recall at What Cost?” compares full transcript, a 10-turn rolling window, and several memory systems for conversations up to 400 turns. Its central finding is that **there is no globally best cost/accuracy memory regime**: observed memory-system accuracy spanned roughly 21–54% depending on backbone/configuration, and the point at which a memory system became cheaper than repeatedly sending full history ranged from immediate to beyond the 400-turn horizon.  [20][21]

At long histories, full transcript prompting could become **up to 12.7× more expensive than a memory system**, but under small-message workloads one tested memory system could itself cost up to **3.3× full history** because ingestion and retrieval dominated early. Mem0's break-even point ranged as late as hundreds of turns in tested settings, while other systems had different curves.  [21][22]

That leads to a practical default:

| State | Default retention | Why |
|---|---|---|
| User requirements, safety/authority rules, acceptance criteria | **Pinned for entire episode** | Compaction loss can directly create policy violations.  [13] |
| Last few causally relevant observations | **Keep verbatim** | Bounded history can outperform full history.  [[4]](https://github.com/SWE-agent/SWE-agent) |
| Large raw tool results | **Store externally; retain handle + digest** | Prevents prompt expansion while preserving retrievability. Compaction/tool-routing evidence supports bounded exposure.  [15][28] |
| Tests, files, database state, generated artifacts | **Persistent authoritative artifact state** | Needed for deterministic verification and environment recovery.  [25][26] |
| Completed reasoning narrative | **Discard or summarize** | Full history can hurt and costs accumulate.  [[4]](https://github.com/SWE-agent/SWE-agent)[21] |
| Failed hypotheses | **Distill the falsifying fact, not necessarily the full trace** | Rewind memory and Reflexion show value in carrying actionable failure information forward.  [25][[23]](https://github.com/noahshinn024/reflexion) |
| Cross-episode experience | **Persist only with retrieval/provenance policy** | Memory gains are task- and model-dependent.  [18] |

The exact database—SQLite, key-value store, vector DB, graph store—is **not independently shown by these studies to improve success**. What is measured is the behavior of retention, consolidation, indexing, retrieval, and conflict-resolution policies.

## Failure handling and recovery

### Retries are valuable only after distinguishing transient from semantic failure

A useful failure handler should first classify what changed between attempts.

A **transient infrastructure failure**—timeout, rate limit, unavailable tool—can often justify a bounded retry with backoff, provided the action is idempotent or the runtime can establish whether a side effect already committed. The agent literature does not yet contain a strong general benchmark giving one universal retry count; this remains mostly a systems-policy choice.

A **semantic failure**—wrong patch, failed test, wrong search strategy—needs new information. Reflexion gives controlled evidence for this. Its HumanEval result rose from GPT-4's 80.1% to **91.0% pass@1**, while ALFWorld improved by 22 absolute points across iterative episodes.  [[23]](https://github.com/noahshinn024/reflexion) Yet on the paper's difficult HumanEval-Rust ablation, base success was 0.60, self-reflection without generated tests fell to **0.52**, tests without reflection remained 0.60, and reflection plus tests reached **0.68**.  [[23]](https://github.com/noahshinn024/reflexion) Self-critique is therefore not itself a reliable error signal; **grounding the retry in an external verifier materially changes its value.**

The 2025 retrial study reinforces the cost side: under matched inference budgets, simpler repeat attempts often matched or exceeded more elaborate Tree-of-Thought/Reflexion-style strategies.  [24] That makes “retry with diversity and a deterministic verifier” a sensible baseline to beat before adding a separate critic agent.

### Deterministic guardrails are unusually high-value

The SWE-agent linting ablation is small but clean: rejecting edits that introduce major lint errors raised SWE-bench Lite resolution from **15.0% to 18.0%** with GPT-4 Turbo.  [[4]](https://github.com/SWE-agent/SWE-agent)[[5]](https://github.com/SWE-agent/SWE-agent) This is exactly the kind of failure handling that should live outside the LLM: fast, objective, cheap, and directly connected to the action being taken.

The same pattern generalizes to unit tests, schema validation, type checks, policy checks, precondition assertions, duplicate-operation detection, and completion criteria. Where a deterministic checker exists, it should normally precede an LLM critic because it is cheaper and less ambiguous. AgentRewind's benchmark similarly uses programmatic acceptance criteria rather than an LLM judge for its core task-success measurement.  [25]

### Restarting from zero throws away valuable correct state

AgentRewind directly compares four strategies under otherwise matched task environments and termination rules: continue from the bad state; restart with experiences; safety review; and rewind to a prior aligned checkpoint of both context and environment.  [25]

For GPT-5.4 on MettleBench:

| Strategy | Task success | Checklist progress |
|---|---:|---:|
| Continue | 62.2% ± 2.1 | 81.4% ± 1.0 |
| Restart with experiences | 78.0% ± 2.4 | 88.8% ± 1.2 |
| Safety review | 34.1% ± 1.2 | 54.4% ± 2.1 |
| **AgentRewind** | **87.8% ± 1.2** | **94.3% ± 0.5** |

AgentRewind's advantage over Continue was statistically significant under task-level paired testing (Holm-corrected p<0.0001 for GPT-5.4 success), and the effect reproduced across three harnesses: **+25.6 pp with mini-SWE-agent, +23.2 pp with FnCallAgent, and +15.8 pp with CodeAgent**.  [25]

The component ablation is more important than the headline:

| Recovery configuration | Success |
|---|---:|
| Full context + environment rewind + rewind memory | **87.8%** |
| Without environment rewind | 43.9% |
| Without context rewind | 65.9% |
| Without rewind memory | 51.2% |

Restoring only the conversation is thus insufficient if a failed suffix deleted files, modified configuration, or otherwise changed the world.  [25]

AgentRewind also tested Terminal-Bench 2.0: Continue scored 78.7%, restart-with-experience 70.8%, and AgentRewind **83.1%**.  [44] The smaller cross-benchmark gain is a useful reminder that recovery benefit grows with the amount of correct-but-vulnerable intermediate state a task accumulates.

### Checkpointing does not have to be expensive

Crab approaches the same problem at the sandbox/operating-system level. On Terminal-Bench, chat-only restoration recovered correctly in only **8–13%** of cases; chat plus filesystem restoration reached 28–42%; Crab, which also captures relevant process state, reached **100% recovery correctness**.  [26]

Its systems results suggest checkpointing need not become the new bottleneck. Despite one injected crash per task, execution remained within **1.9% of the no-fault runtime**. By contrast, starting over added up to 1.67× runtime on SWE-bench, and naïvely taking full checkpoints on every turn caused up to 3.78× slowdown under dense co-location. Crab could classify up to 87% of turns as needing no checkpoint and overlap much of the remaining work with model-wait time.  [26] Exposing restore as an agent-facing tool reduced wall-clock recovery time by up to 29% and recovery-token consumption by 36% in the reported case studies.  [26]

The architectural lesson is to checkpoint on **semantic state-change boundaries**, not necessarily every model turn.

### Rollback has a hard boundary at irreversible external effects

AgentRewind explicitly restores only its controlled workspace; network requests, external-service calls, and state outside the recovery boundary cannot be undone.  [45] Transaction-oriented agent work makes the same point: restoring agent memory does not unsend an email, reverse an already accepted booking, or reliably compensate every external API mutation.  [46]

For irreversible or costly side effects, the right primitive is therefore not merely “rollback.” The harness needs a **commit protocol**: keep exploration/speculation inside a sandbox or staged state; validate the intended effect; request human authorization where required; then externalize the side effect once. This is a much stronger reliability property than hoping a later agent step can repair the world.

### Human-in-the-loop belongs at information and authority boundaries

Workspace-Bench contains 388 tasks across realistic workspaces with more than 20,000 files and 7,000-plus evaluation rubrics. Its reported human-agent collaboration score is **80.7%**, significantly above fully autonomous configurations; autonomous performance also declines from 57.6% on easy tasks to 40.5% on hard tasks.  [31]

The relevant lesson is not “put a human in every loop.” Human intervention has greatest value when the missing variable cannot be recovered from tools—ambiguous intent, preference, authorization, business judgment—or when an effect is costly to reverse. Recent ambiguity-specific coding benchmarks similarly find that agents often fail to recognize underspecification unless interaction is deliberately supported.  [47]

Thus HITL should usually be **exception-triggered**, with escalation conditions such as:

* competing high-confidence interpretations of the goal;
* action outside previously granted authority;
* irreversible/high-cost side effect;
* repeated verifier failure with no progress; or
* low-confidence recovery after state corruption.

The exact UI—approval button, chat turn, ticket, or workflow task—is stylistic. The **decision boundary and information supplied to the human** are functional.

### Observability matters when it closes the loop; tracing UI by itself has no demonstrated success effect

A 2026 study of 2,823 committed episodes across several agent frameworks tested lightweight telemetry-based failure monitors. Adding a content-grounding telemetry channel raised pooled content-corruption detection from **0.28 to 0.59**, while a deterministic completion check caught **7/7 silent aborts**; the monitor authors report microsecond-scale step overhead.  [32] The same work also found weak transfer to some organic fabrication failures, which is useful negative evidence: generic anomaly detection is not a substitute for domain validation.  [32]

This separates two notions often called “observability”:

**Functional observability** means recording enough structured state to detect loops, repeated tool errors, unusual action sequences, missing completion conditions, token/cost exhaustion, state divergence, or policy violations—and connecting those signals to stop/retry/rewind/escalate actions. This has measurable value.  [32][25]

**Presentation observability** means which tracing dashboard, span visualization, vendor UI, or log viewer displays those events. I found no controlled primary-source evidence in the reviewed literature showing that one such presentation layer directly improves autonomous task success when the underlying signals and recovery policy are held constant. It is primarily an engineering productivity choice.

## Practical recommendations and evidence gaps

### A production default that matches the evidence

For a new general-purpose agent harness, the evidence favors a **single-agent, observation-driven controller as the default**, not a multi-agent hierarchy. Add planning only far enough to identify dependencies and opportunities for concurrency. Elevate to parallel workers when the task has genuinely independent branches—research queries, independent data sources, separable subtasks—not merely because workers are available. The controlled architecture study and Anthropic's research system both support this decomposition criterion.  [1][3]

Use an **execution DAG underneath the controller**. Tool calls whose arguments depend on earlier results stay sequential; independent slow calls run concurrently. This captures the latency benefits demonstrated by LLMCompiler/AsyncFC without forcing the model to reason asynchronously or tolerate races.  [[10]](https://github.com/SqueezeAILab/LLMCompiler)[12]

Expose a **small, composable tool surface**. Tools should have typed inputs, bounded outputs, immediate deterministic errors, and operations at the semantic granularity the task actually needs. Search should return a compact useful batch rather than force dozens of paging turns; edit APIs should validate syntax and expose local context; code execution is valuable when tool composition needs loops or dataflow.  [[5]](https://github.com/SWE-agent/SWE-agent)[[7]](https://github.com/xingyaoww/code-act) For catalogs larger than can reasonably fit in context, retrieve/rerank tools dynamically, with a fallback path when routing confidence is low.  [28][40]

Run potentially mutating code in a **capability-scoped sandbox**. Give workers only the credentials, filesystem paths, network destinations, and side-effect permissions required for the current task. Security benchmarks show that deterministic privilege boundaries can reduce prompt-injection attack success by one to two orders of magnitude; they also make state recovery tractable.  [30][43][26]

Partition context into **pinned invariants + recent working state + external artifacts + retrieved memory**. Never make a lossy summary the sole copy of user constraints, permissions, acceptance criteria, or irreversible-action rules.  [13] Keep raw large tool results outside the main transcript and expose handles/excerpts. Treat persistent semantic memory as an optimization to validate experimentally rather than a mandatory architectural layer, because measured success and cost vary strongly by workload and model.  [18][21]

Build a **typed failure ladder** rather than a generic `try/except → retry`:

```mermaid
flowchart TD
    E[Failure / anomaly] --> T{Failure class}

    T -->|Transient transport| R1[Bounded idempotent retry<br/>backoff / alternate endpoint]
    T -->|Deterministic validation| R2[Return exact verifier feedback<br/>retry changed action]
    T -->|Wrong semantic path| R3[Re-plan / diversify search<br/>retain falsifying evidence]
    T -->|Corrupted controlled state| R4[Rewind context + environment<br/>from checkpoint]
    T -->|Ambiguous intent| H[Ask human]
    T -->|Irreversible external effect| C[Stop before commit<br/>authorize / validate]
    T -->|Repeated no-progress| X[Escalate or terminate]

    R1 --> V[Re-verify]
    R2 --> V
    R3 --> V
    R4 --> V
```

This ordering follows the observed evidence: deterministic feedback improves edits; reflection works best with grounded feedback; stateful rewind beats continuing/restarting on long-horizon tasks; and rollback cannot repair effects outside its state boundary.  [[5]](https://github.com/SWE-agent/SWE-agent)[[23]](https://github.com/noahshinn024/reflexion)[25][45]

### Design choices that should be benchmarked locally rather than standardized dogmatically

**Planner versus reactive controller:** benchmark on representative dependency structures. The public evidence supports both depending on task.  [8][1]

**Multi-agent topology:** benchmark task success *and* tokens/turns. More agents are not independently associated with better performance, and communication can dominate.  [2]

**Memory substrate:** benchmark no-memory, rolling-window, simple flat retrieval, and the proposed sophisticated memory. Do not compare only competing memory products; the no-memory and full-history controls are essential because some memories hurt performance or cost.  [18][21]

**Compaction threshold:** measure success after multiple folds as well as latency. More aggressive compaction lowers prompt growth but can dominate wall time and erase constraints.  [15][13]

**Tool action representation:** code is particularly attractive for multi-tool composition, while small typed functions provide a narrower authority surface. The CodeAct advantage is strong on its complex benchmark, but later minimal coding harnesses demonstrate that model strength can reduce the need for specialized action APIs.  [[7]](https://github.com/xingyaoww/code-act)[34]

### What remains undermeasured

There is still **no mature, cross-benchmark causal literature for agent harnesses comparable to systems research on databases or distributed runtimes**. Many papers introduce both an architecture and a new benchmark, making absolute scores difficult to interpret. Some 2026 results in this report—AgentRewind, Crab, the broad memory-substrate comparison, SkillRouter, and Governance Decay—are recent preprints rather than multiply replicated findings.  [45][26][17][41][13]

Latency reporting is especially inconsistent. Some papers report model tokens, others tool calls, others wall-clock time, and many report none. Tool latency can dominate web/API agents while inference dominates other workloads, so token count is not a reliable latency proxy. LLMCompiler, parallel-compaction work, Crab, and the recent memory-systems studies are exceptions that explicitly report systems metrics.  [[10]](https://github.com/SqueezeAILab/LLMCompiler)[16][26][21]

**Persistent memory durability across software versions, conflicting memories, deletion/privacy, and multi-agent concurrent writes are substantially less evaluated than retrieval accuracy.** The memory-substrate studies expose scalability and conflict-resolution differences, but a general transactional memory benchmark for deployed agents is still missing.  [19]

**Rollback of external SaaS/API effects remains unsolved in the general case.** Current strong recovery results operate inside controlled sandboxes/workspaces; external commits require authorization, transactional APIs, compensating actions, or application-specific semantics.  [45][46]

**Human-in-the-loop cost is rarely normalized against autonomous inference cost and user waiting time.** Workspace-Bench establishes a substantial capability advantage for collaboration, but it does not establish a universal economically optimal escalation rate.  [31]

**Observability presentation is essentially unevaluated as an agent-performance intervention.** Monitoring algorithms can be benchmarked, but the choice of trace vendor or UI remains a developer-operations decision unless it feeds a recovery controller.  [32]

**Protocol standardization is also undermeasured.** MCP benchmarks now test agents navigating hundreds of tools and servers, but those results mostly quantify discovery, retrieval, and composition capability—not a controlled comparison of MCP versus semantically identical native function calling.  [29] The evidence supports standardized protocols for interoperability; it does not yet support claiming that the protocol envelope alone makes an agent more capable.

### What I would measure in an internal harness experiment

The minimal useful factorial evaluation would hold model, prompts, benchmark cases, and inference budget fixed and vary one harness dimension at a time. Each episode should record deterministic final success where possible, partial-progress criteria, wall time, model input/output tokens, tool calls, tool wall time, retry count, compaction events, peak context size, human interventions, and recovery events. The importance of holding the harness constant is illustrated by SWE-agent's large ACI ablations and by the multi-agent study's observation that architecture can swing results from strongly positive to strongly negative.  [[4]](https://github.com/SWE-agent/SWE-agent)[1]

For reliability, inject controlled failures: timeout an idempotent tool, return malformed output, corrupt a file, make a tool unavailable, insert an adversarial tool result, and introduce an ambiguous user requirement. Measure not only whether the final task succeeds but whether the harness recognizes the failure, how much correct work it preserves, and how many extra tokens/minutes it spends recovering. Crab, AgentRewind, AgentDojo, and ambiguity-focused evaluations demonstrate why ordinary clean-run benchmarks miss much of the harness's actual value.  [26][25][[42]](https://github.com/ethz-spylab/agentdojo)[47]

## Timeline and reproducibility

The field has moved from “prompt an LLM to reason and call one tool at a time” toward increasingly explicit execution runtimes: DAG scheduling, memory policies, privilege boundaries, checkpoint/restore, and failure monitors. The timeline below emphasizes implementations that materially changed the non-model harness.

| Period | Paper / implementation | Non-model contribution | Representative evidence / code |
|---|---|---|---|
| 2022–2023 | **ReAct** | Closed-loop interleaving of reasoning, action, and observation | +34 pp ALFWorld and +10 pp WebShop versus prior approaches.  [27] |
| 2023 | **Reflexion** | Episodic failure feedback and iterative retry | HumanEval 80.1%→91.0%; feedback ablations show external tests matter. [Code: Reflexion](https://github.com/noahshinn024/reflexion).  [[23]](https://github.com/noahshinn024/reflexion) |
| 2023 | **ReWOO** | Separate planner and worker; avoid sending every tool observation through main reasoning | 5× token efficiency and +4% HotpotQA accuracy.  [8] |
| 2024 | **CodeAct** | Executable code as compositional action surface | Up to +20.7 pp and fewer turns. [Code: code-act](https://github.com/xingyaoww/code-act).  [[7]](https://github.com/xingyaoww/code-act)[[38]](https://github.com/xingyaoww/code-act) |
| 2024 | **SWE-agent** | Purpose-built agent-computer interface for search/edit/navigation | 18.0% vs 11.0% shell-only on SWE-bench Lite; extensive interface ablations. [Code: SWE-agent](https://github.com/SWE-agent/SWE-agent).  [[4]](https://github.com/SWE-agent/SWE-agent)[[5]](https://github.com/SWE-agent/SWE-agent) |
| 2024 | **LLMCompiler** | Planner + dependency graph + concurrent executor | Up to 3.7× latency, 6.7× cost, ~9% accuracy improvement. [Code: LLMCompiler](https://github.com/SqueezeAILab/LLMCompiler).  [[10]](https://github.com/SqueezeAILab/LLMCompiler)[[35]](https://github.com/SqueezeAILab/LLMCompiler) |
| 2024 | **LATS** | Tree-search control loop combining action, value estimation, and reflection | Reported 92.7% pass@1 on HumanEval with GPT-4 and 75.9 WebShop score with GPT-3.5; useful but more compute-intensive than flat loops. [Code: LanguageAgentTreeSearch](https://github.com/lapisrocks/LanguageAgentTreeSearch).  [[48]](https://github.com/lapisrocks/LanguageAgentTreeSearch) |
| 2024 | **AgentDojo** | Stateful security benchmark for tool agents and prompt-injection defenses | 97 tasks, 629 security cases. [Code: agentdojo](https://github.com/ethz-spylab/agentdojo).  [[42]](https://github.com/ethz-spylab/agentdojo) |
| 2025 | **ToolRet** | Large-scale tool-routing benchmark and retrieval training | 7.6K tasks / 43K tools; tuned retrieval improves downstream pass rate 10–20%.  [28] |
| 2025 | **Anthropic Research multi-agent system** | Production orchestrator/parallel-worker architecture | +90.2% internal research eval; ~15× chat token usage. Vendor/internal benchmark.  [3] |
| 2025–2026 | **mini-SWE-agent** | Minimal bash-centric harness, useful counterexample to scaffold complexity | >74% SWE-bench Verified with a very small scaffold; not a same-model ACI ablation.  [34] |
| 2026 | **Science of Scaling Agent Systems** | Controlled comparison of single, independent, decentralized, centralized, hybrid agent systems | Architecture effects +80.8% to −70%; agent count alone nonsignificant.  [36][2] |
| 2026 | **AsyncFC** | Model-independent asynchronous function-execution layer | Lower end-to-end time while preserving accuracy; gains depend on available parallelism.  [11][12] |
| 2026 | **Governance Decay** | Measures compaction-induced loss of active constraints; constraint pinning | 0%→30% violations after compaction; pinning restores 0% at <0.5% overhead.  [13] |
| 2026 | **Parallel Context Compaction** | Treats context folding as a systems bottleneck; parallel block summarization | Synchronous folds can consume >50% E2E time; selected throughput gains 1.37–2.13×.  [14][16] |
| 2026 | **Crab** | Semantics-aware sandbox checkpoint/restore | 100% recovery correctness, ≤1.9% no-fault runtime overhead in tested fault regime.  [26] |
| 2026 | **AgentRewind** | Joint context/environment rewind plus failure memory | +15.8 to +25.6 pp across three harnesses on MettleBench.  [25] |
| 2026 | **Prismata / privilege-constrained agents** | Contextual least privilege over observation and action surfaces | Attack success 85.5%→0.7%, utility under attack 4.5%→23.0%.  [30] |
| 2026 | **Harness the Memory** | Broad comparison of eleven agent-memory substrates | Memory ranges from meaningful success gains to regressions and major latency penalties.  [17][18] |

The progression suggests a broader architectural convergence. The earliest work concentrated intelligence inside the model's iterative reasoning loop; the strongest recent harnesses increasingly move **scheduling, validation, authority, retention, state recovery, and telemetry into deterministic runtime components**. That shift is supported by independent evidence: planner/DAG scheduling cuts latency and cost; deterministic edit validation improves SWE success; tool retrieval raises downstream pass rate; pinned constraints prevent compaction failures; environment-aware rollback dramatically improves recovery; and least-privilege enforcement sharply improves adversarial robustness.  [[10]](https://github.com/SqueezeAILab/LLMCompiler)[[5]](https://github.com/SWE-agent/SWE-agent)[28][13][25][30]

The strongest practical principle is therefore not “use the most sophisticated agent framework.” It is: **put each responsibility in the component that can implement it most reliably.** Let the model handle ambiguous semantic judgment and local adaptation; let the harness handle dependency scheduling, exact validation, permissions, bounded context, artifact persistence, retries, checkpoints, and side-effect commits. The public evidence increasingly shows that those boundaries—not framework branding or architectural ornament—are what measurably determine whether long-running agents finish their tasks.  [2][[4]](https://github.com/SWE-agent/SWE-agent)[13][25]
---

## Sources

48 citations. 9 name a source you can open; 39 point to an internal reference that did not survive the export.

1. *No link given.*
2. *No link given.*
3. *No link given.*
4. [https://github.com/SWE-agent/SWE-agent](https://github.com/SWE-agent/SWE-agent)
5. [https://github.com/SWE-agent/SWE-agent](https://github.com/SWE-agent/SWE-agent)
6. *No link given.*
7. [https://github.com/xingyaoww/code-act](https://github.com/xingyaoww/code-act)
8. *No link given.*
9. *No link given.*
10. [https://github.com/SqueezeAILab/LLMCompiler](https://github.com/SqueezeAILab/LLMCompiler)
11. *No link given.*
12. *No link given.*
13. *No link given.*
14. *No link given.*
15. *No link given.*
16. *No link given.*
17. *No link given.*
18. *No link given.*
19. *No link given.*
20. *No link given.*
21. *No link given.*
22. *No link given.*
23. [https://github.com/noahshinn024/reflexion](https://github.com/noahshinn024/reflexion)
24. *No link given.*
25. *No link given.*
26. *No link given.*
27. *No link given.*
28. *No link given.*
29. *No link given.*
30. *No link given.*
31. *No link given.*
32. *No link given.*
33. *No link given.*
34. *No link given.*
35. [https://github.com/SqueezeAILab/LLMCompiler](https://github.com/SqueezeAILab/LLMCompiler)
36. *No link given.*
37. *No link given.*
38. [https://github.com/xingyaoww/code-act](https://github.com/xingyaoww/code-act)
39. *No link given.*
40. *No link given.*
41. *No link given.*
42. [https://github.com/ethz-spylab/agentdojo](https://github.com/ethz-spylab/agentdojo)
43. *No link given.*
44. *No link given.*
45. *No link given.*
46. *No link given.*
47. *No link given.*
48. [https://github.com/lapisrocks/LanguageAgentTreeSearch](https://github.com/lapisrocks/LanguageAgentTreeSearch)