The Agent Harness: State of the Art (2024–2026)

The organizing equation of the field: Agent = Model + Harness.

An LLM by itself is a stateless function. It doesn't remember, it doesn't execute code, it doesn't touch the network, it doesn't persist work. Everything that turns that function into something that does things over time — the loop, the tools, the memory, the sandbox, the guardrails, the recovery logic — is the harness.

The central thesis of 2024–2026 — harness engineering as a named discipline — is that the model is the smallest part of the system, and the harness (not the model) is increasingly where performance and reliability are won or lost.

The quantitative case

The honest matiz: the effect is regime-dependent. For long-horizon, tool-intensive, multi-step tasks the harness dominates; for short tasks with certain model families it's within noise (Scale AI, METR). And the harness leans on a platform layer (durable execution, governance, cost, data integration) that the discourse still underestimates.

What a harness actually is

An agent harness is the deterministic software infrastructure that wraps a non-deterministic LLM so it acts reliably over time. Synthesizing LangChain's "Anatomy of an Agent Harness", the Claude Code Agent SDK docs and Anthropic's engineering posts, a complete harness has these components:

(a) The main loop

The heart of every harness is a while-loop alternating model reasoning and tool execution:

1. Receive prompt — the model gets prompt + system prompt + tool definitions + conversation history.
2. Evaluate and respond — the model emits text and/or tool-call requests.
3. Execute tools — the harness runs each tool and collects results.
4. Repeat — results go back to the model as the next turn.
5. Finish — the loop ends when the model responds without tool calls.

Each full cycle is a turn. The loop is bounded by max_turns and/or max_budget_usd. The academic origin is ReAct (Yao et al., ICLR 2023): the model interleaves reasoning and action, with observations informing the next thought.

flowchart LR
    P[Receive prompt<br/>system + tools + history] --> E[Evaluate and respond]
    E -->|text only| R[Return result<br/>loop ends]
    E -->|tool calls| T[Execute tools<br/>collect results]
    T --> O[Append results<br/>as next turn]
    O -->|repeat| E
    T -. read-only .-> PAR[(in parallel)]
    T -. mutating .-> SEQ[(sequentially)]

(b) Tools / function calling

A tool registry (JSON-schema descriptions), an executor, and a result formatter. The key insight from LangChain: instead of pre-designing a tool per possible action, give the agent a general-purpose tool (bash / code interpreter) so it can write its own tools on the fly. MCP solved connectivity (a common interface for external tools/servers) — not coordination, access control, or sandboxed execution. Read-only tools run in parallel; mutating tools run sequentially.

(c) Context management — the hard problem

The model only reasons over what's in its window, and performance degrades as it fills (context rot). Mechanisms:

• Compaction — when the window nears its limit, the harness summarizes/offloads old context.
• Tool-call offloading — keep head + tail tokens of large tool outputs in context; the full output goes to the filesystem.
• Progressive disclosure / Skills — don't pre-load every tool/MCP server into context (it degrades performance before it starts). Load cheap front-matter, expand on demand.
• Context scheduling — the Claude Code leak shows 3 memory layers: an always-loaded MEMORY.md index (pointers <150 chars), on-demand project context, and a session cache with auto-compression. Lesson: planning what to load and when matters as much as raw window size.

(d) Memory

Two levels: agent-side (working memory assembled for the current turn) and harness-side (persistent stores accessed via index). The filesystem is the fundamental memory primitive: agents get a workspace to read data, offload intermediate outputs, and persist state that survives the session. Git adds versioning for rollback and experiment branching. AGENTS.md / CLAUDE.md files are a continuous-learning mechanism — injected at session start and edited by the agent so future sessions inherit the knowledge.

(e) Planning

The model decomposes a goal into steps; the harness supports it with plan files, injected reminders to consult the plan, and an effort/reasoning knob (Claude Code: low/medium/high/xhigh/max) that trades token cost and latency for reasoning depth.

(f) State and persistence

Checkpoints and intermediate outputs determine whether a fallen agent resumes from the last successful step or starts over. For long-running agents this is where durable execution (the platform layer) starts to matter.

(g) Error handling / auto-verification

Because the model is non-deterministic, the harness must handle hallucinated tool calls, malformed arguments, failed tools, and false completion (the model declares "done" when it isn't). Mechanisms: hooks (intercept/modify/block tool calls; run a test suite and feed errors back), auto-verification loops (the agent writes code, runs tests, inspects logs/screenshots and fixes errors), and retry with backoff plus budget/turn limits. Anthropic found that explicitly prompting Claude to test end-to-end with browser automation (Puppeteer MCP) "drastically improved" the rate of features marked done-but-broken.

(h) Sandboxing

Agent-generated code must run somewhere safe and scalable: isolated execution, command allow-lists, network isolation, on-demand create/teardown. Good sandboxes ship pre-installed runtimes, git, test CLIs and browsers so the agent can verify its own work.

(i) Guardrails / permissions / governance

What the agent may do, under whose authority, and what audit trail it produces. Claude Code's model: allowed_tools (auto-approve), disallowed_tools (block), permission_mode (how much human supervision), evaluated in fixed order; individual calls can be scoped (Bash(npm *)). Enterprise concerns: identity propagation, prompt-injection sanitization, and avoiding a shared system identity. This is where most enterprise projects stall.

(j) Observability and evals

Observability records what the agent did and why — the reasoning trace, which memory fragments and tool calls drove each decision. It's different from app monitoring: the question is "was the decision correct?", not "did the system respond?". Evals run out-of-band over that observability to decide if an agent is ready for production. The hard part: deterministic success criteria don't map cleanly to non-deterministic systems.

Why the harness is the differentiator

The model is wrapped as: system prompt + tool definitions + history → model → parse output (text vs. tool calls) → if tool calls, execute and append results → repeat. The model is treated as an opaque, non-deterministic black box inside a deterministic control loop. The harness owns everything outside the API call.

• Vercel removed ~80% of its tools and success jumped 80% → 100% with the same model.
• LangChain moved its coding agent from the bottom of Terminal Bench 2.0 to top 5 (52.8% → 66.5%) by changing only the harness.
• The Claude Code leak made it concrete: ~512,000 lines of TS / ~1,900 files, of which actual model interaction is a small fraction. The rest is harness.

The deeper reason the harness exists: the wrapped component is non-deterministic. A harness is designed to (1) raise the probability of getting it right the first time, and (2) provide feedback loops that self-correct as many issues as possible before a human is involved. The goal isn't to eliminate human input — it's to direct it to where it matters most.

Co-evolution model ↔ harness

Current products (Claude Code, Codex) are post-trained with models and harnesses in the loop — the model learns to be good at exactly the actions the harness exposes (fs ops, bash, planning, subagents). That creates a feedback loop but also overfitting: e.g., Codex's apply_patch logic means a model trained on one patch format underperforms if you change the patch method. And it means the best harness for your task isn't necessarily the one your model was post-trained with.

Key design patterns

• ReAct (Yao et al., 2023): interleave reasoning traces and actions in a loop. The default loop of most harnesses.
• Plan-and-Execute: generate a full plan first, then execute step by step, re-planning as needed. Better for long-horizon tasks than pure ReAct, which can be myopic.
• Reflection / Reflexion: after acting, the agent critiques its own output (self-evaluation or "LLM-as-judge") and iterates. Watch for self-evaluation bias — agents grade their own work too generously.
• Multi-agent: decompose into specialized agents (planner, coder, tester, reviewer) with isolated contexts. The Claude Code leak shows 3 models: Fork (byte-identical context copy, for parallel reads), Teammate (file-based mailbox, independent tasks), Worktree (own git branch, isolated writes).
• The "Ralph Loop": a hook that intercepts the model's attempt to exit and re-injects the original prompt in a clean context, forcing continuous work toward a completion goal. Works because the filesystem lets each fresh context read the previous state.
• Initializer + worker split (Anthropic): a first session with a setup prompt prepares the environment (init script, progress file, feature list, git commit); later sessions make incremental progress and leave clean state. Directly attacks the two failure modes of "one-shotting the whole app" and "declaring victory prematurely".
• Context engineering (popularized ~2025): governing what information the model sees at any given moment — write, select, compress, isolate.

The discipline's hierarchy: prompt engineering optimizes a single turn; context engineering governs the window; harness engineering designs the full operating environment for hours-long autonomous execution — and contains the other two as parts.

The state of the art, 2025–2026

SWE-bench Verified — top of the leaderboard

What tops the board: simple but well-engineered harnesses (mini-SWE-agent's 100-line bash loop; live-SWE-agent's runtime self-evolution) on frontier models — not elaborate multi-agent stacks. The mini-SWE-agent insight is the field's punchline: minimal harness + strong model ≈ maximalist harness + weak model.

SOTA harness techniques

1. Context engineering is the central discipline. Claude Code's 5-layer compaction pipeline; OpenAI's compaction + "retained reasoning" (the ARC-AGI-3 13.3%→38.3% case). Anthropic's long-running pattern: feature-list JSON + progress file + git history as cross-session memory (filesystem-as-memory beats vector stores for coding).
2. Incremental progress + clean-state handoffs — one feature per session, git commits as checkpoints (enables rollback), session-end summaries.
3. Sub-agents / parallelism — isolated-context subagents returning summaries; dynamic workflows that spawn many subagents.
4. Sandboxed execution — Docker/podman/bubblewrap/contree; Codex's sandbox + approval policies; OpenHands' Docker runtime.
5. Auto-verification with real tools — explicit end-to-end testing with browser automation kills "feature marked done but broken".
6. Token efficiency — prompt caching, minimal harnesses, cost-per-instance as a leaderboard metric (MiniMax M2.5 at $36.6 total vs. $377 for Opus).
7. ACI design (SWE-agent's insight): the tool interface (bash-only vs. rich tools) is a first-class design variable.
8. Auto-evolution / RL — live-SWE-agent (runtime learning); Harbor for RL rollouts.

Protocols and standards

• MCP (Anthropic, Nov 2024) — the de-facto standard for agent↔tool/data connectivity. Revisions added structured output, OAuth 2.1 (MCP servers as Resource Servers), elicitation, then deprecation lifecycle and per-request version negotiation. Governance moved to the Agentic AI Foundation.
• A2A (Google, Apr 2025, 50+ partners; donated to the Linux Foundation in June 2025) — v1.0 is the first production-stable release: multi-protocol bindings, version negotiation, Agent Cards for discovery, streaming/async, multi-tenancy, enterprise auth. Positioned as complementary to MCP (A2A = agent↔agent; MCP = agent↔tool).
• Agentic AI Foundation (AAIF) — Linux Foundation, Dec 2025, with MCP and A2A as founding projects (founders include Anthropic, Google, Microsoft, OpenAI, AWS). The biggest interoperability milestone of the period.

Benchmarks at a glance

The evaluation trend: from peak accuracy to reliability (min–max confidence intervals across runs), cost-per-task, and time-horizon.

Open challenges (documented)

1. Error accumulation over long horizons — the central open problem. Per-step success is high, but multi-step reliability collapses: if each step succeeds with probability $p$, the probability that an $n$-step task succeeds is $P(n) = \prod_{i=1}^{n} p_i \leq p^{n}$, which decays exponentially even for large $p$ — at $p = 0.95$, a 50-step task succeeds only $\approx 8\%$ of the time. That is why METR measures time-horizons at 50% success rather than raw capability: even at ~5 hours of 50%-horizon, "days/weeks of work" is out of reach.
2. Cross-context-window amnesia — Anthropic's two documented failure modes: (a) one-shotting until the context runs out, leaving semi-built undocumented state; (b) later sessions seeing partial progress and declaring victory early.
3. Premature completion / unverified "done" — marking features passing without end-to-end tests; needs explicit verification tooling.
4. Context limits and compaction loss — compaction "doesn't always pass perfectly clear instructions to the next agent"; 1M-token windows help but don't remove the need for structured state.
5. Benchmark saturation and gaming — SWE-bench Verified approaching ~80–95% for frontier models; the team now publishes cheating detection; new branches (Multilingual, Multimodal, ProgramBench, SWE-bench-Live, OSWorld 2.0) stay ahead.
6. Reliability variance — HAL explicitly paused adding models to measure run-to-run reliability instead of peak accuracy.
7. Computer use still far behind — OSWorld: humans 72% vs. best ~12–38%; the market pivoted from "standalone browser agent" to agents embedded in workflow products.
8. Security/safety — prompt injection via fetched web content, tool abuse, permission scoping; harnesses respond with permission modes + ML classifiers, approval gates, sandboxing.
9. Single- vs. multi-agent — Anthropic leaves it explicitly open; no consensus yet.
10. Evaluation difficulty — no benchmark captures open-ended, multi-day, multi-system work.

A rigorous mental model: the cybernetic governor

Birgitta Böckeler (Thoughtworks, April 2026) models the harness as a cybernetic governor with 2 control types × 2 execution types:

• Guides (feedforward) — steer the agent before it acts (raise first-try success): AGENTS.md, Skills, code-mods, bootstrap scripts.
• Sensors (feedback) — observe after it acts and enable self-correction: linters, type checkers, structural tests, "LLM-as-judge" review.

Each can be computational (deterministic, CPU, fast, reliable — tests/linters) or inferential (semantic, GPU, slower, non-deterministic — LLM-as-judge). You need both: feedback alone → the agent repeats the same errors; feedforward alone → you encode rules but never learn whether they worked. The steering loop: the human iterates on the harness; every time an issue repeats, improve the guides/sensors to make it less likely.

Ashby's Law of Requisite Variety gives the formal backbone: a regulator must have at least as much variety as the system it governs, $V_{regulator} \geq V_{system}$. A committed harness topology reduces the variety the agent can produce — which is precisely what makes a comprehensive harness reachable at all.

The open "elephant in the room": the behaviour harness — how to verify that the app functionally does what's required. Maintainability and architecture fitness have tooling; behaviour verification doesn't yet.

Conclusion

An agent harness is the complete deterministic system wrapping a non-deterministic LLM to make it act reliably over time: loop (ReAct), tools (general-purpose bash/code + MCP), context management (compaction, offloading, progressive disclosure), memory (filesystem + git + AGENTS.md as durable/continuous state), planning, state/checkpointing, errors + auto-verification (hooks, tests, browser automation), sandboxing, guardrails/permissions, and observability/evals.

The evidence (Vercel 80→100%, LangChain bottom→top-5, CORE-Bench 42→78%, Harvey 2×, OpenAI ARC-AGI-3 13→38%) supports the thesis that the model is the smallest part of the system and the harness is where performance and reliability are increasingly won or lost. The winning pattern of 2025–2026 is minimal but disciplined: a bash-centric tool loop, aggressive compaction, filesystem-based persistent state (feature lists + progress files + git), sandboxed execution, subagents for parallelism, and mandatory end-to-end auto-verification with real tools.

Interoperability consolidated under the Agentic AI Foundation (MCP + A2A), and evaluation is shifting from peak accuracy to reliability, cost-per-task, and time-horizon.

Final matiz: the effect is regime-dependent, and the harness sits on top of a platform layer (durable execution, governance, cost, data integration) that the discourse is only beginning to name.

Compiled from primary sources (arXiv, Anthropic/OpenAI engineering posts, SWE-bench/METR leaderboards, protocol docs) plus secondary reporting on the Claude Code leak and industry events; where figures come from secondary sources they are flagged in the text.