Pith.

Software Engineering · arXiv:2610.20344 v1 · submitted 14 Oct 2026 · 22 references

Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents

Yuyao Ge, Shenghua Liu, Peizhong Gao, Chendi Ge, Chenxu Liu, Yiwei Wang, Lingrui Mei, Baolong Bi

show 3 more authors

Jiayu Yao, Jiafeng Guo, Xueqi Cheng

The paper says

“Agents driven by large language models are increasingly deployed on long-horizon tasks, where they use tools and modify real workspaces. How well they perform depends as much on the harness around the model as on the model itself.”

more of the abstract

Existing benchmarks measure downstream task success under a fixed harness, leaving harness engineering itself unexplored. We propose Doc2Harness, a two-stage benchmark that evaluates long-horizon harness engineering directly. In Stage 1, a build model operating inside a fixed builder harness receives an interface-only contract skeleton and a development document that consolidates behavior across twelve frontier harnesses, and then constructs a complete candidate harness in an isolated container. In Stage 2, the frozen candidate harness runs in containers separate from the construction environment, one per test case, and is driven end to end by a scripted mock model. We score each test case using four observable channels, namely the requests sent to the model, the files on disk, the process exit code, and the command output; positive controls rule out vacuous passes. Across nineteen frontier models, each evaluated with three independent builds on 328 test cases, GPT-6 Astra achieves the highest pass rate at 53.05%, while the weakest model reaches 27.44%, a gap of roughly twenty-six percentage points. Even the strongest model fails nearly half of the test cases, and failures cluster around how cross-cutting functionality is split across modules. To our knowledge, Doc2Harness provides the first deterministic benchmark for measuring long-horizon harness engineering.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 15 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. A real 'first': Doc2Harness scores a built harness directly against its spec via a scripted mock model, but the 328 test assertions it rests on are assumed correct, never validated against a reference harness. This line is wrong

3 major objections · 5 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 22 references say.

The paper says, at §2.3–§2.4, Table 2

The review did not quote this passage.

Pith says: Objection 1 of 3 · major

The load-bearing premise that the 328 test-case assertions encode feasible behavior required by the development document is assumed, not demonstrated. Section 2.4 opens with "Given that the assertion is correctly constructed," and the positive controls it describes rule out vacuous passes (the feature under test never ran) but not wrong assertions (the feature ran and behaved acceptably while the assertion demands something the document does not require, or asserts something self-contradictory).

more of objection 1

The synthesis pipeline of Section 2.3 is an author-driven filtering process, and no gold or reference harness is ever scored on the suite; this contrasts with the Section 5.1 benchmarks, where Commit0 and NL2Repo-Bench use upstream tests, ProgramBench uses a reference executable, and DeepSWE hand-writes verifiers. The KR-20 of 0.913 in Section 4.2 measures internal consistency, that is, whether items co-vary, not whether they are correct. If even a modest fraction of assertions are wrong, the 53.05 versus 27.44 spread, the category gaps in Table 3, and the model ranking shift together. I therefore request a validity check: run at least one reference implementation (for example, a fully implemented mini-swe-agent, or a mature open harness adapted to the contract skeleton) through the entire stage-two protocol and report its per-category pass rates; and/or have independent annotators re-derive a sample of assertions from the clauses and report agreement, along with the rate at which assertions were corrected or discarded during synthesis.

The paper says, at §2.2.1

The review did not quote this passage.

Pith says: Objection 2 of 3 · major

The contamination-reduction claim is unmeasured. The paper states that the development document is rewritten from the twelve harnesses' documentation "after merging and disambiguation" to reduce reliance on parametric knowledge, and Section 5.3 emphasizes that the deliverable "does not correspond to any existing implementation."

more of objection 2

No measurement supports either claim: frontier models have very likely been trained on the public documentation of Claude Code, Codex CLI, and the other ten harnesses, so a model that has memorized those feature semantics could score higher on the corresponding clauses without being better at harness engineering. I recommend a concrete probe (for example, token-overlap statistics between the development document and the twelve source documentations, or a comparison against an unmodified source document on a per-clause-family basis) or a softened claim that states the residual familiarity risk; as written, interpreting the ranking as "harness-engineering ability" rests partly on this assumption.

The paper says, at §4.3

The review did not quote this passage.

Pith says: Objection 3 of 3 · major

The fragmentation finding, which appears in the abstract (b9) and is concluded as "Fragmentation is therefore not free," is an interpretation that outstrips the observational evidence. Table 4 shows that 95.9% of the variance in code lines is between models, so the partial correlation of -0.423 between module count and pass rate (controlling for code lines) remains a between-model correlation …

more of objection 3

… in which model identity is the natural confound: more capable models may simply write more consolidated code, rather than consolidation causing higher pass rates. The leave-one-model-out range (-0.484 to -0.309) shows stability but does not address confounding. I suggest a within-model check using the three builds per model, a causal disclaimer, or a design that manipulates module structure directly; the introduction's wording ("tend to place ... in fewer modules") is appropriately correlational, and the abstract and Section 4.3 should stay at that level.

5 minor objections

The paper says, at Abstract; §7

Pith says: minor objection 1 of 5

The abstract and conclusion call Doc2Harness a "deterministic benchmark" (b190), but determinism holds only for Stage 2; Stage 1 construction is a stochastic LLM build. Section 2.1 states this distinction correctly, so the abstract and conclusion should qualify the claim (for example, "deterministic evaluation of the frozen harness").

The paper says, at §2.3.1

Pith says: minor objection 2 of 5

The step that ranks the 21,083 collected issues by "relevance to the development document" is not described: it is unclear whether ranking is done by an LLM, by a retrieval score, or manually, and how the stopping threshold was chosen; please specify the procedure and report the number of issues actually reviewed.

show 3 more

The paper says, at §2.2.2; §2.4

Pith says: minor objection 3 of 5

Please state the release plan for the development document, the contract skeleton, the 328 assertions with their positive controls, and the stage-two scoring harness beyond the project URL in the byline, so that the reproducibility and determinism claims can be independently verified.

The paper says, at §4.2

Pith says: minor objection 4 of 5

Given the authors' own acknowledgement that single-variable expansion yields correlated test cases (b193-b194), the KR-20 of 0.913 should be read as partly reflecting that construction correlation; one sentence noting this would prevent readers from treating the coefficient as evidence of 328 independent measurements.

The paper says, at §4.1; Table 2

Pith says: minor objection 5 of 5

Adjacent models in the middle of the ranking differ by less than their three-build standard deviations (for example, GPT-5.6-sol 43.80 +/- 2.44 versus Grok 4.7 43.50 +/- 2.49), so the fine ordering around positions 4-8 is within construction noise; consider reporting noise-aware rank bands or explicitly limiting the ranking claims to coarse tiers.

What would settle it

Recruit two independent engineers who have not seen the test suite. Give each only the development document and the contract skeleton, and ask them to implement a candidate harness that passes the contract check. Run all 328 test cases against both frozen harnesses. If a carefully built, contract-check-passing harness fails more than a small fraction of assertions, the document underdetermines the asserted behavior or the assertions encode unstated requirements, so the benchmark's validity is not established. For each failing case, require the engineers to point to the document clause that justifies the assertion; if they cannot, that assertion is not derivable from the specification. This also provides a calibration point: a conforming harness should score near ceiling, far above the 27-53 percent model range.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 15 Oct 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 49 s · 19,009 tokens read, 5,122 written
  3. Editor wrote the verdict above 1 min 8 s · 19,934 tokens read, 7,417 written
  4. Referee major revision: 3 major objections, 5 minor 1 min 49 s · 19,223 tokens read, 13,535 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Short version: this is the first benchmark I know that scores the harness itself rather than the downstream task, and the move that makes it work — driving the frozen harness with a scripted mock model in stage two — is the right call. But the instrument's validity rests on an assumption the paper states rather than demonstrates: section 2.4 opens with 'Given that the assertion is correctly constructed,' and no reference harness is ever scored against the 328 test cases.

What's genuinely new and good. Prior harness-creation benchmarks (HarnessDev, Evo-Bench, HarnessOpt-Bench) score through downstream task success, which conflates harness quality with the driving model's ability. Doc2Harness decouples the two. The mock model makes stage-two evaluation deterministic and re-runnable; the four observation channels — requests, files, exit code, output — make constraints like 'a rejected operation must not happen' directly checkable; and the positive-control design (channel, same-form, reverse) is a serious attempt to kill vacuous passes. The empirical discipline is real: three independent builds per model with reported standard deviations, isolated containers, frozen deliverables, 130 of 328 cases gated by controls, and an honest limitations section that flags the early-submitter lower-bound problem. The KR-20 is properly scoped as internal consistency, not validity, and the architecture analysis uses the right checks: partial correlation of module count with pass rate given code lines, with leave-one-model-out bounds.

Where it is soft, in proportion. The big one is assertion validity, and the stress-test concern lands on reading the paper. The synthesis pipeline in section 2.3 is author-driven filtering — issue clustering, cross-harness diffing, manual attribution to clauses — with no gold or reference harness scored on the suite. Positive controls rule out vacuous passes; they do not validate assertions. If a modest fraction of the 328 assertions are over-strict or misstate what the document requires, every entry in Table 2 and the 53-to-27 spread shift together. I do not think that sinks the paper — the assertions are anchored to a document the authors wrote, so this is a calibration gap rather than an invented result — but it is load-bearing, and the fix is straightforward: release the suite and score a strong reference harness. Release is the blocking worry in the strictest sense, since the artifact link in the paper is garbled and nothing can be inspected. Secondary: the contamination story — document and tests derived from the same repository family the models trained on — is acknowledged in design but never measured. And the conclusion's claim that failures 'concentrate on' module fragmentation outruns the evidence: it is a partial correlation across models, and the results section is more careful than the conclusion.

Who this is for: people building agent-evaluation benchmarks, and anyone working on the harness-versus-model confound. It deserves a serious referee. My recommendation: send it out, and make artifact release plus reference-harness validation the conditions; with those two, the central claim becomes credible.

The strongest objection

The central claim requires that a PASS or FAIL on each of the 328 test cases reflects genuine conformance of the candidate harness to the development document. Nothing in the paper establishes this. Section 2.4's positive controls rule out vacuous passes, not wrong assertions; the section itself opens with 'Given that the assertion is correctly constructed.' The synthesis pipeline in section 2.3 is an author-driven filtering process (issue clustering, cross-harness diffs, single-variable expansion, manual ranking) with no gold or reference harness scored on the suite. If even a modest fraction of assertions are over-strict, self-contradictory, or test behavior the document does not actually require, every entry in Table 2, the 53.05 versus 27.44 spread, and the model ranking shift together. The KR-20 of 0.913 does not help: internal consistency measures correlated test items, not whether the items are correct. The authors acknowledge related limitations (b191, b193-b194), but not assertion validation. This concern is prior to the contamination worry in section 2.2.1: rewriting the documentation may reduce memorization, but it cannot fix a suite whose assertions misstate the requirements.

The strongest claim is b190: deterministic assessment of how well a model develops a harness. Stage 2 is indeed deterministic; the open question is whether the verdicts are correct. I read the construction and evaluation in good faith. The mock-model design and positive controls are genuine advances: the controls address vacuous passes in section 2.4, and the architecture analysis is internally consistent. The paper honestly reports limitations such as the single-build protocol (b191-b192) and correlated expansion variants (b193-b194). The claim I could not support from the text is assertion validity. The test suite is the measurement instrument; unlike SWE-bench, where assertions are upstream unit tests, here the assertions are synthesized by the authors from issue reports, source diffs, and expansion, then filtered by judgment. There is no external check. The sentence 'Given that the assertion is correctly constructed' in section 2.4 explicitly brackets the exact condition on which the central claim depends. I considered contamination as the alternative concern; it is real for model ranking, but even a perfectly uncontaminated suite fails if assertions misstate requirements, so assertion validity is prior. I also considered the module-fragmentation conclusion; the paper's own numbers are correlational and the abstract's causal phrasing overshoots, but that is a secondary finding rather than the benchmark's central claim. The proposed independent-implementation study tests whether the development document is sufficient to build a harness that passes the assertions, which is the missing feasibility check for assertion correctness. This does not ask the authors to re-prove any published result; it validates their own artifact. Conditional remains the appropriate verdict.

The referee's report

The paper addresses a real gap: existing harness benchmarks either fix the harness and measure downstream task success, or infer harness quality from downstream outcomes, which cannot separate the harness's contribution from that of the driving model (Section 5.2). Making the harness itself the object of measurement, adjudicated clause by clause, is a genuine conceptual advance. The protocol is carefully engineered, with three independent builds per model and reported standard deviations, preflight checks, a frozen deliverable after the contract check, per-test-case isolated containers, and positive controls gating 130 of 328 test cases; the stage-two measurement is externally observable and reproducible, since a frozen harness actually runs and produces requests, files, exit codes, and outputs that are checked deterministically. The suite discriminates in this cohort (a roughly 26-point top-bottom spread; 256 of 328 test cases are discriminating per Table 3), the synthesis pipeline is transparently documented, and the authors explicitly state their main limitations (b191-b194).

Pith's reading of the paper (a machine's summary, not the authors')

Doc2Harness is a proposed benchmark for measuring whether a large language model can engineer the control software, the "harness," that surrounds a coding agent, as opposed to solving a downstream coding task. The paper's central claim is that harness-engineering skill can be assessed deterministically and without confounding by the driving model: a build model constructs a candidate harness from an interface-only contract skeleton and a development document consolidating the behavior of twelve existing harnesses, then a scripted mock model drives the frozen result through 328 test cases, with verdicts read from four observable channels (requests sent to the model, files on disk, exit code, and command output) and positive controls ruling out vacuous passes. Across nineteen frontier models with three independent builds each, pass rates range from 27.44% to 53.05%, and the paper reports high internal consistency with a KR-20 of 0.913. Because harness quality is currently inferred only from downstream task success, where the driving model's ability contaminates the score, a working version of this benchmark would let harness quality be measured, ranked, and ultimately improved separately from model quality.

What carries the argument. The central object is the "driven harness": the frozen candidate harness paired with a scripted mock model that plays the role a real LLM would play, so that the harness alone is under test. Three parts of the mechanism carry the argument. First, the contract check in Stage 1 freezes the deliverable at the model's own submission, verifying only form, that it compiles, installs, exposes the required commands, and keeps the three contract interfaces, so the scored artifact is what the model chose to hand in. Second, the four-channel observation scheme, the requests sent to the model, the files on disk, the process exit code, and the command output, makes every clause of the development document verifiable from external behavior. Third, positive controls, which gate 130 of the 328 test cases, share the scenario and fixture but change the access route or input, so a test case is recorded as PASS only when the precondition for the observed effect actually holds.

Core claim. The discovery the paper seeks to establish is that the quality of a harness, the layer that mediates between an LLM and its tools, workspace, permissions, and session state, can be evaluated on its own, without running a downstream task and without a real model in the loop. The evaluation protocol carries the argument: freeze the candidate harness after construction, then drive it end to end with a scripted mock model in a fresh container per test case; an assertion-based verdict aggregates effects across the four channels, and positive controls ensure that a pass means the feature actually ran rather than that the observation mechanism silently failed. On this protocol the paper reports that GPT-6 Astra reaches a 53.05% pass rate, the weakest evaluated model, Hy3, reaches 27.44%, and even the strongest model fails nearly half of the 328 test cases. The authors further report that the fifty-seven scored harnesses reveal one systematic correlate of success: after controlling for code size, harnesses that spread functionality across more modules score lower, with a partial correlation of $-0.423$ between module count and pass rate. The paper claims this makes Doc2Harness the first benchmark able to assess deterministically how well a model develops a harness; the determinism holds for the evaluation stage, since construction itself is a stochastic LLM build.

The premise it rests on. The load-bearing premise is that the 328 test-case assertions are correct, meaning each one encodes behavior that a conforming harness can actually exhibit, because the suite is never validated against a reference or gold harness and the paper's own positive-control analysis opens with the caveat "Given that the assertion is correctly constructed."

If the paper is right (Pith's inferences, not the authors')

Harness quality can be ranked as a number independent of the model that would drive it, so benchmark scores no longer conflate the harness with the agent.

The 26-point gap between the strongest and weakest evaluated models shows the suite discriminates within this cohort of frontier models, and the KR-20 of 0.913 indicates the test cases behave as if they measure one underlying capability.

The architecture analysis predicts a concrete design rule: models that concentrate cross-cutting functionality in fewer modules tend to build better-scoring harnesses.

Because a frozen harness on a given test case yields the same verdict every time, results are reproducible, making the benchmark a stable measurement instrument for future model generations.

Model ranks shift by up to sixteen positions across categories, so a single overall standing is not the whole story; category-level scores distinguish which aspects of harness construction a model handles.

Editorial extension: because Stage 2 is deterministic and cheap to run per verdict, the pass rate could double as a reward signal for training harness-building models, even though the current protocol explicitly measures a single build rather than an iterative repair loop.

Editorial extension: the paper assumes that rewriting the twelve source harnesses' documentation removes the parametric advantage of models trained on those repositories, but no contamination measurement is reported; a testable check would be to hold one harness's documentation entirely out of the development document and see whether the ranking shifts.

Editorial extension: the positive-control mechanism transfers beyond harnesses, since any agent-safety assertion of the form "a forbidden action did not happen" can be vacuous, and the same precondition-gating logic could be applied to other absence-style claims in agent evaluation.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Agents driven by large language models are increasingly deployed on long-horizon tasks, where they use tools and modify real workspaces. How well they perform depends as much on the harness around the model as on the model itself. Existing benchmarks measure downstream task success under a fixed harness, leaving harness engineering itself unexplored. We propose Doc2Harness, a two-stage benchmark that evaluates long-horizon harness engineering directly. In Stage 1, a build model operating inside a fixed builder harness receives an interface-only contract skeleton and a development document that consolidates behavior across twelve frontier harnesses, and then constructs a complete candidate harness in an isolated container. In Stage 2, the frozen candidate harness runs in containers separate from the construction environment, one per test case, and is driven end to end by a scripted mock model. We score each test case using four observable channels, namely the requests sent to the model, the files on disk, the process exit code, and the command output; positive controls rule out vacuous passes. Across nineteen frontier models, each evaluated with three independent builds on 328 test cases, GPT-6 Astra achieves the highest pass rate at 53.05%, while the weakest model reaches 27.44%, a gap of roughly twenty-six percentage points. Even the strongest model fails nearly half of the test cases, and failures cluster around how cross-cutting functionality is split across modules. To our knowledge, Doc2Harness provides the first deterministic benchmark for measuring long-horizon harness engineering.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-15

The authors' main claim, as the classifier picked it out 0.68

“Even the strongest model fails nearly half of the test cases, and failures cluster around how cross-cutting functionality is split across modules.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.94 · states limitation 0.65 · human participants 0.03 · controlled comparison 0.69 · randomized assignment 0.03 · states quantitative result 0.99 · compares with prior results 0.45 · mentions code or data availability 0.05

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Agents driven by large language models are increasingly deployed on long-horizon tasks, where they use tools and modify real workspaces. background 1.00 0.05 0.02 asserted 0.99
How well they perform depends as much on the harness around the model as on the model itself. background 0.98 0.11 0.08 asserted 1.00
Existing benchmarks measure downstream task success under a fixed harness, leaving harness engineering itself unexplored. gap 1.00 0.06 0.04 asserted 1.00
We propose Doc2Harness, a two-stage benchmark that evaluates long-horizon harness engineering directly. contribution 1.00 0.92 0.13 asserted 1.00
In Stage 1, a build model operating inside a fixed builder harness receives an interface-only contract skeleton and a development document that consolidates behavior across twelve frontier harnesses, and then constructs a complete candidate harness in an isolated container. method 0.99 0.86 0.04 asserted 1.00
In Stage 2, the frozen candidate harness runs in containers separate from the construction environment, one per test case, and is driven end to end by a scripted mock model. method 1.00 0.84 0.03 asserted 0.99
We score each test case using four observable channels, namely the requests sent to the model, the files on disk, the process exit code, and the command output; positive controls rule out vacuous passes. method 1.00 0.95 0.04 asserted 1.00
Across nineteen frontier models, each evaluated with three independent builds on 328 test cases, GPT-6 Astra achieves the highest pass rate at 53.05%, while the weakest model reaches 27.44%, a gap of roughly twenty-six percentage points. result 1.00 0.96 0.28 asserted 1.00
Even the strongest model fails nearly half of the test cases, and failures cluster around how cross-cutting functionality is split across modules. result 0.74 0.94 0.28 asserted 1.00
To our knowledge, Doc2Harness provides the first deterministic benchmark for measuring long-horizon harness engineering. contribution 0.93 0.82 0.23 asserted 0.84

What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from observation or experiment 0.98

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-15: 1 request, 7,620 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (5)
Figure 1 of the paper
Figure 1. Pass rate of the nineteen models on the Doc2Harness benchmark, ordered from the strongest model on the left to the weakest on the right. Each model is scored on three independently built candidate harnesses, and each bar gives their mean. Abstract Agents driven by large language models are increasingly deployed on long￾horizon tasks, where they use tools and modify real workspaces. How well they perform depends as much on the harness around the model as on the model itself. Existing benchmarks measure downstream task success under a fixed harness, leaving harness engineering itself…
Figure 2 of the paper
Figure 2. Overview of Doc2Harness. Top (Stage 1): the builder, formed from the build model under evaluation and a fixed builder harness (OpenCode), takes the development document and the contract skeleton as input and emits a submission; the contract check verifies form rather than behavior, freezing the deliverable as the candidate harness on success and returning it to the builder on failure. Bottom (Stage 2): the frozen candidate harness and a mock model form the driven harness, which runs the scenario of each test case; effects are read from four channels, namely the requests sent to the…
Figure 3 of the paper
Figure 3. The three test case synthesis methods. Collection. We pull every issue from the public repositories of four frontier harnesses (Claude Code, OpenCode, Gemini CLI, and Kimi Code); after filtering out pull requests these number 9,868, 5,438, 4,831, and 946 respectively, 21,083 in total. Ranking and stopping. More than 20,000 records cannot be reviewed manually in full, so we first rank them by relevance to the development document. Manual review starts at the top of the ranking and stops when the marginal yield falls below the cost of further review. Clustering. The same defect is…
Figure 4 of the paper
Figure 4. File operations and permissions have the highest positive-control coverage, whereas the interactive terminal has the lowest. Artifact checks is marked NA, because the contract check already settles in full what its test cases examine, namely that the deliverable is complete. 2026); Kimi K3 (Kimi Team, 2026); GLM 5.2 (Z.ai, 2026a) and GLM 5.3 (Z.ai, 2026b); DeepSeek V4 Pro (DeepSeek-AI, 2026b) and DeepSeek V4.1 Flash (DeepSeek-AI, 2026a); Hy3 (Ten￾cent Hunyuan, 2026a) and Hy4 preview (Tencent Hunyuan, 2026b); Doubao Seed 2.1 Pro (ByteDance Seed, 2026); MiMo-V2.6-Flash (Xiaomi MiMo,…
Figure 5 of the paper
Figure 5. Code lines against pass rate. Left, one point per model at its mean over the three scored builds, with a least-squares fit and its 95% confidence band. Right, each model’s three scored builds, spanned by a segment with the mean marked, one row per model, ordered by pass rate. are 87.9% for module count, 80.6% for lines per module, and 95.7% for largest-module share;

Figures are the authors', reproduced from the paper.

References (22)
  1. [6] Sanderson Oliveira de Macedo. What makes a harness a harness: necessary and sufficient conditions for an agent harness.arXiv preprint arXiv:2606.10106, on PithPith's reviewarXiv
  2. [7] DeepSeek-AI. DeepSeek-V4.1-Flash: Smarter, faster, more efficient. https://api-docs.deep seek.com/news/news260910/, 2026a. DeepSeek-AI. DeepSeek-V4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026b. Xiang Deng, Jeff Da, Edwin ... on PitharXiv
  3. [8] Yuyao Ge, Lingrui Mei, Zenghao Duan, Tianhao Li, Yujia Zheng, Yiwei Wang, Lexin Wang, Jiayu Yao, Tianyu Liu, Yujun Cai, Baolong Bi, Fangda Guo, Jiafeng Guo, Shenghua Liu, and Xueqi Cheng. A survey of vibe coding with large language models.arXiv preprint arXiv:2510.12399, on PitharXiv
  4. [9] Jianyuan Guo, Zhiwei Hao, Chengcheng Wang, Cheng Fan, Tingzhang Luo, Hongguang Li, Ying Gao, Hefei Mei, Jiankun Peng, Rongjian Xu, et al. From question answering to task completion: A survey on agent system and harness design.arXiv preprint arXiv:2606.20683, on PithPith's reviewarXiv
  5. [10] Jinyi Han, Yuanjian Xu, Ying Liao, Xinyi Wang, Zishang Jiang, Zixiang Di, Fanyang Lu, Zhichao Hu, and Yanghua Xiao. Skill-Use: Can LLMs actually use skills in agentic harnesses?arXiv preprint arXiv:2608.04828, on PithPith's reviewarXiv
  6. [11] Lisheng Huang, Chen Yang, Hao Zhou, Huatong Song, Zongchao Chen, Ran Le, Yang Song, Wayne Xin Zhao, and Tao Zhang. Evo-Bench: Can language models improve agent harness?arXiv preprint arXiv:2608.09096, 2026a. Wenqi Huang, Charley Lee, Leonard Tng, and Serena Ge. DeepSWE: Measur... on PithPith's reviewarXiv
  7. [12] Zixuan Ke, Vaidehi Patil, Haizhou Shi, Yang Li, Ye Liu, Sarath Shekkizhar, Anurag Koul, Jiayu Wang, Xuan Phi Nguyen, Semih Yavuz, et al. EvoHarnessBench: Can your agents keep pace with an evolving harness?arXiv preprint arXiv:2609.04280, on PithPith's reviewarXiv
  8. [13] 15 Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents Kimi Team. Kimi K3: Open frontier intelligence.arXiv preprint arXiv:2607.24653, on PitharXiv
  9. [14] Chengzhi Liu, Yichen Guo, Yepeng Liu, Yuzhe Yang, Qianqi Yan, Xuandong Zhao, Wenyue Hua, Sheng Liu, Sharon Li, Yuheng Bu, and Xin Eric Wang. Auditing agent harness safety. arXiv preprint arXiv:2605.14271, on PithPith's reviewarXiv
  10. [15] Lingrui Mei, Jiayu Yao, Yuyao Ge, Yiwei Wang, Baolong Bi, Yujun Cai, Jiazhi Liu, Mingyu Li, Zhong-Zhi Li, Duzhen Zhang, Chenlin Zhou, Jiayi Mao, Tianze Xia, Jiafeng Guo, and Shenghua Liu. A survey of context engineering for large language models.arXiv preprint arXiv:2507.13334, on PithPith's reviewarXiv
  11. [16] Damien Sileo and Dimitri Kachler. CordisBench: Can language models reason about component lifecycles in dynamic agent harnesses?arXiv preprint arXiv:2609.01600, on PithPith's reviewarXiv
  12. [17] Tencent Hunyuan. Hy3 model card.https://huggingface.co/tencent/Hy3, 2026a. Tencent Hunyuan. Hy4 preview model card. https://huggingface.co/tencent/Hy4-previ ew, 2026b. Varun Ursekar, Apaar Shanker, Yash Maurya, Shehab Yasser, Vijay S. Kalmath, Veronica Chatrath, and Yuan Xue. ... on PithPith's reviewarXiv
  13. [18] Yike Wang, Huaisheng Zhu, Zhengyu Hu, Yige Yuan, Zhengyu Chen, Shakti Senthil, Han- naneh Hajishirzi, Yulia Tsvetkov, Noah A. Smith, Pradeep Dasigi, and Teng Xiao. Re- thinking the evaluation of harness evolution for agents.arXiv preprint arXiv:2607.12227, on PithPith's reviewarXiv
  14. [19] Yuhao Wu, Jingyuan Zhang, Jiajun Shi, Xinping Lei, Qingshui Gu, Yuxuan Zhang, Zexuan Wang, Chen He, Chen Huang, Maojia Song, et al. HarnessDev: Can LLMs create and evolve their own agent harness?arXiv preprint arXiv:2609.01437, on PithPith's reviewarXiv
  15. [20] Yilun Yao, Xinyu Tan, Chao-Hsuan Liu, Yaoming Li, Zhengyang Wang, Wenhan Yu, Zhewen Tan, Yuxuan Tian, Guangxiang Zhao, Lin Sun, et al. Harness-Bench: Measuring harness effects across models in realistic agent workflows.arXiv preprint arXiv:2605.27922, on PithPith's reviewarXiv
  16. [21] Z.ai. GLM-5.2: Built for long-horizon tasks.https://z.ai/blog/glm-5.2, 2026a. Z.ai. GLM-5.3: Frontier coding with emergent cyber capabilities. https://z.ai/blog/gl m-5.3, 2026b. Xiao Zhang, Yusheng Wang, Yuhao Fei, Dongyuan Li, Zian Liang, Liuyu Xiang, Hongxun Gu, and Zhaofeng... on PithPith's reviewarXiv
  17. [22] Mengyu Zheng, Kai Han, Boxun Li, Haiyang Xu, Yuchuan Tian, Wei He, Hang Zhou, Jianyuan Guo, Hailin Hu, Lin Ma, et al. Claw-SWE-Bench: A benchmark for evaluating OpenClaw-style agent harnesses on coding tasks.arXiv preprint arXiv:2606.12344, on PithPith's reviewarXiv
  18. [2021] Yajing Bai, Jinhao Duan, Jie Peng, Xianfeng Wu, Sijia Liu, Song Wang, and Tianlong Chen. HarnessRisk: A lifecycle-oriented benchmark for agent harness safety.arXiv preprint arXiv:2608.17597, on PithPith's reviewarXiv
  19. [2023] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, on PithPith's reviewarXiv
  20. [2024] 14 Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents Dotan Davidovich, Yair Amar, Hai Rozencwajg, and Or Hiltch. Permission denied: Policy-graded evaluation of coding agents in hardened environments.arXiv preprint arXiv:2608.02670, on PithPith's reviewarXiv
show 2 more
  1. [2025] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evalu- ating large language models trained on code.arXiv preprint arXiv:2107.03374, on PithPith's reviewarXiv
  2. [2026] Paul Barbaste, Tristan Darrigol, Germain Vu, and Tom Wiltberger. Harness Engineering: Anatomy, architecture, and evolution of coding agents – A source-code study of eleven systems.arXiv preprint arXiv:2609.00006, on PithPith's reviewarXiv

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment