Pith.

Machine Learning · arXiv:2610.26770 v2 · submitted 19 Oct 2026 · 26 references

HarnessPro: Recursive Harness Self-Improvement with Pareto Optimization for Performance-Cost Trade-offs

Zhuoshi Pan, ZhiYuan Feng, Yukuo Cen, Honglin Lin, Junru Lu, Zhongxia Chen, Shuo Yang, H. Vicky Zhao

show 1 more author

Jie Tang

The paper says

“Harness development plays a crucial role in building capable AI agents. Recent approaches improve harnesses through recursive self-improvement (RSI), but their performance gains often come with higher execution costs.”

more of the abstract

We introduce HarnessPro, a label-free method for evolving harnesses to improve task performance while reducing execution cost. In this setting, candidate harnesses are evaluated using LLM-generated tests without access to ground-truth solutions or official tests. Specialized agents diagnose execution trajectories, plan updates with different performance-cost priorities, and implement them as candidate harnesses. Rather than returning a single harness, HarnessPro uses normalized weighted squared-distance selection to provide cost-focused (lite), balanced (medium), and performance-focused (heavy) harnesses for different deployment budgets. Selected harnesses enter the next iteration, where execution feedback guides further updates. Our experiments cover Terminal-Bench 2, SWE-Bench Pro, and ProgramBench, using GPT-5.5 with Codex and GLM-5.2 with Claude Code. Across these benchmarks using GPT-5.5 with Codex, the heavy mode of HarnessPro achieves an average pass-rate gain of 2.0 percentage points and an average per-task cost reduction of 26.1% relative to RHO. This advantage also holds when using GLM-5.2 with Claude Code. Code is available at https://github.com/pzs19/HarnessPro

from the abstract

Pith says

Machine review · deepseek-v4-flash · 21 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. Real cost-cuts, honestly packaged, sitting on a weakly validated internal proxy: the direction of the claim holds, the margins over RHO are thinner than the headlines suggest. This line is wrong

4 major objections · 6 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 26 references say.

The paper says, at §4.3, Fig. 2; Algorithm 1 lines 7, 17-19; Eqs. (7), (9)

The review did not quote this passage.

Pith says: Objection 1 of 4 · major

Every internal decision in evolution, including the rollout pass labels on the training coreset (Algorithm 1, line 7), the pass-rate floor and Pareto filter in Eq. (7), and mode selection in Eq. (9), is made on the LLM-generated tests S(x), while the reported results use official tests that enter only in the final evaluation (Algorithm 1, line 22).

more of objection 1

The authors themselves call Figure 2 preliminary, and at that size (about 20 pooled tasks from two of three benchmarks, with 12.5% false fails on official-pass tasks and 16.7% false passes on official-fail tasks) it supports marginal failure recall rather than ranking validity. The load-bearing question is whether the S(x)-based ordering of candidates agrees with the official-test ordering at each filter/selection step; a 12.5-16.7% task-level disagreement can change which harness survives Eq. (7) or is chosen by Eq. (9), and if the errors favor harnesses that satisfy weak generated tests, the final official numbers would be an optimistically biased view of the frontier. The careful design of the test synthesizer (fixed suites, no official-test access, validation against the frozen Empty product in Listing 4) mitigates but does not remove this concern, because the residual risk is cross-candidate selection bias. Please report, on the validation splits for which official graders exist, the S-based versus official-based pass rates and rankings for all candidates in the final epoch and at least one intermediate epoch, and state how many Eq. (7)/(9) decisions would change under official grading.

The paper says, at §3, §5.4, Fig. 4

The review did not quote this passage.

Pith says: Objection 2 of 4 · major

Section 3 states that the test set is used only for final evaluation after evolution, and Algorithm 1 (line 22) emphasizes that the returned harnesses are fixed before held-out test evaluation. Figure 4, however, plots test-set pass rate and cost at every intermediate epoch, which requires executing candidate harnesses on held-out test tasks during the evolution period.

more of objection 2

Please clarify the actual protocol: were test tasks run at intermediate epochs, what was visible to the authors versus the evolution agents, and how was the final evaluation kept blind? If test tasks were executed during evolution even for visualization, the current wording in Section 3 is inaccurate, and the possibility that human designers adjusted epochs, weights, seeds, or code in response to intermediate test-set plots should be explicitly addressed, even in the absence of any evidence that such adjustment occurred.

The paper says, at §5.2, Table 2

The review did not quote this passage.

Pith says: Objection 3 of 4 · major

The headline claim that the heavy mode improves pass rate while reducing cost rests on uneven evidence across benchmarks. Terminal-Bench 2 shows a 4.5-pp pass-rate gain, but the SWE-Bench Pro gain is 0.7 pp (80.7±0.6 vs. 80.0) and the ProgramBench gain is 0.8 pp (69.6±1.03 vs. 68.8), each within the reported run-to-run standard deviation, while RHO is reported as a single number with no variance.

more of objection 3

The abstract's average pass-rate gain of 2.0 pp is dominated by Terminal-Bench 2. Because the cost reductions are large and consistent, the cost half of the claim is robust, but the 'improve task performance' half is currently supported at the noise level on two of the three benchmarks. Please report per-seed paired results (including RHO run variance), significance tests such as paired bootstrap over the held-out tasks, or an explicit statement of which benchmarks support the pass-rate claim at what precision.

The paper says, at §5.1, §4.1, Appendix A.1

The review did not quote this passage.

Pith says: Objection 4 of 4 · major

Evolution is driven by a single 10-task DPP coreset with G=3 rollouts, and the DPP difficulty scores and failure fingerprints come from an LLM judge (Appendix A.1). The paper states this setup clearly but provides no sensitivity evidence that the coreset is representative: there is no comparison against a random coreset, no variation in k (e.g., 5 or 20), and no variation in rollout count G.

more of objection 4

Since the final harnesses must generalize to 59-100 held-out test tasks, a misrepresentative coreset would compromise the mechanism claims even if the reported numbers are accurate. Please add at least one coreset-size or coreset-selection sensitivity experiment, or report the DPP-selected task list and its coverage statistics so that readers can judge representativeness.

6 minor objections

The paper says, at Fig. 2

Pith says: minor objection 1 of 6

Please report the exact N and cell counts for each confusion matrix and state whether the units are tasks or artifacts. With N on the order of 20, the 12.5% and 16.7% error rates each correspond to one or two tasks, and this granularity should be visible in the figure itself rather than only in the text.

The paper says, at Table 2, §5.1

Pith says: minor objection 2 of 6

Please state explicitly whether the RHO and other baseline numbers were recomputed by the authors under the same pricing tables, hardware, and grading settings or quoted from the cited papers; dollar costs are only comparable across methods under identical pricing assumptions.

show 4 more

The paper says, at Eq. (8)

Pith says: minor objection 3 of 6

Please specify the convention when P_max equals P_min or C_max equals C_min (the normalized deviations become 0/0), and state explicitly that the extrema are taken over the filtered candidate set of the current epoch, so the selection criterion is relative to each epoch's pool.

The paper says, at §3, §5.1, Table 2

Pith says: minor objection 4 of 6

Section 3 defines Pass(tau)=1 only if all official tests pass, but Section 5.1 and Table 2 use the ProgramBench macro-averaged pass rate, so the pass-rate metric is defined differently across benchmarks; the cross-benchmark comparisons should either unify the definition or explicitly report each benchmark's metric.

The paper says, at Appendix D.2, Listing 21

Pith says: minor objection 5 of 6

The 'confidence overrides' field with its 0.9 threshold is not explained in the main text; please define what confidence signal the classifier produces and how the override interacts with the profile rubric when 'lean' is selected.

The paper says, at References

Pith says: minor objection 6 of 6

Please check for PDF-generation spacing artifacts in the reference list, such as 'The F ourteenth' in the Erol et al. and Suzgun et al. entries, and verify that the title and author names render correctly in the camera-ready version.

What would settle it

Run official graders on every candidate harness retained at every epoch and mode (and ideally all evaluated candidates) on the full validation sets of all three benchmarks, without using these labels to change evolution. Then compare (i) per-benchmark task-level agreement between generated-test and official-test pass labels on validation tasks, and (ii) for each mode and epoch, the fraction of runs in which the harness selected by Eq. 9 under S(x) would change if official validation pass rates were used instead. If that selection swap rate is material (e.g., above 10% of mode-epoch choices), or if ProgramBench agreement is much lower than the Figure 2 pooled numbers, the headline trade-off should be attributed to the proxy rather than to genuine frontier improvement; if agreement and selection stability are high, the concern is resolved.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 21 Oct 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 58 s · 33,544 tokens read, 6,518 written
  3. Editor wrote the verdict above 30 s · 34,480 tokens read, 3,653 written
  4. Referee major revision: 4 major objections, 6 minor 1 min 25 s · 33,988 tokens read, 13,524 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

You should know two things about this paper. First, the headline result is real: the heavy tier beats RHO on both pass rate and per-task cost on all three benchmarks, and the cost cuts are large (17.8–33.9%) and consistent across modes, benchmarks, and both backbones. Second, every internal selection decision in evolution — the pass-rate floor, the Pareto filter, the mode selection — runs on LLM-generated tests S(x), whose only validation is the small confusion matrix in Figure 2, pooled from about twenty training tasks. The authors call it preliminary themselves, and they never check whether S-based candidate rankings agree with official-test rankings at any epoch. That is the load-bearing soft spot.

What's actually new: putting cost into the objective instead of treating it as a side effect, running multiple evolution rounds (RHO is single-round), returning a lite/medium/heavy Pareto set rather than one harness, and replacing LLM self-preference with a test synthesizer. The ablations in Table 3 support the mechanism story — removing the planner or the test synthesizer degrades results — and the evolved harnesses in Appendix D are real artifacts, not vapor: profile rubrics with reasoning-effort gates, budget-denial hooks, RTK-style output compaction. The 3.4–10.0 pp drop when the test synthesizer is removed is the strongest evidence for the proxy; if the generated tests were worthless, removing them wouldn't hurt that much.

Where it's soft, in proportion. The cost side is solid: measured directly from token usage, consistent everywhere. The pass-rate side is credible but thinner than the abstract suggests. The 4.5 pp gain on Terminal-Bench 2 is real; the gains over RHO on SWE-Bench Pro (0.7 pp) and ProgramBench (0.8 pp) sit inside the reported standard deviations, and the baselines carry no variance. The 10-task coreset with three rollouts is small, though it follows RHO's protocol. The proxy gap is the main fixable issue, and it is fixable cheaply: checking S-versus-official agreement on validation candidates at one epoch would settle whether selection is optimizing the right thing. The paper's final official-test numbers are genuine measurements, so this is a selection-optimality concern, not a fabrication concern — but it can make the evolved harnesses look better or worse than they are.

Who this is for: anyone building or benchmarking harness-optimization pipelines for deployed coding agents under budget constraints. The tiered packaging maps directly onto real budgets, which is genuinely useful. Recommendation: yes, send it to review. It deserves referee time, with a request for the ranking-agreement check on the generated-test proxy and confidence intervals on baselines before acceptance.

The strongest objection

The central claim—that recursive harness improvement advances the official performance-cost frontier—depends on S(x) being a reliable surrogate for official tests. Algorithm 1 uses S(x) for every internal decision: training rollouts (line 7), the pass floor and Pareto filtering in Eq. 7, and mode selection in Eq. 9. Official tests enter only in the final evaluation after evolution (line 22). The support for this surrogate is Figure 2, which the authors themselves call preliminary: it pools training tasks from Terminal-Bench 2 and SWE-Bench Pro (about 20 tasks), omits ProgramBench, and shows 12.5% false fails on official-pass tasks and 16.7% false passes on official-fail tasks under generated tests. Because Eq. 7 discards candidates below the generated-test floor and Eq. 9 chooses representatives using generated-test pass rates, a 15% task-level disagreement between S and official outcomes can change which harness is kept. If the proxy errors are not random but reward harnesses that satisfy weak generated tests, the final official-test numbers would be an optimistic or at least unrepresentative view of the true frontier. The paper reports no check of whether the S-based ordering of candidates at any epoch matches the official-test ordering on the corresponding validation tasks, so the mechanism behind the headline gains is unverified.

The reader's weakest assumption identifies the same load-bearing premise: S(x) drives all selection while official tests drive reported results. My reading of the full text confirms this is the least secure point in the argument. The authors explicitly describe Figure 2 as a preliminary experiment, but the remainder of the evaluation pipeline does not quantify how representative S(x) is for validation candidates, on ProgramBench, or after harnesses evolve away from the initial harness whose trajectories generated S(x). A second concern, the 10-task DPP coreset, is real but secondary: it limits breadth of training signal, whereas an unreliable S(x) directly corrupts every pass/fail label used in Eqs. 7 and 9. I am not treating the final official-test results as fabricated or even as irrelevant; if the proposed validation check shows high agreement and stable selections, the conditional should be lifted. But without it, the paper's causal reading of the heavy-mode gains is not established. Because this matches the reader's conditional verdict exactly, no adjustment is needed. The cost reductions are large and consistent, and the ablations give the method internal support, which is why I would not move toward rejection; the missing piece is validation of the selection signal, not of the reported final evaluations themselves.

The referee's report

The empirical claim is non-circular: pass rates and dollar costs are measured on real executions, the selection hyperparameters are fixed across epochs (Appendix A.2), and official tests enter only in the final evaluation (Algorithm 1, line 22). If the results hold, the paper makes a genuine contribution by demonstrating that harness RSI can advance the performance-cost frontier on both axes rather than trading one against the other, and by replacing LLM preference judgments with fixed generated tests plus measured cost. The paper is unusually transparent: full evolution prompts (Appendix B), representative diagnoses and plans (Appendix C), deployed policy/hook files (Appendix D), pseudocode (Algorithm 1), and a code release are provided, and Table 3's RHO-variant ablations are a strong mechanism check, showing that single evolved mechanisms copied onto RHO do not reproduce the frontier gain. The main risks are the thin validation of the generated-test surrogate and the small margins of the pass-rate gains on two of the three benchmarks; both are addressable with additional analysis.

Pith's reading of the paper (a machine's summary, not the authors')

The paper sets out to show that a harness — the software layer that governs how an LLM agent reads, acts, and verifies — can be improved recursively so that task performance rises while execution cost falls, rather than trading one for the other. HarnessPro runs an evolution loop in which specialized agents diagnose failed or wasteful trajectories, propose harness edits under cost-focused, balanced, and performance-focused priorities, implement them, and score the candidates label-free on LLM-generated tests. Instead of returning one harness, it keeps the Pareto frontier and ships three operating points: a cost-focused lite mode, a balanced medium mode, and a performance-focused heavy mode. If the claim holds, deployment teams could pick a tier by budget without giving up measured performance, because the heavy mode reports higher pass rates than the RHO baseline on all three benchmarks while spending 26.1% less per task on average.

What carries the argument. The load-bearing machinery is a four-agent evolution loop plus a Pareto selection rule. A trajectory analyzer inspects several parallel rollouts per training task to separate genuine failures from avoidable overhead; an evolution planner proposes three edits per candidate under cost-focused, balanced, and performance-focused instructions; a harness editor implements them as executable candidates; and a test synthesizer produces a fixed suite $S(x)$ per task from the description and the initial trajectory, so every candidate is scored label-free on pass rate and mean cost. Candidates below the initial pass-rate floor or dominated by another candidate on both metrics are discarded, and the survivors are ranked by normalized weighted squared-distance selection, $J_m(h) = \alpha_m\,d_{\text{perf}}(h)^2 + \beta_m\,d_{\text{cost}}(h)^2$, with mode-specific weights, to pick the lite, medium, and heavy representatives. The evolved runtime mechanisms that deliver the cost savings are per-task profile routing — a rubric that maps each task description to a reasoning-effort level and a tool-call budget — plus runtime hooks that inject economy and convergence reminders at preset tool counts and deny further tool calls past the budget, together with tool-output compression and streamlined instructions.

Core claim. The central claim, on the paper's own terms, is that cost-aware recursive self-improvement advances the performance-cost Pareto frontier instead of sliding along it. Across Terminal-Bench 2, SWE-Bench Pro, and ProgramBench, with GPT-5.5 paired with Codex and GLM-5.2 paired with Claude Code, the heavy mode of HarnessPro is reported to beat RHO in pass rate while spending less per task: for instance, 80.8% versus 76.3% pass rate on Terminal-Bench 2 at 33.9% lower mean cost, and 69.6% versus 68.8% on ProgramBench at 17.8% lower cost. The medium mode holds nearly all of RHO's pass rate at far lower cost, cutting SWE-Bench Pro's per-task cost from $2.68 to $1.26. The paper attributes the gains to harnesses that assign each task a reasoning-effort level and a tool-call budget from the description alone, use runtime hooks to push execution from exploration into convergence, compress verbose tool output, and shorten instructions, so avoidable overhead disappears without removing the reasoning and verification that determine success. It further claims this works without labels: during evolution, candidates are judged only on fixed LLM-generated test suites, with no access to ground-truth solutions or official tests.

The premise it rests on. The whole claim rests on the LLM-generated test suites used for every in-evolution decision being a faithful proxy for the official tests that produce the reported results, because all filtering and tier selection run on generated-test pass rate while the headline numbers come from official tests.

If the paper is right (Pith's inferences, not the authors')

Performance and cost need not be traded against each other in harness optimization: the retained frontier moves in both directions across epochs, with the heavy mode beating RHO on pass rate and cost simultaneously.

Deployments with different budgets can pick lite, medium, or heavy directly from the evolved Pareto set, without re-running evolution for each budget.

Label-free evaluation can support recursion: because the generated tests are fixed before evolution and the validation and test sets stay hidden from the planner, candidates are compared on a stable score with no test-set leakage.

The gains come from a combination of mechanisms rather than one tweak: copying dynamic routing, budget hooks, or low reasoning effort individually onto RHO is Pareto-dominated by the evolved harnesses.

The reach of the claim depends on how well the LLM-generated tests track the official tests. Figure 2's confusion matrix pools roughly 20 training tasks from two benchmarks and shows a 16.7% false-pass rate on official-fail tasks, so an independent generated-versus-official verdict comparison on a much larger sample would directly bound how much of the reported gain is real.

The evolved profile-routing harnesses are effectively learned resource-allocation policies read off task text alone; the same loop could target other cost surfaces, such as latency under rate limits or energy per task, without changing the evolution machinery.

The test synthesizer turns evaluation into an authoring problem: any self-improving agent, not only harness optimization, could use frozen LLM-authored test suites as a stable selection signal, provided those suites are first validated against official outcomes.

Because cost is computed from token prices, the lite-medium-heavy ranking is price-sensitive; under a different pricing regime the same Pareto set could reorder, so the tiers would need re-selection rather than re-evolution.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper · Review history (2 versions)

The paper

Abstract

Harness development plays a crucial role in building capable AI agents. Recent approaches improve harnesses through recursive self-improvement (RSI), but their performance gains often come with higher execution costs. We introduce HarnessPro, a label-free method for evolving harnesses to improve task performance while reducing execution cost. In this setting, candidate harnesses are evaluated using LLM-generated tests without access to ground-truth solutions or official tests. Specialized agents diagnose execution trajectories, plan updates with different performance-cost priorities, and implement them as candidate harnesses. Rather than returning a single harness, HarnessPro uses normalized weighted squared-distance selection to provide cost-focused (lite), balanced (medium), and performance-focused (heavy) harnesses for different deployment budgets. Selected harnesses enter the next iteration, where execution feedback guides further updates. Our experiments cover Terminal-Bench 2, SWE-Bench Pro, and ProgramBench, using GPT-5.5 with Codex and GLM-5.2 with Claude Code. Across these benchmarks using GPT-5.5 with Codex, the heavy mode of HarnessPro achieves an average pass-rate gain of 2.0 percentage points and an average per-task cost reduction of 26.1% relative to RHO. This advantage also holds when using GLM-5.2 with Claude Code. Code is available at https://github.com/pzs19/HarnessPro

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-20

The authors' main claim, as the classifier picked it out 0.86

“Across these benchmarks using GPT-5.5 with Codex, the heavy mode of HarnessPro achieves an average pass-rate gain of 2.0 percentage points and an average per-task cost reduction of 26.1% relative to RHO.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.43 · human participants 0.03 · controlled comparison 0.92 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.97 · mentions code or data availability 0.98

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Harness development plays a crucial role in building capable AI agents. background 1.00 0.04 0.02 asserted 1.00
Recent approaches improve harnesses through recursive self-improvement (RSI), but their performance gains often come with higher execution costs. background 0.62 0.05 0.03 asserted 0.90
We introduce HarnessPro, a label-free method for evolving harnesses to improve task performance while reducing execution cost. contribution 1.00 0.87 0.26 asserted 0.99
In this setting, candidate harnesses are evaluated using LLM-generated tests without access to ground-truth solutions or official tests. method 0.99 0.60 0.03 asserted 1.00
Specialized agents diagnose execution trajectories, plan updates with different performance-cost priorities, and implement them as candidate harnesses. method 1.00 0.73 0.03 asserted 1.00
Rather than returning a single harness, HarnessPro uses normalized weighted squared-distance selection to provide cost-focused (lite), balanced (medium), and performance-focused (heavy) harnesses for different deployment budgets. method 0.86 0.92 0.16 asserted 1.00
Selected harnesses enter the next iteration, where execution feedback guides further updates. method 1.00 0.62 0.04 asserted 0.99
Our experiments cover Terminal-Bench 2, SWE-Bench Pro, and ProgramBench, using GPT-5.5 with Codex and GLM-5.2 with Claude Code. method 0.96 0.92 0.03 asserted 0.97
Across these benchmarks using GPT-5.5 with Codex, the heavy mode of HarnessPro achieves an average pass-rate gain of 2.0 percentage points and an average per-task cost reduction of 26.1% relative to RHO. result 1.00 0.96 0.41 asserted 1.00
This advantage also holds when using GLM-5.2 with Claude Code. result 0.99 0.89 0.10 asserted 0.99
Code is available at https://github.com/pzs19/HarnessPro contribution 0.65 0.47 0.02 asserted 0.51

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.99

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-20: 1 request, 8,150 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (4)
Figure 1 of the paper
Figure 1. Overview of HARNESSPRO. In each epoch, it analyzes rollout trajectories, generates candidate harnesses, and selects representatives based on performance and cost. 4.1 OVERVIEW To keep trajectory collection and analysis manageable, we begin by selecting a training coreset Dtrain. Following RHO (Pan et al., 2026a), we use the Determinantal Point Process (DPP) ker￾nel (Kulesza & Taskar, 2012) to balance task difficulty and diversity (Appendix A.1). Starting from an executable harness h0, we maintain a candidate set At of harnesses at epoch t, initialized as A0 = {h0}. In each epoch,…
Figure 2 of the paper
Figure 2. Confusion matrices pooled across training tasks from Terminal￾Bench 2 and SWE-Bench Pro. Test Synthesizer. During harness evolution, we follow RHO (Pan et al., 2026a), adopting a label-free setting without access to ground-truth solutions, official tests, or grader feedback. RHO relies on LLM self-preference to select harnesses. However, we find that LLM judges tend to be overly optimistic when assessing the execution tra￾jectories, often treating flawed implementations as suc￾cessful. We therefore introduce a dedicated test-synthesis agent to generate test cases for evaluation; its…
Figure 3 of the paper
Figure 3. Schematic of recursive har￾ness update, filtering, and selection.2 Here, h ′ ≺ h denotes Pareto dominance: h ′ is no worse than h on both metrics and strictly better on at least one. We select representatives using a normalized weighted squared-distance criterion (Marler & Arora, 2004) that accounts for actual differences in performance and cost. We first measure each candidate’s normalized deviations from the highest pass rate and lowest cost: dperf(h) = (Pmax − PRval(h))/(Pmax − Pmin), dcost(h) = (Cval(h) − Cmin)/(Cmax − Cmin). (8) Here, Pmin, Pmax and Cmin, Cmax are the extrema…
Figure 4 of the paper
Figure 4. Evolution dynamics of HARNESSPRO with GPT-5.5 and Codex. Dashed lines connect the harnesses retained in each epoch, and diamonds denote baselines. Intermediate-epoch test-set metrics are reported for visualization only and were not used for filtering or selection. Final-epoch points show means across repeated test runs

Figures are the authors', reproduced from the paper.

References (26)
  1. [1] Yehudai, Asaf and Eden, Lilach and Li, Alan and Uziel, Guy and Zhao, Yilun and Bar-Haim, Roy and Cohan, Arman and Shmueli-Scheuer, Michal , booktitle =. A Survey on Evaluation of. 2026 , doi = on Pith
  2. [2] Agents. 2026 , howpublished = on Pith
  3. [3] Grace, Mikaela and Hadfield, Jeremy and Olivares, Rodrigo and De Jonghe, Jiri , howpublished =. Demystifying evals for. 2026 , month = jan, note = on Pith
  4. [4] Zhang, Jenny and Hu, Shengran and Lu, Cong and Lange, Robert and Clune, Jeff , booktitle =. Darwin. 2026 , url = on Pith
  5. [5] Zhang, Jiayi and Xiang, Jinyu and Yu, Zhaoyang and Teng, Fengwei and Chen, Xionghui and Chen, Jiaqi and Zhuge, Mingchen and Cheng, Xin and Hong, Sirui and Wang, Jinlin and Zheng, Bingnan and Liu, Bang and Luo, Yuyu and Wu, Chenglin , booktitle =. 2025 , url = on Pith
  6. [6] Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses , author =. arXiv preprint arXiv:2604.25850 , year = on PithPith's reviewarXiv
  7. [7] Chen, Mengzhuo and Wang, Junjie and Liu, Zhe and Wang, Yawen and Zheng, Haiming and Wang, Qing , journal =. From Failed Trajectories to Reliable. 2026 , url = on Pith
  8. [8] Chen, Tingyang and Lu, Shuo and Zhao, Kang and Meng, Weicheng and Teng, Hanlin and Li, Tianhao and Li, Chao and Liu, Xule and Liang, Jian and Zhang, Zhizhong and Xie, Yuan and Qu, Heng and Shao, Kun and Luan, Jian , journal =. 2026 , url = on Pith
  9. [9] Evolving Agents in the Dark: Retrospective Harness Optimization via Self-Preference , author =. arXiv preprint arXiv:2606.05922 , year = on PithPith's reviewarXiv
  10. [10] Survey of multi-objective optimization methods for engineering , author =. Structural and Multidisciplinary Optimization , volume =. 2004 , doi = on Pith
  11. [11] Pan, Zhuoshi and Pei, Qizhi and Lu, Junru and Lin, Honglin and Zhao, H. Vicky and Yin, Di and Sun, Xing , booktitle =. 2026 , address = on Pith
  12. [12] Yang, Yifan and Gong, Ziyang and Huang, Weiquan and Yang, Qihao and Zhou, Ziwei and Huang, Zisu and Li, Yan and Gao, Xuemei and Dai, Qi and Liu, Bei and Qiu, Kai and Yang, Yuqing and Chen, Dongdong and Yang, Xue and Luo, Chong , journal =. 2026 , url = on Pith
  13. [13] Gong, Jingzhi and Gu, Ruizhen and Fei, Zhiwei and Cao, Yazhuo and Twist, Lukas and Geiger, Alina and Han, Shuo and Sobania, Dominik and Sarro, Federica and Zhang, Jie M. , journal =. 2026 , url = on Pith
  14. [14] Wang, Xuhui and Shu, Ruoqi and Dan, Chen and Xu, Tianhua and Luo, Mengxi and Mai, Yanming and Wan, Bo , journal =. 2026 , url = on Pith
  15. [15] Lei, Songxin and Ouyang, Kun and Ruan, Weilin and Wu, Yuqian and Guo, Zhijiang and Sun, Yushi and Tsung, Fugee , journal =. 2026 , url = on Pith
  16. [16] Budget-Aware Tool Use Enables Effective Agent Scaling , author =. arXiv preprint arXiv:2511.17006 , year = on PithPith's reviewarXiv
  17. [17] Gong, Jingzhi and Zhang, Jie M. and Jahangirova, Gunel and Huang, Dong and Mousavi, Mohammad Reza and Harman, Mark , journal =. 2026 , url = on Pith
  18. [18] Determinantal Point Processes for Machine Learning , author =. Foundations and Trends in Machine Learning , volume = on Pith
  19. [19] 2025 , month = apr, url = on Pith
  20. [20] Deng, Xiang and Da, Jeff and Pan, Edwin and He, Yannis Yiming and Ide, Charles and Garg, Kanak and Lauffer, Niklas and Park, Andrew and Rane, Chetan and Sampath, Karmini and Krishnan, Maya and Kundurthy, Srivatsa and Hendryx, Sean and Wang, Zifan and Zhang, Chen Bo Calvin and ... on Pith
show 6 more
  1. [21] Yang, John and Lieret, Kilian and Ma, Jeffrey and Thakkar, Parth and Pedchenko, Dmitrii and Sootla, Sten and McMilin, Emily and Yin, Pengcheng and Hou, Rui and Synnaeve, Gabriel and Yang, Diyi and Press, Ofir , journal =. 2026 , url = on Pith
  2. [22] Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory , author =. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) , publisher =. 2026 , doi = on Pith
  3. [23] Ouyang, Siru and Yan, Jun and Hsu, I-Hung and Chen, Yanfei and Jiang, Ke and Wang, Zifeng and Han, Rujun and Le, Long and Daruki, Samira and Tang, Xiangru and Tirumalashetty, Vishy and Lee, George and Rofouei, Mahsan and Lin, Hangfei and Han, Jiawei and Lee, Chen-Yu and Pfiste... on Pith
  4. [24] Sleep-time Compute: Beyond Inference Scaling at Test-time , author =. arXiv preprint arXiv:2504.13171 , year = on PithPith's reviewarXiv
  5. [25] Cost-of-Pass: An Economic Framework for Evaluating Language Models , author =. The Fourteenth International Conference on Learning Representations , year = on Pith
  6. [26] Szymkowiak, Patrick , year = on Pith
Cited by 3 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment