Software Engineering · arXiv:2610.28069 v1 · submitted 20 Oct 2026 · 43 references
Training the Whole Agent, Not Just the Model
The paper says
“Training coding agents typically updates the policy while keeping the harness fixed. During training, trajectories reveal flawed repair strategies and interaction problems that vary across tasks and change as the policy learns.”
more of the abstract
We introduce HarnessAdapt-RL, which trains the policy and adapts its harness during reinforcement learning. For flawed repair strategies, a frozen teacher generates task-specific mechanism-level skills. For interaction problems recurring across tasks, a separate teacher model evolves tools and hooks, keeping each change only if it improves the observed pass rate or the observed trainable rate without lowering either. The policy trains with the adapted harness, and its new trajectories guide further adaptation. With a 9B base model, HarnessAdapt-RL improves pass@1 over GRPO by 9.8, 12.6, and 6.4 percentage points on SWE-bench Verified, Multilingual, and Pro, and outperforms GRPO with the human-designed OpenHands harness. Gains from mechanism-level skills persist after their removal at evaluation. Re-evolving tools and hooks as the policy changes outperforms freezing early versions. Removing them at deployment, in contrast, costs up to 7.7 percentage points. Training therefore produces a model-harness pair: the trained policy and the evolved tools and hooks, with skills removed at deployment.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 21 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. A careful, honest co-adaptation study — direction well supported, headline magnitudes disclosed single draws, worth serious refereeing. This line is wrong
2 major objections · 6 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 43 references say.
The paper says, at Section 3.3, Appendix B.2, Table 15
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.85):
“The reported confidence intervals characterize task-level variation for these checkpoints; they do not quantify variability across training seeds or independent harness-search runs.”
Pith says: Objection 1 of 2 · major
The acceptance gate is too weak for single-run evidence to support the method-level headline. Each comparison uses 184 rollouts per side (Appendix B.2), so at the pass rates observed in Figure 2 and Table 15 (roughly 30-50%) the standard error of a rate difference is about 5 percentage points.
more of objection 1
The six accepted proposals in Table 15 have relative pass-rate gains of 3.5-13.7%, which at a 40% baseline correspond to absolute gains of roughly 1.4-5.5 percentage points, and the relative trainable-rate gains of 3.2-7.8% are similar in magnitude; most accepted candidates therefore sit within about one standard error of zero. Because the rule keeps a candidate whenever at least one observed rate rises and neither falls, a null candidate is accepted with substantial probability from noise alone, and the acceptance record cannot by itself separate genuine harness improvements from favorable draws. As the authors acknowledge in the Limitations, every configuration is trained once (Appendix B.4) and the Table 10 intervals characterize only task-level evaluation noise, not training-seed or search variance ([b265], [b269]). The concern is partially mitigated by the held-out evaluation benchmarks, by the cross-policy benefit of H2 in Table 2, and by the Frozen-H2 reuse result, but none of these bounds the distribution of harnesses the noisy gate would accept. What remains unresolved is that the headline deltas of Table 1 may be partly a favorable search draw. A concrete fix within the manuscript's scope is a sensitivity analysis showing that the six acceptance decisions in Table 15 survive +-1-SE perturbations of the observed rates, plus at least two or three independent re-runs of the search-and-train pipeline for the central HarnessAdapt-RL versus GRPO comparison.
The paper says, at Section 4.6, Appendix B.1
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.74):
“These identifiers do not pin immutable teacher snapshots, and the historical CLI package versions were not preserved.”
Pith says: Objection 2 of 2 · major
The reuse experiment and the teacher dependence make the method-level claims conditional on components that the paper cannot regenerate or audit. Frozen-H2+Mechanism retrains a fresh policy but reuses the same realized H2, so it shows that this particular harness transfers to a new training run while bounding nothing about the harnesses the Section 3.3 gate would accept; the re-evolution comparison (HarnessAdapt-RL versus H1+Mechanism) and the pre-training-search comparison (versus Pre-Evolved) are likewise single draws from the evolution process.
more of objection 2
The paper states at [b407] that the teacher identifiers do not pin immutable snapshots and that historical CLI package versions were not preserved; because every skill and every tool-and-hook proposal originates from GLM-5.3-Flash and GPT-5.6 Sol, the method cannot be re-executed or independently audited, even though the released weights and H2 make the realized artifact reproducible. I ask the authors to state explicitly in the abstract or conclusion that the method-level claims are conditional on the specific teacher snapshots used, and to provide at least one additional search realization, even at reduced budget, demonstrating that accepted bundles and the headline deltas are not the artifact of a single favorable draw.
6 minor objections
The paper says, at Abstract; Sections 4.2-4.3; Table 10
Pith says: minor objection 1 of 6
Several per-benchmark statements are stronger than the intervals support: the Multilingual interval for HarnessAdapt-RL versus OpenHands-GRPO is [-0.3, +8.3] (Table 10), and only the Multilingual interval for the re-evolution gain over H1+Mechanism excludes zero ([+1.3, +7.9]). These claims are supported at the pooled level (+6.3 [+4.3, +8.0] and +2.6 [+0.9, +4.3]); please qualify the prose with 'on average' or cite the pooled intervals where individual benchmark intervals include zero.
The paper says, at Appendix B.3, Table 6
Pith says: minor objection 2 of 6
In Table 6 the entry '4n a' for skill-variant preparation rollouts should read '4 x n_a'; as printed it is easy to misread as an unlabeled column of raw numbers.
show 4 more
The paper says, at Section 3.3 and Appendix D.3
Pith says: minor objection 3 of 6
The selection metric is called 'trainable rate' in Section 3.3 and Appendix B.2 but 'trainable-rollout ratio' in the harness-proposal prompt (Appendix D.3, part 1); please standardize the terminology so the teacher prompt and the method text refer to the same quantity.
The paper says, at Section 4.1 and Appendix B.1
Pith says: minor objection 4 of 6
The GRPO baseline's eight rollouts per task versus four for the other configurations is a conservative choice, but no sentence explains this intention; one sentence of rationale would prevent readers from misreading the asymmetry as favoring HarnessAdapt-RL.
The paper says, at Appendix C.8, Eq. (2)
Pith says: minor objection 5 of 6
The relative-change metric in Eq. (2) compares each candidate with a different current-harness baseline per round, and training also changes the policy between rounds; the paper warns that changes are not cumulative, but reporting the absolute pass and trainable rates per round in Table 15 would let readers convert the relative gains into percentage points.
The paper says, at Appendix C.2, Table 8
Pith says: minor objection 6 of 6
The length comparison in Table 8 mixes differences in outcome composition with differences in redundancy, which the paper acknowledges; conditioning the token counts on matched tasks would strengthen the claim that near-answer hints shorten trajectories rather than simply succeed more often.
What would settle it
Re-run the full HarnessAdapt-RL loop with three independent seeds, varying only rollout-sampling and teacher-call randomness while keeping the shared task pool, task order, and budgets fixed, and report the spread of final pass@1 (Table 1) and of the fixed-policy H2-vs-H0 deltas (Table 2). If any headline delta moves by more than ~3 points across seeds, the magnitudes are single-draw artifacts and must carry the seed variance; if all stay within ~2 points, the concern is retired. A cheaper first step: apply a paired test over the 46 tasks to each of the nine selection comparisons in Table 15 and count how many of the six accepted proposals show a significant improvement at 5% with a light multiple-comparison correction; if most acceptances are within noise, the Section 4.4 evolution narrative and the re-evolution benefit should be downgraded to single-draw evidence pending the seed runs.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 21 Oct 2026
- Reader holds up, with conditions
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee major revision: 2 major objections, 6 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Two things you should know. This is one of the more careful method papers I've seen in the SWE-agent RL space: the design controls for the right confounds and the authors disclose every source of variance they can't control. And the headline numbers, while directionally well supported, are single draws from a noisy, teacher-mediated pipeline whose limits the authors themselves spell out in Limitations.
What's actually new: co-adapting the GRPO-trained policy with the harness, under an acceptance rule that tracks the trainable rate — the share of rollouts usable for policy updates — alongside pass rate. The trainable-rate criterion is the genuinely reusable idea: all-success groups inflate pass rate while producing no learning signal, and the rule captures that. The three-way skill comparison is also better than the usual "skills help" story. Near-answer hints raise training success but hurt final performance, with a plausible mechanism (fewer mixed reward groups: 39.0% vs 46.5%), while mechanism-level skills survive removal.
Experimental hygiene is above the field's norm: shared task pool and order, dedup versus the eval benchmarks, disclosed rollout budgets, a GRPO baseline given twice the training rollouts, paired bootstrap CIs, and a checkpoint chosen before evaluation. The fixed-policy H0-vs-H2 swap (Table 2) cleanly separates harness value from policy training: the evolved harness is worth 4.8–7.7 points at deployment, and training with it matters more than adding it after the fact.
Soft spots, in proportion. The acceptance gate is genuinely weak as a statistical procedure: 184 rollouts per side at ~40% pass rates gives ~5-point standard errors, so most accepted proposals sit within about one standard error of zero, and the final H2 is one realized draw. That caps how precisely you can quote the +9.8/+12.6/+6.4 magnitudes. But the authors say as much in Limitations, and two results in the paper partially answer the worry: a fresh policy trained on the realized H2 matches the original within 1.0 point (noise-driven acceptance would not transfer that way), and an independently searched Pre-Evolved harness also helps. Direction is supported; precision is not. Single training run per configuration, unpinned GLM-5.3-Flash and GPT-5.6 Sol teachers, and one task family with one 9B model are all stated in the text; the released H2 and weights keep downstream evaluation reproducible.
This paper deserves a serious referee. The main push should be repeated training runs or fresh search draws; the core design and the central contrasts are solid.
The strongest objection
The central claim requires that the reported gains are stable properties of the method, not favorable draws, and that condition is least secure. First, the acceptance gate (Section 3.3, Appendix B.2) keeps a proposal if one observed rate improves and neither decreases, measured on 184 rollouts per side with no significance test, as the authors state. At pass rates near 30-50%, the per-comparison standard error is roughly 4-5 percentage points, so most of the six accepted proposals in Table 15 (relative pass-rate gains of 3.5-13.7%) sit within about one standard error of zero; a substantial share of acceptances could be noise-driven. Second, every configuration is trained once (Limitations; sentence [b265]), and the Appendix C.4 intervals cover only task-level evaluation noise, which the authors say does not estimate variation across training seeds or harness-search runs ([b269]). The headline deltas (9.8/12.6/6.4), the re-evolution benefit (2.2/4.6/2.0; pooled +2.6 [+0.9, +4.3] and individually significant only on Multilingual), and the 7.7-point deployment cost are therefore single-run observations. Third, the only cross-realization check, Frozen-H2+Mechanism (Section 4.6), retrains the policy but reuses the same realized H2, so it confirms that this particular H2 helps a fresh policy but bounds nothing about the distribution of harnesses the noisy gate would accept. The unpinned teachers (Appendix B.1, [b407]) compound the problem: because H1 and H2 cannot be regenerated, search variance cannot be re-estimated independently. This is a robustness concern, not an internal inconsistency, and the authors acknowledge each component; what remains unresolved is the combined magnitude.
I read HarnessAdapt-RL as a controlled empirical study whose central claim, that jointly training policy and harness yields a deployable pair beating fixed-harness RL, rests on the Section 4.2/Table 1 deltas and the Section 4.5 corollary. The paper earns real credit: shared task pool and order, deduplication against evaluation benchmarks, disclosed rollout budgets, paired bootstrap intervals, honest Limitations (single run, non-significant selection, unpinned teachers), and released weights, H2 code, and data with no teacher needed for inference. I considered the proprietary unpinned teachers (Appendix B.1) as the prime suspect, as the reader did, and judge them secondary: the acceptance gate is empirical, and the measured end states are directly checkable from released artifacts; teacher pinning threatens re-running the loop, not the validity of the measured results. I also considered whether H2's eval-time test-running inflates the headline comparison; the paper's own cross-evaluation (Table 2) isolates a generic H2-at-eval benefit of 2.0-5.3 points for a policy never trained with H2, and reports a pooled co-training interaction of +2.5 [+1.0, +4.2], so the harness effect's existence is substantially controlled. What is least secure is stability: the Section 3.3 gate operates within about one standard error of the observed rates, every configuration is a single run, the CIs exclude search and seed variance, and the Frozen-H2 reuse result reuses the same realized H2, so nothing in the paper bounds the variance of the evolution pipeline. If that variance is large, the distinctive results (re-evolution benefit, deployment cost, headline deltas) are single-draw facts rather than method-level facts. The authors acknowledge each ingredient; the missing piece is the magnitude, which seed runs or a significance-aware re-analysis of Table 15 would settle. Because the claim is not contradicted internally and the uncertainty is fully disclosed, the reader's CONDITIONAL verdict stands, now sharpened to an explicit condition: quantify pipeline variance before the magnitudes are taken at face value.
The referee's report
The paper's conceptual claim is important and, if it holds across repeated runs, would justify a new default for agent post-training: instead of fixing the harness and training the policy, adapt both, and deploy the pair. The experimental design is careful in ways that strengthen the empirical core: a shared, deduplicated task pool with fixed task order; a conservative GRPO baseline that receives twice the policy-training rollouts of other configurations; paired task-level bootstrap confidence intervals following Miller [41]; fully disclosed rollout and teacher budgets (Appendix C.7); and a Limitations section that explicitly concedes single-run training, the non-significance-test acceptance rule, and unpinned teacher snapshots. Two diagnostics are particularly well designed: the H0-versus-H2 swap in Table 2, which holds policy weights fixed and isolates harness value, and the Frozen-H2 reuse test in Section 4.6, which shows the realized harness transfers to a fresh training run. The skill-design comparison (mechanism versus hint versus bank), which is independent of the harness-evolution loop, is cleanly controlled and supports the mechanism-level-skills claim.
Pith's reading of the paper (a machine's summary, not the authors')
Most agent training fixes the harness — the tools, runtime hooks, and textual guidance wrapped around the model — and updates only the policy weights. This paper argues that trajectories produced during reinforcement learning are themselves evidence about the harness, exposing flawed repair strategies and recurring interaction failures that vary by task and change as the policy learns, so the harness should adapt during training. HarnessAdapt-RL does this with two teacher models: one writes task-specific 'mechanism-level' skills that explain when and why a repair strategy applies without revealing the answer, and another proposes new tools and hooks, kept only if they improve the pass rate or the share of rollouts usable for policy updates without lowering either. With a 9B base model, the trained agent beats plain GRPO (the standard group-relative policy-gradient RL baseline) by 9.8, 12.6, and 6.4 percentage points in pass@1 — tasks solved on the first attempt — on three SWE-bench software-repair suites, and beats GRPO trained on a human-designed harness; the skill gains survive removal at deployment, while the evolved tools and hooks carry up to 7.7 points. If the claim holds, agent pipelines should train, ship, and evaluate a model together with its evolved runtime rather than weights alone.
What carries the argument. The load-bearing object is the harness pair $(H, s)$: $H = (H_0, T, K)$ bundles the base Bash runtime $H_0$, a set of added tools $T$, and runtime hooks $K$, while $s$ is the optional skill guidance the policy sees during training. Two teacher loops operate on it. A frozen skill teacher compares two trajectories per task and writes a mechanism-level skill — the 'when and why' of a repair strategy with repository paths, symbols, and code removed — which is shown during training and dropped at deployment. A separate harness teacher scans trajectories across tasks, diagnoses recurring interaction problems, and implements concrete tool and hook proposals; each candidate runs against the current harness on 184 fresh rollouts per side and is kept only if the observed pass rate or the observed trainable rate strictly improves and neither falls. The trainable rate is the second load-bearing idea: it counts rollouts whose task group contains both successes and failures, because GRPO's group-relative advantage learns only from such contrasts. That metric lets harness selection optimise for learning signal as well as task success, and it ties the skill result to the evolution result: mechanism-level skills yield 46.5% mixed-outcome groups versus 39.0% for near-answer hints, which instead produce more all-success groups that give GRPO no reward contrast.
Core claim. The paper's central claim, on its own terms, is that a coding agent is a model–harness pair and that reinforcement learning should train both halves together: HarnessAdapt-RL interleaves GRPO policy updates with harness adaptation driven by the current policy's trajectories. The harness carries two kinds of support with different roles at deployment. Mechanism-level skills are task-specific, teacher-written explanations of when and why a repair strategy applies, stripped of file paths, symbols, and code; after the skills are removed, models trained with them reach 38.3% average pass@1, beating a general skill bank (34.1%) and near-answer hints (32.7%), because hints shorten trajectories by 40.8% and leave the policy unable to do the skipped work on its own. Tools and hooks are evolved by a second teacher that diagnoses recurring interaction failures across tasks, and a proposal is accepted only if it improves the observed pass rate or the observed trainable rate without lowering either. The anchor numbers: average pass@1 rises from 35.3% (GRPO) to 44.9%; gains over GRPO are 9.8, 12.6, and 6.4 points on SWE-bench Verified, Multilingual, and Pro; re-evolving the harness in the second epoch adds 2.2–4.6 points over freezing the first-epoch version; and with the trained policy weights held fixed, replacing the evolved harness with the base harness costs up to 7.7 points, while the skills can be discarded without losing their gains.
The premise it rests on. The load-bearing premise is that the two proprietary teacher models — GLM-5.3-Flash for skills and GPT-5.6 Sol for tools and hooks — can turn trajectory evidence into working harness code, since the paper notes that their snapshots are not pinned; if that fails, the adaptation loop collapses and all downstream gains disappear, and the acceptance rule's reliance on 184-rollout comparisons without a significance test (an acknowledged limitation) adds further noise to every selection.
If the paper is right (Pith's inferences, not the authors')
The deployment artifact is the pair (trained policy, evolved harness): with policy weights frozen, replacing the evolved tools and hooks with the base harness costs up to 7.7 percentage points, so releases should bundle and benchmark the pair.
Harness search must run on a learning policy's trajectories rather than a fixed base policy: Pre-Evolved, which searches with the base model frozen, trails HarnessAdapt-RL by 2.4 to 4.3 points.
Harness evolution should continue as the policy changes: re-evolving $H_1$ into $H_2$ in the second epoch beats keeping $H_1$ fixed by 2.2 to 4.6 points, motivated by the observed decline in verification behaviour in later trajectories.
Training guidance should be judged after removal: near-answer hints raise training success but end 2.6 points below GRPO once removed, while mechanism-level skills are the only variant that beats GRPO on all three benchmarks without them.
A discovered harness is reusable: training a fresh policy from the same base model on the frozen $H_2$ stays within 1.0 point of the full method, making harness discovery a one-time cost per domain.
Editorial inference: the trainable-rate criterion is a portable diagnostic for sparse-reward RL — the share of mixed-outcome groups measures how much learning signal a data pipeline is producing, so it could serve as an early-stop or guidance-design signal in any GRPO-style setup without a full evaluation.
Editorial inference: the near-answer-hint result suggests a competence-erosion mechanism, where training aids that substitute for reasoning steps get internalised as dependencies; this is testable outside coding by comparing post-removal performance after concrete-hint versus mechanism-only guidance in domains such as mathematical reasoning.
Editorial inference: because the authors flag that teacher snapshots are not pinned, replacing the proprietary teachers with open-weights models would separate the co-adaptation principle from the specific teachers' capabilities and is the natural first replication.
Editorial inference: if the model–harness pair result holds, standardised coding-agent evaluation should pin the harness as part of the protocol, since the paper's own table shows that harness choice alone moves pass@1 by 2.0 to 7.7 points with identical weights.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Training coding agents typically updates the policy while keeping the harness fixed. During training, trajectories reveal flawed repair strategies and interaction problems that vary across tasks and change as the policy learns. We introduce HarnessAdapt-RL, which trains the policy and adapts its harness during reinforcement learning. For flawed repair strategies, a frozen teacher generates task-specific mechanism-level skills. For interaction problems recurring across tasks, a separate teacher model evolves tools and hooks, keeping each change only if it improves the observed pass rate or the observed trainable rate without lowering either. The policy trains with the adapted harness, and its new trajectories guide further adaptation. With a 9B base model, HarnessAdapt-RL improves pass@1 over GRPO by 9.8, 12.6, and 6.4 percentage points on SWE-bench Verified, Multilingual, and Pro, and outperforms GRPO with the human-designed OpenHands harness. Gains from mechanism-level skills persist after their removal at evaluation. Re-evolving tools and hooks as the policy changes outperforms freezing early versions. Removing them at deployment, in contrast, costs up to 7.7 percentage points. Training therefore produces a model-harness pair: the trained policy and the evolved tools and hooks, with skills removed at deployment.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-21
The authors' main claim, as the classifier picked it out 0.93
“Training therefore produces a model-harness pair: the trained policy and the evolved tools and hooks, with skills removed at deployment.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.08 · states limitation 0.60 · human participants 0.04 · controlled comparison 0.96 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.98 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Training coding agents typically updates the policy while keeping the harness fixed. | background 0.99 | 0.06 | 0.02 | asserted 1.00 |
| During training, trajectories reveal flawed repair strategies and interaction problems that vary across tasks and change as the policy learns. | background 0.78 | 0.55 | 0.03 | asserted 1.00 |
| We introduce HarnessAdapt-RL, which trains the policy and adapts its harness during reinforcement learning. | contribution 0.98 | 0.92 | 0.16 | asserted 0.99 |
| For flawed repair strategies, a frozen teacher generates task-specific mechanism-level skills. | method 0.99 | 0.84 | 0.05 | asserted 0.99 |
| For interaction problems recurring across tasks, a separate teacher model evolves tools and hooks, keeping each change only if it improves the observed pass rate or the observed trainable rate without lowering either. | method 0.99 | 0.90 | 0.08 | asserted 0.99 |
| The policy trains with the adapted harness, and its new trajectories guide further adaptation. | method 1.00 | 0.80 | 0.07 | asserted 1.00 |
| With a 9B base model, HarnessAdapt-RL improves pass@1 over GRPO by 9.8, 12.6, and 6.4 percentage points on SWE-bench Verified, Multilingual, and Pro, and outperforms GRPO with the human-designed OpenHands harness. | result 1.00 | 0.95 | 0.26 | asserted 1.00 |
| Gains from mechanism-level skills persist after their removal at evaluation. | result 0.99 | 0.94 | 0.11 | asserted 1.00 |
| Re-evolving tools and hooks as the policy changes outperforms freezing early versions. | result 0.99 | 0.92 | 0.10 | asserted 1.00 |
| Removing them at deployment, in contrast, costs up to 7.7 percentage points. | result 0.99 | 0.92 | 0.08 | asserted 0.97 |
| Training therefore produces a model-harness pair: the trained policy and the evolved tools and hooks, with skills removed at deployment. | interpretation 0.68 | 0.86 | 0.70 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.98
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-21: 1 request, 8,057 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (21)
Figures are the authors', reproduced from the paper.
References (43)
- [41] Evan Miller. Adding error bars to evals: A statistical approach to language model evaluations.arXiv preprint arXiv:2411.00640, 2024. URLhttps://arxiv.org/abs/2411.00640. 13 Appendix A Data and Harness Specifications A.1 Task Pool and Assisted Subset Repository Tasks Share Repo... on PithPith's reviewarXiv
- [1] Naman Jain, Jaskirat Singh, Manish Shetty, Tianjun Zhang, Liang Zheng, Koushik Sen, and Ion Stoica. R2E- Gym: Procedural environment generation and hybrid verifiers for scaling open-weights SWE agents. InSecond Conference on Language Modeling, 2025. URLhttps://openreview.net/f... on Pith
- [2] John Yang, Kilian Lieret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. SWE-smith: Scaling data for software engineering agents. In Advances in Neural Information Processing Systems, 2025. URLhttps://... on Pith
- [3] Zhiheng Xi, Jixuan Huang, Chenyang Liao, Baodai Huang, Jiaqi Liu, Honglin Guo, Yajie Yang, Rui Zheng, Junjie Ye, Jiazheng Zhang, Wenxiang Chen, Wei He, Yiwen Ding, Guanyu Li, Zehui Chen, Zhengyin Du, Xuesong Yao, YufeiXu, JiecaoChen, TaoGui, ZuxuanWu, QiZhang, XuanjingHuang, a... on Pith
- [4] Zhengyu Chen, Teng Xiao, Huaisheng Zhu, Yige Yuan, Luan Zhang, and Jingang Wang. Co-Harness: Co- evolving harnesses and model weights for LLM agents.arXiv preprint arXiv:2607.22688, 2026. URLhttps: //arxiv.org/abs/2607.22688 on PithPith's reviewarXiv
- [5] Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta-Harness: End-to-end optimization of model harnesses.arXiv preprint arXiv:2603.28052, 2026. URLhttps://arxiv.org/ abs/2603.28052 on PithPith's reviewarXiv
- [6] Wen Jiang, Mingmin Chu, Yimeng Tian, Qianxin Zhang, Haofei Yang, Rui Yang, Yang Liu, Tao Lv, and Fangming Li. HarnessEvolve: Learning from reference trajectories for reliable agent self-evolution.arXiv preprint arXiv:2609.00829, 2026. URLhttps://arxiv.org/abs/2609.00829 on PithPith's reviewarXiv
- [8] Chuan Xiao, Zhengbo Jiao, Shaobo Wang, Wei Wang, Bing Zhao, Hu Wei, Linfeng Zhang, and Lin Qu. Socratic- SWE: Self-evolving coding agents via trace-derived agent skills.arXiv preprint arXiv:2606.07412, 2026. URL https://arxiv.org/abs/2606.07412 on PithPith's reviewarXiv
- [9] Shengjie Wang, Guanghe Li, Zonghan Yang, and Yang Gao. Hindsight hint distillation: Scaffolded reasoning for SWE agents from CoT-free answers.arXiv preprint arXiv:2605.11556, 2026. URLhttps://arxiv.org/abs/2605 .11556 on PithPith's reviewarXiv
- [10] Yipeng Shi, Zhipeng Ma, Yue Wang, Qitai Tan, Yang Li, Peng Chen, and Zhengzhou Zhu. PATS: Policy- aware training scaffolding for agentic reinforcement learning.arXiv preprint arXiv:2607.21419, 2026. URL https://arxiv.org/abs/2607.21419. 10 on PithPith's reviewarXiv
- [11] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024 on PithPith's reviewarXiv
- [12] Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Carlos E. Jimenez, John Yang, Leyton Ho, Tejal Patwardhan, Kevin Liu, and Aleksander Madry. Introducing SWE-bench verified. OpenAI blog,... on Pith
- [14] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations, on Pith
- [15] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. InAdvances in Neural Information Processing Systems, 2024. URLhttps://arxiv.org/abs/2405.15793 on PithPith's reviewarXiv
- [16] Tingxu Han, Yi Zhang, Wei Song, Chunrong Fang, Zhenyu Chen, Youcheng Sun, and Lijie Hu. SWE-Skills- Bench: Do agent skills actually help in real-world software engineering?arXiv preprint arXiv:2603.15401, 2026. URLhttps://arxiv.org/abs/2603.15401 on PithPith's reviewarXiv
- [17] Mirac Suzgun, Mert Yuksekgonul, Federico Bianchi, Dan Jurafsky, and James Zou. Dynamic cheatsheet: Test- time learning with adaptive memory.arXiv preprint arXiv:2504.07952, 2025. URLhttps://arxiv.org/abs/25 04.07952 on PithPith's reviewarXiv
- [18] Peng Xia, Jianwen Chen, Hanyang Wang, Jiaqi Liu, Kaide Zeng, Yu Wang, Siwei Han, Yiyang Zhou, Xujiang Zhao, Haifeng Chen, Zeyu Zheng, Cihang Xie, and Huaxiu Yao. SkillRL: Evolving agents via recursive skill- augmented reinforcement learning.arXiv preprint arXiv:2602.08234, 202... on PithPith's reviewarXiv
- [19] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Juncai Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Ru Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jia... on Pith
- [20] Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URLhttps://qwen.ai/blog?id=qwen 3.5 on Pith
- [21] Z.ai. GLM-5.3-Flash/FlashX: Model overview and api access. Z.AI Developer Documentation, 2026. URL https://docs.z.ai/guides/vlm/glm-5.3-flash. Accessed October 20, 2026 on Pith
show 23 more
- [22] OpenAI. GPT-5.6 Sol model. OpenAI API documentation, 2026. URLhttps://developers.openai.com/api/do cs/models/gpt-5.6-sol. Accessed October 20, 2026 on Pith
- [23] OpenAI. Codex is now generally available. OpenAI, October 2025. URLhttps://openai.com/index/codex-now -generally-available/ on Pith
- [24] Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brenna... on PithPith's reviewarXiv
- [25] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? InInternational Conference on Learning Representations, 2024 on Pith
- [26] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models.Transactions on Machine Learning Research, 2024. URLhttps://openreview.net/forum?id=ehfRiF0R3a on Pith
- [27] Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsidhar Kamanuru, Jay Rain- ton, Chen Wu, Mengmeng Ji, Hanchen Li, Urmish Thakker, James Zou, and Kunle Olukotun. Agentic context engineering: Evolving contexts for self-improving language models. InInter... on PithPith's reviewarXiv
- [28] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. InAdvances in Neural Information Processing Systems, 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/1b44b878b... on Pith
- [29] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, ShrimaiPrabhumoye, YimingYang, ShashankGupta, BodhisattwaPrasadMajumder, KatherineHermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-Refine: Iterative refi... on Pith
- [30] Wenbo Pan, Shujie Liu, Chin-Yew Lin, Jingying Zeng, Xianfeng Tang, Xiangyang Zhou, Yan Lu, and Xiao- hua Jia. Evolving agents in the dark: Retrospective harness optimization via self-preference.arXiv preprint arXiv:2606.05922, 2026 on PithPith's reviewarXiv
- [31] Hyunin Lee, Jinglue Xu, Jeffrey Seely, Donghyun Lee, Matei Zaharia, and Yujin Tang. Recursive harness self- improvement.arXiv preprint arXiv:2607.15524, 2026. URLhttps://arxiv.org/abs/2607.15524 on PithPith's reviewarXiv
- [32] Shidong Yang, Ziyu Ma, Tongwen Huang, Yiming Hu, Yong Wang, and Xiangxiang Chu. CoEvolve: Training LLM agents via agent-data mutual evolution. InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 23015–23036. ... on Pith
- [33] Xinshun Feng, Xinhao Song, Lijun Li, Gongshen Liu, and Jing Shao. SEARL: Joint optimization of policy and tool graph memory for self-evolving agents. InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 24518–... on Pith
- [34] Huatong Song, Fei Bai, Ming Yang, Renyuan Li, Jia Deng, Jujie He, Zhange Zhang, Daixuan Cheng, Yan Xing, Qi Yun, Xuxing Chen, Danyang Li, Feng Chang, Chuan Hao, Ran Tao, Jian Yang, Bryan Dai, Wayne Xin Zhao, Mingjie Tang, and Ji-Rong Wen. ClawGym II: Exploring black-box RL on ... on PithPith's reviewarXiv
- [35] Zonghan Yang, Shengjie Wang, Kelin Fu, Wenyang He, Weimin Xiong, Yibo Liu, Yibo Miao, Bofei Gao, Yejie Wang, Yingwei Ma, Yanhao Li, Yue Liu, Zhenxing Hu, Kaitai Zhang, Shuyi Wang, Huarong Chen, Flood Sung, Yang Liu, Yang Gao, Zhilin Yang, and Tianyu Liu. Kimi-dev: Agentless tr... on PitharXiv
- [36] Shiyi Cao, Dacheng Li, Fangzhou Zhao, Shuo Yuan, Sumanth R. Hegde, Connor Chen, Charlie Ruan, Tyler Griggs, Shu Liu, Eric Tang, Richard Liaw, Philipp Moritz, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. SkyRL-Agent: Efficient RL training for multi-turn LLM agent.arXiv pr... on PitharXiv
- [37] Sunghwan Kim, Junhee Cho, Beong-woo Kwak, Taeyoon Kwon, Liang Wang, Nan Yang, Xingxing Zhang, Furu Wei, and Jinyoung Yeo. On training large language models for long-horizon tasks: An empirical study of horizon length.arXiv preprint arXiv:2605.02572, 2026. URLhttps://arxiv.org/... on PithPith's reviewarXiv
- [38] Zilin Zhu, Chengxing Xie, Xin Lv, and slime Contributors. slime: An LLM post-training framework for RL scaling. GitHub repository, 2025. URLhttps://github.com/THUDM/slime. Accessed September 18, 2026. 12 on Pith
- [39] OpenCode Contributors. OpenCode: The open source coding agent. GitHub repository, 2026. URLhttps: //github.com/anomalyco/opencode. Accessed September 18, 2026 on Pith
- [40] Michael Luo, Naman Jain, Jaskirat Singh, Sijun Tan, Ameen Patel, Qingyang Wu, Alpay Ariyak, Colin Cai, Tarun Venkat, Shang Zhu, Ben Athiwaratkun, Manan Roongta, Ce Zhang, Li Erran Li, Raluca Ada Popa, Koushik Sen, and Ion Stoica. DeepSWE: Training a fully open-sourced, state-o... on Pith
- [200] Reading and diff outputs also have character limits.run_checkallows at most 45 seconds, preserves pipeline failures, and reports runner problems, zero-test signals, and timeouts. A zero exit code alone does not establish that the task requirements were tested. InH1,prepare_pat... on Pith
- [2023] URLhttps://arxiv.org/abs/2210.03629 on PithPith's reviewarXiv
- [2025] URLhttps://arxiv.org/abs/2509.16941 on PithPith's reviewarXiv
- [2026] URLhttps://arxiv.org/abs/2607.26598 on PithPith's reviewarXiv
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment