Pith.

Artificial Intelligence · arXiv:2608.13951 v1 · submitted 14 Aug 2026 · 22 references

HELIX: Model-Harness Co-evolution for Recursive Self-Improvement

Tianyu Fan, Chao Huang

The paper says

“Scaling agent capability has largely focused on improving the model, yet an interactive agent acts through a runtime harness that mediates context, tools, control flow, and stopping.”

more of the abstract

The harness shapes both what a model can accomplish and the trajectories from which it learns. This coupling motivates model-harness co-evolution for recursive self-improvement: build harnesses for a fixed model, update the model from verified sibling trajectories, and rebuild the harnesses as model capabilities change. Realizing this loop requires a controlled way to evolve harnesses while preserving intervention identity and effect. We present HELIX, a source-traceable substrate for harness evolution. HELIX decomposes agent systems into typed ports, reusable atoms, recipes, product shells, and runtime policies. It makes interventions explicit and auditable while retaining trajectories, test outcomes, and provenance. Harness evolution thus serves two linked roles: improving fixed-model execution and producing matched successes, regressions, near misses, and alternative solutions as data for subsequent model improvement. We evaluate HELIX in one evolution round on code repair. A 65-candidate portfolio discovers a fixed harness that improves task coverage by 4.0% over Pi, while the full portfolio exposes up to 58.0% more verified coverage through complementary sibling behavior. Selected candidates are assessed with repeated runs and the SWE-bench evaluator. A 200-slot sibling slice yields 438 verified SFT, critic, filter, and preference records. These results show how harness, model, and data form a feedback system: harness evolution expands current capability and creates learning signal for the next model; model updates motivate the next round of harness evolution. HELIX provides an auditable interface for studying this recursive process. Code is available at https://github.com/HKUDS/HELIX.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 27 Aug 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. Worth reading for the auditable harness substrate and honest limitations section, but the headline execution gain is a two-task selected maximum and the recursive loop is not yet demonstrated. This line is wrong

2 major objections · 5 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 22 references say.

The paper says, at §5.4 and §6.1

The review did not quote this passage.

Pith says: Objection 1 of 2 · major

The reported 4.0% fixed-harness improvement (best candidate 52/100 vs Pi 50/100) is not established as a real harness effect. The 65-candidate matrix runs each candidate once on each of the same 100 tasks used both to select the winner and to report its score, with the model configured as deterministic=false; the 52/100 figure is the maximum over the candidate set, so it is a selected maximum rather than an unbiased estimate.

more of objection 1

For a paired two-task margin, even the most favorable split of discordant pairs gives a one-sided McNemar p-value of about 0.25, so the difference is within ordinary single-attempt noise. The repeated-run rows in Table 4 do not repair this: they use the same task set, a subset chosen for mechanism analysis rather than by performance, and no held-out split. Section 9.3 explicitly concedes that rankings under different attempt budgets are not directly comparable and that future work should use separate tasks for evolution and final evaluation. Finding 1 should therefore be rephrased as preliminary, or the paper should add a protocol with separate selection and evaluation tasks, multiple attempts per candidate, and confidence intervals.

The paper says, at §6.1 and Tables 4–5

The review did not quote this passage.

Pith says: Objection 2 of 2 · major

The selected-member follow-ups, as reported, do not provide a valid test of whether evolution selects stronger harnesses. The LCB repeated-run gap (75/100 vs 72/100) is a three-task paired difference and the SWE-bench gap (46/55 vs 44/55) is a two-instance paired difference, with no variance estimate or significance test; the successful-slot differences (549 vs 529 on 1,000 slots, and 86 vs 81 on 110 slots) are likewise small relative to the attempt budget.

more of objection 2

Section 5.2 states that the OpenCode-family members were chosen for deeper mechanism analysis rather than because they were top-ranked, so these rows are best treated as case studies of harness-conditioned behavior rather than as evidence about the evolution process. The fact that the preferred member changes between the LCB and SWE-bench protocols, which Section 9.3 says are not directly comparable, further weakens the claim of a general execution-side improvement.

5 minor objections

The paper says, at §6.1

Pith says: minor objection 1 of 5

Please state the absolute gain (2 tasks out of 100) together with the relative 4.0% improvement, so readers do not infer a four-task absolute improvement.

The paper says, at §6.1 and §5.2

Pith says: minor objection 2 of 5

Because the OpenCode-family members in Table 4 were selected for mechanism analysis rather than by ranking, the discussion of their 75/100 result should explicitly remind the reader that they were not the top candidates in the one-attempt matrix.

show 3 more

The paper says, at §2.4 and §7.3

Pith says: minor objection 3 of 5

Use 'oracle/post-hoc union' terminology consistently for the 58.0% coverage figure; the abstract's 'verified coverage' is acceptable only if the oracle nature is stated in the same sentence.

The paper says, at Title and §9.5

Pith says: minor objection 4 of 5

Since the evaluation does not train an updated model, the phrase 'recursive self-improvement' in the title overstates the demonstrated loop; a qualifier such as 'toward recursive self-improvement' would better match the one-round evidence.

The paper says, at §3.3

Pith says: minor objection 5 of 5

The sentence 'Coupling session with hooks and acceptance with the turn loop leaves five independent source choices' is hard to verify from the text; please spell out the five dimensions and show how the 4^5 and 4^6 counts are derived.

What would settle it

Hold out a fresh sample of at least 100 LiveCodeBench tasks (or a fresh SWE-bench split) that was never used to select or evaluate the 65 candidates. Run Pi and the 52-task winning harness, preferably all 65 candidates, for 10 attempts per task under the same deterministic=false protocol, fixing the winner from a development subset and measuring coverage on the held-out subset. Report a paired bootstrap or McNemar confidence interval for the winner-minus-Pi coverage difference. If the interval includes zero, or a different harness wins under repeated re-sampling, the 4.0% fixed-harness result should be treated as unestablished rather than as evidence for Finding 1.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 27 Aug 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 1 min 9 s · 18,928 tokens read, 6,774 written
  3. Editor wrote the verdict above 19 s · 19,725 tokens read, 1,925 written
  4. Referee major revision: 2 major objections, 5 minor 1 min 38 s · 19,154 tokens read, 11,483 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

First thing you should know: this paper is worth reading for the HELIX substrate and the sibling-data handoff, not for the 4.0% execution gain. The gain is 52 vs 50 on a 100-task set where the same tasks select and evaluate the winner, one attempt per candidate, with deterministic=false. Best-of-64 selection makes that gap easy noise, and the stress-test note is right that a paired two-task discordance can carry a McNemar p-value around 0.25.

What is actually new: HELIX decomposes four heterogeneous harnesses into typed ports, atoms, recipes, product shells, and runtime policies, and preserves intervention identity through lockfiles and an evidence plane. That is a genuine systems contribution relative to HarnessX and Recursive Harness Self-Improvement, which are cited and positioned as complementary rather than dismissed. The 438-row materialization from a 200-slot sibling slice into SFT, critic, filter, and preference records is concrete, and the paper is unusually transparent: it labels oracle coverage as non-deployable, admits rankings are not directly comparable, and Section 9.3 explicitly calls for separate evolution and evaluation tasks. That honesty is real and should be credited.

Soft spots, in proportion: the central recursive self-improvement claim is not tested. There is one evolution round, no model update, and no rebuild. The data handoff is materialized but never used to train or update anything. More load-bearing, the execution-side Finding 1 is not established by the reported numbers: 52/100 is a selected maximum on the same tasks used for selection. The repeated-run follow-ups use different candidates and attempt budgets, and the preferred member flips between LCB and SWE-bench, which Section 9.3 concedes. So the paper should not present "harness evolution improves execution today" as a settled finding. The portfolio coverage numbers are explicitly post-hoc oracle unions, so they are fine as evidence of potential yield, not as deployable performance. The Pytest and Xarray trace pairs are nice qualitative illustrations, but they are descriptions of non-deterministic rollouts, not causal estimates.

Citation pattern looks fine. The self-citation to CLI-Anything is peripheral, and the positioning against recent harness work is accurate.

Who this is for: people working on agent harnesses, data curation for agent training, or RSI infrastructure will get value from the design and the honest evaluation template. The paper deserves a serious referee, but the evaluation section needs major revision: repeated-run evidence with a stable winner, a held-out evaluation set, and ideally one small closed build-update-rebuild round before the recursive claim is credible. I would send it to peer review with that expectation rather than desk reject it.

The strongest objection

The execution-side half of the central claim rests on a two-task coverage difference between Pi (50/100) and the best of 64 evolved candidates (52/100) in a single-attempt matrix (Sections 5.2 and 6.1), run with deterministic=false (Section 5.4). Because the same 100 tasks are used both to choose the winner and to report its score, 52/100 is a selected maximum, not an unbiased estimate. A paired 52-vs-50 comparison is already within ordinary one-attempt noise: the discordant margin can be as small as two tasks (McNemar one-sided p about 0.25 in the most favorable split), and best-of-64 selection inflates the observed gap further. The repeated-run rows do not repair this: they use the same task set, a different subset of candidates, and the preferred member changes between LCB and SWE-bench; Section 9.3 explicitly says the rankings are not directly comparable. The 58.0% portfolio-coverage figure is an oracle union, not a deployable fixed-harness score, so it cannot substitute for the fixed-harness comparison. The data-materialization contribution may survive, but the paper's first Finding, that harness evolution improves execution today, is not supported by the reported numbers as they stand.

The reader's weakest assumption is the same vulnerability I would flag: the single-attempt, best-of-64, same-task selection-and-evaluation design makes the 52-vs-50 execution gain inseparable from selection noise. The paper's own Section 9.3 admits selection effects and non-comparable rankings, and Section 9.5 admits that no model update was performed, so the recursive closure is also unexecuted. These admissions are to the authors' credit, and the HELIX substrate, provenance tracking, and the 438-row data-accounting exercise are real contributions. The central claim is nonetheless conditional on a statistically secure execution result and on at least one closed build-update-rebuild round. I do not see a reason to move beyond the reader's CONDITIONAL verdict: the proposed held-out repeated-run test could either rescue or falsify the execution-side claim, and until it is run the paper should not be read as having demonstrated that harness evolution reliably improves fixed-model execution.

The referee's report

If the execution-side improvement were robustly established, this paper would be a valuable contribution to agent-systems and RSI research: it offers an auditable substrate for harness evolution, a concrete mechanism for producing matched sibling trajectories, and a clear data-handoff path to SFT/critic/filter/preference records. The HELIX representation (typed ports, atoms, recipes, product shells, policies) and the evidence plane are genuine engineering contributions, and the use of the official SWE-bench evaluator plus the explicit limitation statements in Section 9 are strengths. However, the quantitative support for the 'execution today' half of the central claim is weak: the headline 4.0% gain rests on a single-attempt, same-task selection/evaluation design, and the follow-up comparisons lack significance testing and use a non-random candidate subset. The data-materialization side is more convincing as a feasibility demonstration, but the recursive loop itself is not closed because no model update is performed. The significance is therefore conditional on strengthening the execution-side evidence.

Pith's reading of the paper (a machine's summary, not the authors')

HELIX treats the runtime harness as a co-evolutionary partner to the model: a harness shapes what a fixed model can do and which trajectories it leaves behind. The paper's central claim is that one harness-evolution round yields two coupled outputs—a stronger fixed runtime for today's model and verified sibling trajectories (successes, regressions, near misses, alternatives) that become SFT, critic, filter, and preference data for tomorrow's model update. In one code-repair round, evolutionary search over 65 harness candidates found a fixed harness covering 52/100 tasks versus 50/100 for the baseline Pi, while the whole portfolio covered 79/100; a selected 200-slot slice yielded 438 labeled training records. The authors argue this closes a feedback loop: model updates change the capability profile, so the harness portfolio should be rebuilt for the next round.

What carries the argument. The load-bearing machinery is the verified sibling trajectory with persistent intervention identity. HELIX represents each harness as a recipe of source-derived atoms behind stable typed ports, compiles it to a lockfile, runs it under the same model and task, and stores trace, workspace diff, verifier status, and policy evidence in one envelope. Because runs are matched by model, task, and attempt while differing only in declared harness choices, their outcomes form sibling sets: resolved versus regression, clean versus noisy, near miss versus hit. The paper formalizes the round as producing an execution reward $J_{\mathrm{deploy}}$ and a data reward $Q_{\mathrm{data}}$ from the sibling set, and the recursive state as $Z_t=(M_{\theta_t},\Phi_t)$ returning to $Z_{t+1}$ through build–update–rebuild.

Core claim. On the paper's own terms, the discovery is that harness evolution can be made explicit and auditable, and that its value is dual. HELIX decomposes four existing harnesses into typed ports, atoms, recipes, product shells, and runtime policies, so an intervention remains identifiable from declaration through execution and evidence capture. In one round, a 65-candidate portfolio on 100 LiveCodeBench-derived repair tasks produced a best fixed harness at 52/100 (Pi at 50/100), a post-hoc portfolio union of 79/100, and—after selected candidates were re-run and checked with the official SWE-bench evaluator—438 verified SFT, critic, filter, and preference rows from 200 sibling slots. The authors read these numbers as showing that the process improving execution today generates the data needed for tomorrow's model update, making the model–harness pair the unit of recursive improvement.

The premise it rests on. The load-bearing premise is that the two-task gap between the best evolved harness and the baseline (52 vs 50 out of 100) is a real harness effect rather than single-attempt sampling noise, since each candidate ran once per task with a nondeterministic model and the same 100 tasks were used to select and score the winner.

If the paper is right (Pith's inferences, not the authors')

If the central claim holds, a model can become more capable on a fixed benchmark without any weight change, simply by choosing a different source-traceable harness.

The verified sibling records give a concrete handoff: 200 matched slots became 438 SFT, critic, filter, and preference rows, so model updates can be trained on contrasts rather than on a single success bit.

After a model update, the previously best harness may no longer be best; rebuild is required because $\arg\max_\phi J(M_{\theta_{t+1}},H_\phi)$ need not equal $\arg\max_\phi J(M_{\theta_t},H_\phi)$.

The gap between the best fixed harness and the portfolio union (52 vs 79 on the LCB matrix) bounds the possible gain of a future router or staged selector, but no such selector is demonstrated.

The loop is verifier-grounded and task-external, so improvement is bounded and auditable rather than unrestricted self-rewriting.

My inference: the two-task fixed-harness advantage (52 vs 50) is fragile under one attempt per candidate-task pair with deterministic=false; the data-yield result would survive even if that ordering were noise, since it comes from repeated-run and official-evaluator follow-ups.

My inference: the 79/100 union implies that a substantial fraction of tasks are solvable by at least one harness in the portfolio but not by the baseline; a cheap early-trace classifier trained on these siblings could be tested as a router, with the union as an upper bound.

My inference: the same sibling-contrast machinery should transfer to non-coding agents, such as web navigation or database repair, where 'resolved versus no-action' and 'clean versus noisy' contrasts are equally meaningful; the paper does not test this.

My inference: the 438 records are derived rows, not independent rollouts; a convincing multi-round test would need to train a model on them and measure whether the next round's harness ranking shifts as predicted.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Scaling agent capability has largely focused on improving the model, yet an interactive agent acts through a runtime harness that mediates context, tools, control flow, and stopping. The harness shapes both what a model can accomplish and the trajectories from which it learns. This coupling motivates model-harness co-evolution for recursive self-improvement: build harnesses for a fixed model, update the model from verified sibling trajectories, and rebuild the harnesses as model capabilities change. Realizing this loop requires a controlled way to evolve harnesses while preserving intervention identity and effect. We present HELIX, a source-traceable substrate for harness evolution. HELIX decomposes agent systems into typed ports, reusable atoms, recipes, product shells, and runtime policies. It makes interventions explicit and auditable while retaining trajectories, test outcomes, and provenance. Harness evolution thus serves two linked roles: improving fixed-model execution and producing matched successes, regressions, near misses, and alternative solutions as data for subsequent model improvement. We evaluate HELIX in one evolution round on code repair. A 65-candidate portfolio discovers a fixed harness that improves task coverage by 4.0% over Pi, while the full portfolio exposes up to 58.0% more verified coverage through complementary sibling behavior. Selected candidates are assessed with repeated runs and the SWE-bench evaluator. A 200-slot sibling slice yields 438 verified SFT, critic, filter, and preference records. These results show how harness, model, and data form a feedback system: harness evolution expands current capability and creates learning signal for the next model; model updates motivate the next round of harness evolution. HELIX provides an auditable interface for studying this recursive process. Code is available at https://github.com/HKUDS/HELIX.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-09-23

The authors' main claim, as the classifier picked it out 0.98

“These results show how harness, model, and data form a feedback system: harness evolution expands current capability and creates learning signal for the next model; model updates motivate the next round of harness evolution.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.07 · states limitation 0.44 · human participants 0.04 · controlled comparison 0.79 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.88 · mentions code or data availability 0.98

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Scaling agent capability has largely focused on improving the model, yet an interactive agent acts through a runtime harness that mediates context, tools, control flow, and stopping. background 0.96 0.07 0.03 asserted 0.99
The harness shapes both what a model can accomplish and the trajectories from which it learns. background 0.98 0.10 0.06 asserted 1.00
This coupling motivates model-harness co-evolution for recursive self-improvement: build harnesses for a fixed model, update the model from verified sibling trajectories, and rebuild the harnesses as model capabilities change. contribution 0.50 0.15 0.28 asserted 0.92
Realizing this loop requires a controlled way to evolve harnesses while preserving intervention identity and effect. gap 0.87 0.08 0.04 asserted 0.98
We present HELIX, a source-traceable substrate for harness evolution. contribution 0.98 0.88 0.07 asserted 1.00
HELIX decomposes agent systems into typed ports, reusable atoms, recipes, product shells, and runtime policies. method 0.87 0.77 0.04 asserted 1.00
It makes interventions explicit and auditable while retaining trajectories, test outcomes, and provenance. method 0.62 0.66 0.06 asserted 1.00
Harness evolution thus serves two linked roles: improving fixed-model execution and producing matched successes, regressions, near misses, and alternative solutions as data for subsequent model improvement. interpretation 0.49 0.27 0.34 asserted 1.00
We evaluate HELIX in one evolution round on code repair. method 0.67 0.96 0.03 asserted 0.97
A 65-candidate portfolio discovers a fixed harness that improves task coverage by 4.0% over Pi, while the full portfolio exposes up to 58.0% more verified coverage through complementary sibling behavior. result 1.00 0.96 0.13 asserted 1.00
Selected candidates are assessed with repeated runs and the SWE-bench evaluator. method 1.00 0.91 0.03 asserted 0.91
A 200-slot sibling slice yields 438 verified SFT, critic, filter, and preference records. result 0.96 0.96 0.04 asserted 1.00
These results show how harness, model, and data form a feedback system: harness evolution expands current capability and creates learning signal for the next model; model updates motivate the next round of harness evolution. interpretation 1.00 0.83 0.75 asserted 1.00
HELIX provides an auditable interface for studying this recursive process. contribution 0.86 0.73 0.13 asserted 1.00
Code is available at https://github.com/HKUDS/HELIX. contribution 0.84 0.51 0.02 asserted 0.91

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.93

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 10,701 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

References (22)
  1. [1] Yuhao Yang, Tianyu Fan, and Chao Huang.CLI-Anything: Towards Agent-Native Computer Use. arXiv:2606.03854, 2026. https://arxiv.org/abs/2606.03854 on PithPith's reviewarXiv
  2. [2] Tingyang Chen, Shuo Lu, Kang Zhao, Weicheng Meng, Hanlin Teng, Tianhao Li, Chao Li, Xule Liu, Jian Liang, Zhizhong Zhang, Yuan Xie, Heng Qu, Kun Shao, and Jian Luan.HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundry. arXiv:2606.14249v3, 2026. https://arxiv.o... on PithPith's reviewarXiv
  3. [3] Hyunin Lee, Jinglue Xu, Jeffrey Seely, Donghyun Lee, Matei Zaharia, and Yujin Tang.Recursive Harness Self-Improvement. arXiv:2607.15524, 2026. https://arxiv.org/abs/2607.15524 on PithPith's reviewarXiv
  4. [4] Mingguang Chen, Licheng Wang, and Bo Qu.Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops. arXiv:2607.07663, 2026. https://arxiv.org/abs/26 07.07663 on PithPith's reviewarXiv
  5. [5] Jürgen Schmidhuber.Gödel Machines: Self-Referential Universal Problem Solvers Making Provably Optimal Self-Improvements. arXiv:cs/0309048, 2003. https://arxiv.org/abs/cs/0309048 on PithPith's reviewarXiv
  6. [6] Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune.Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents. arXiv:2505.22954, 2025. https://arxiv.org/abs/25 05.22954 on PithPith's reviewarXiv
  7. [7] Shengran Hu, Cong Lu, and Jeff Clune.Automated Design of Agentic Systems. International Conference on Learning Representations, 2025. arXiv:2408.08435. https://arxiv.org/abs/2408.084 35 on PithPith's reviewarXiv
  8. [8] Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xiong-Hui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu.AFlow: Automating Agentic Workflow Generation. International Conference on Learning Represent... on PithPith's reviewarXiv
  9. [9] Zelong Li, Shuyuan Xu, Kai Mei, Wenyue Hua, Balaji Rama, Om Raheja, Hao Wang, He Zhu, and Yongfeng Zhang.AutoFlow: Automated Workflow Generation for Large Language Model Agents. arXiv:2407.12821, 2024. https://arxiv.org/abs/2407.12821 on PithPith's reviewarXiv
  10. [10] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press.SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. Advances in Neural Information Processing Systems 37, 2024. arXiv:2405.15793. https://arxiv... on PithPith's reviewarXiv
  11. [11] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica.LiveCodeBench: Holistic and Contamination Free Eval- uation of Large Language Models for Code. International Conference on Learning Representations, on Pith
  12. [12] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan.SWE-bench: Can Language Models Resolve Real-World GitHub Issues?International Conference on Learning Representations, 2024. arXiv:2310.06770. https://arxiv.org/abs/2310.06... on PithPith's reviewarXiv
  13. [13] Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training Software Engineering Agents and Verifiers with SWE-Gym. Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:47717–47737, 2025. arXiv:2412.21139. ht... on PithPith's reviewarXiv
  14. [14] John Yang, Kilian Lieret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang.SWE-smith: Scaling Data for Soft- ware Engineering Agents. Advances in Neural Information Processing Systems 38, Datasets and Ben... on PithPith's reviewarXiv
  15. [15] Hongjin Su, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, and Sercan Ö. Arık.Learn-by- interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environments. Interna- tional Conference on Learning Representations, 2025. arXiv:2501.10893. https://arxiv.org/abs... on PithPith's reviewarXiv
  16. [16] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.ReAct: Synergizing Reasoning and Acting in Language Models. International Conference on Learning Representations, 2023. arXiv:2210.03629. https://arxiv.org/abs/2210.03629 on PithPith's reviewarXiv
  17. [17] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.Toolformer: Language Models Can Teach Themselves to Use Tools. Advances in Neural Information Processing Systems 36, 2023. arXiv:2302... on PithPith's reviewarXiv
  18. [18] Anomaly.OpenCode source repository. https://github.com/anomalyco/opencode on Pith
  19. [19] Earendil Works.Pi Mono source repository. https://github.com/earendil-works/pi on Pith
  20. [20] HKUDS.Nanobot source repository. https://github.com/HKUDS/nanobot on Pith
show 2 more
  1. [21] Nous Research.Hermes Agent source repository. https://github.com/NousResearch/hermes-agent. 27 on Pith
  2. [2025] arXiv:2403.07974. https://arxiv.org/abs/2403.07974 on PithPith's reviewarXiv
Cited by 7 Pith papers
How the literature uses it

2 arXiv papers cite this in 4 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

Papers citing it · 3 sentences without a confident label

Cites it as background 1 paper
  • HELIX [ref] rebuilds a fixed model's harness, updates the model from verified trajectories, and rebuilds the harness again; Co-Harness [ref] alternates validated harness edits with fine-tuning on trajectories produced by the improved harness.

    Aspire: Can Models Self-Evolve from Vague Goals? · Related Work · 0.92

What it cites · 13 sentences without a confident label

Uses data or values from 2 papers
Cites as background 6 papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment