Pith. sign in

REVIEW 3 major objections 7 minor 32 references

Looping coding agents can find a correct patch and then destroy it; reliability needs state-bound evidence and typed revision contracts, not more revisions.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 10:46 UTC pith:IUBJNPEI

load-bearing objection Careful common-state evidence that looping ≠ reliability, plus honest negatives on the contract—worth engaging even if transportability is thin. the 3 major comments →

arxiv 2607.24604 v1 pith:IUBJNPEI submitted 2026-07-27 cs.CL cs.AI

Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair

Xueping Gao , Jianwei Yang , Qiang Yang This is my paper
classification cs.CL cs.AI
keywords coding agentsagent reliabilitycode repairverificationstopping policiesstate-bound evidencetyped revision contractsLLM evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Coding agents usually work as generate–test–revise loops, but this paper argues that finding a correct patch is not the same as keeping it, verifying it against the right state, and safely stopping. Across sealed multi-seed repairs, forced revision makes current correctness fall even while ever-correct rises, so correctness is not an absorbing state. Holding the program fixed and only changing the evidence shows that stale traces—valid failure reports about a different code state—can resurrect fixed bugs, with a large, significant harm increase for a stronger coder. Model-family diversity does not guarantee low verifier risk or independent errors. The authors therefore split reliability into admission, preservation, grounded certification, competence, and liveness, and derive an evidence-bound typed loop contract whose enforceable core binds verifier evidence to exact code hashes, keeps last-known-good checkpoints, and issues auditable admission receipts. The reference implementation is a conformance artifact, not a claim of better repair skill.

Core claim

Proposal search is not completion reliability. Under forced revision, current correctness with current traces falls from 0.820 after one revision to 0.673 after two while ever-correct rises; under common-state branching that removes post-treatment bias, stale traces harm 34/135 correct starts versus 4/135 with current traces for the 14B coder (+22.2 points). Orchestration must separately enforce state-bound evidence, preservation, grounded certification, and stopping rather than relying on loop depth alone.

What carries the argument

Evidence-bound typed loop contract: every observation is an envelope binding payload to code-state and suite hashes; revisions are typed Keep | Patch | Escalate; last-known-good checkpoints are preserved; completion requires fresh certification at a declared risk–coverage operating point. Its mechanically enforceable subset is instantiated as StateSeal, which admits only hash-matched, gate-checked states and emits auditable receipts.

Load-bearing premise

The sealed HumanEval-style test splits and the 24 small selected repository bugs are good enough oracles and settings that the measured state-mismatch harms and contract obligations transfer to real agent deployment.

What would settle it

A common-state experiment on held-out repair tasks where stale traces do not raise correct-to-wrong transitions relative to current traces at matched coder capacity, or a prospective guarded loop that both eliminates correct-start harm and meets a joint unsafe-superiority plus sound-completion non-inferiority criterion without a severe completion floor.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Agent evaluations should report current versus ever-correct, common-state transition rates, coverage–risk, sound completion, and conditional verifier dependence—not only pass@k or best-of-k.
  • Orchestrators must reject or refresh evidence whose code hash does not match the live state; payload truth is not enough if provenance is wrong.
  • Stopping policies need calibrated selective risk and error dependence, not model-family diversity as a proxy for independence.
  • Admission, checkpoint preservation, and fresh certification can be enforced mechanically even when the model and oracle remain fallible.
  • More revision rounds sample more transitions; without a preservation and stopping rule they can lower final correctness while raising ever-correct.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Any long-horizon tool agent that feeds its own past observations back as context faces the same stale-evidence resurrection risk once the world or workspace has moved on.
  • Supply-chain style provenance (bind evidence to exact artifact identity before acting) is a natural fit inside agent transactions, not only for build pipelines.
  • Severe repository completion floors suggest that contract gains will stay invisible until base repair competence and action-interface fit clear a minimum liveness threshold.
  • Verifier ensembles should be scored at the actual acceptance operating point for excess joint false accepts, not marketed as independent because architectures differ.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper distinguishes proposal search (ever producing a correct patch) from completion reliability (retaining, verifying, and submitting it) in generate–test–revise coding agents. A sealed five-seed forced-revision study over 30 HumanEval repair tasks (900 trajectories) shows current correctness is non-absorbing: it falls from 0.820 to 0.673 under continued revision while ever-correct rises. Two prespecified common-state studies (2,430 branches at 7B and 14B) intervene on evidence provenance from identical frozen states; at 14B, stale traces harm 34/135 correct starts vs 4/135 with current traces (+22.2 pp, task-cluster CI [8.9, 37.0], Holm p=0.0337), the study's single Holm-significant confirmatory contrast. Verifier analyses show model-family diversity implies neither low selective risk nor conditional error independence. A prospective guarded policy eliminates observed correct-start harm but fails its preregistered joint criterion (sound-completion non-inferiority not met), and repository factorials over 24 bugs and four coder stacks show floor effects with no Holm-significant component effects. The authors derive an evidence-bound typed loop contract (state-bound evidence envelopes, typed Keep/Patch/Escalate actions, last-known-good checkpoints, fresh recertification, auditable stopping) and ship a reference implementation (StateSeal) whose 35 fault-injection conformance cases pass, explicitly disclaimed as effectiveness evidence.

Significance. If the results hold, the paper contributes (i) a clean empirical demonstration, protected against post-treatment risk-set bias by common-state branching, that state-misaligned evidence materially increases correct-to-wrong transitions; (ii) a reproducible measurement methodology (sealed suites, hash lineage, preregistered families, task-cluster inference, exact sign-flip sensitivity) that is a model for agent-evaluation work; (iii) a rare honest accounting in which the authors' own prospective policy and repository factorials fail their joint criteria, reported rather than buried; and (iv) a machine-checkable contract with a 35-case fault-injection conformance suite — a falsifiable artifact rather than a prose proposal. The transportability caveat (27 HumanEval-derived tasks, shared HumanEval+ suite construction, 24 small Python repository bugs) is real but acknowledged; the claims are framed as mechanism results, which the evidence supports.

major comments (3)
  1. [§4.3 / §5.2, Table 3] The headline causal contrast (14B stale-wrong-trace vs current trace on correct starts, 34/135 vs 4/135) compares a report claiming test failures against a report showing all-pass, because a correct program's own current trace cannot report failure. The significant contrast therefore bundles state misalignment with failure-report content: an aligned but false failure report might produce the same regression without any staleness. The design contains exactly the needed control -- the flipped-binary condition (Table 1) -- but only in the trajectory study, and its correct-to-wrong transitions are never reported in §5. The matched control (neutral text) addresses length/attention but not failure semantics, and stale-minus-control does not survive Holm (p=0.3934). The HumanEval/89 example in §5.3 qualitatively supports the provenance mechanism, but the quantitative claim that state alignment
  2. [§5.2, Table 3 and Figure 2] The paper's single Holm-significant confirmatory result (stale minus current, 14B) is estimated from 27 task clusters with harms concentrated in 12 clusters and two tasks (HumanEval/10, /46) regressing in 5/5 seeds (§5.3). Given that the prespecified 7B family is entirely null (all six contrasts Holm p=1.0), the reader needs to know how much of the +22.2-point estimate and its p=0.0337 depend on one or two tasks. Please add a leave-one-task-out (or influence) sensitivity for this contrast, analogous to the leave-one-project-out checks already reported for the repository study in §5.6.
  3. [§4.4 and §8 (Reproducibility)] The confirmatory claims rest on sealed challenge/hidden suites and hash-locked adjudication, but §8 states that raw trajectories and private suites are withheld even from review, and the artifact is described as a 'numerical package' supporting 'reconstruction and hash auditing.' As written I cannot determine whether the package contains the task-level branch outcomes needed to recompute Table 3, the bootstrap CIs, and the exact sign-flip p-values, or only summary statistics. Since the zero-event bounds (§5.5) and the single significant contrast are the load-bearing numbers, please specify precisely which matrices are released and provide reviewer-accessible adjudication samples sufficient to spot-check the sealed-suite pipeline.
minor comments (7)
  1. [§5.1] The RQ1 trajectory numbers (0.820 → 0.673 → 0.693 current correctness; 16.0% correct-then-lost) are reported without task-cluster uncertainty, unlike the common-state contrasts. Given that all inference elsewhere is task-clustered, please add CIs here for consistency.
  2. [§4.3] 'Lexicographically first v0.4.3 revision' -- the version string v0.4.3 is never defined; clarify what it refers to (harness version?) and why lexicographic order is the right tie-break.
  3. [§5.5, Table 5] The 'Challenge executable' row (Cov 0.788) versus 'Self + challenge' (0.778): the difference between these two gates is not explained; a one-line definition would help.
  4. [§5.2] Prompt-token means for stale trace vs matched control differ by ~10% (498.4 vs 451.7). The text calls this an 'audited nuisance variable' but does not show the audit; a brief sensitivity or at least a pointer to the artifact would close the loop.
  5. [Title/Abstract] The title and abstract phrasing ('Looping Is Not Reliability') is stronger than the deliberately scoped forced-revision evidence; §5.1's own scoping paragraph is excellent and could be echoed in the abstract (e.g., 'under forced revision').
  6. [§5.6] The +4.17pp unsafe-completion effect under Contract ON in the Qwen2.5-Coder-32B replication (gate false-passed 7/237 incorrect commits) is an important cautionary result for the contract thesis; consider promoting it from the replication paragraph into §7's discussion of completion-gate dependence.
  7. [§3 / §4.3] In §3, R_W(e) and R_C(e) are defined against a matched nonactionable control e0, but Table 3's correct-start branches in the 7B study lack a stale-binary arm; a sentence explaining the asymmetric branch sets (4 wrong-state vs 5 correct-state conditions) would help readers map the design to the estimands.

Circularity Check

0 steps flagged

No significant circularity: empirical transition measurements and a motivated contract, not a self-justifying derivation.

full rationale

This paper measures proposal-search versus completion-reliability gaps with sealed HumanEval trajectories, common-state branches, verifier risk–coverage, and repository factorials, then motivates a typed orchestration contract from those observed failure modes. The quantities α_e, β_e, R_W, R_C, Risk_v(c), and D_ij(c) are definitional measurement tools and elementary probability identities (Eqs. 1–8), not fitted parameters later relabeled as predictions. Empirical contrasts (e.g., 14B stale vs current harm 34/135 vs 4/135) are assignment-based outcomes against external executable suites, not closures of the authors’ own definitions. The contract and StateSeal reference implementation are explicitly scoped as an executable specification and conformance artifact, not as proof of improved competence or calibrated verifier dependence. Related work is external; there is no load-bearing self-citation uniqueness theorem, no ansatz smuggled in as a forced result, and no renaming of a known law as a first-principles derivation. Transportability and oracle limitations are external-validity concerns, not circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 3 invented entities

Central claims rest on standard probabilistic transition accounting, selective-prediction risk–coverage, executable unit-test oracles as correctness proxies, and design choices for sealed suites, frozen states, gates, and budgets. No physical constants or large fitted natural-law parameters; free parameters are experimental knobs and policy thresholds. Invented entities are protocol constructs (typed actions, evidence envelopes, StateSeal broker), introduced as engineering interfaces rather than hidden natural kinds.

free parameters (4)
  • Forced-revision depth (3) and sampling (T=0.3, top-p=0.9) = 3 revisions; T=0.3; top-p=0.9
    Trajectory kernel and non-absorption magnitudes depend on these author-chosen generation settings.
  • Native verifier accept threshold 0.5 and retrospective coverage cutoffs (e.g. 70%) = 0.5 native; matched coverage including 70%
    Risk, dependence φ, and policy comparisons are operating-point conditional; cutoffs are chosen for analysis, not learned from a held-out deployment calibration.
  • Prospective/repo non-inferiority margin (−5 pp sound completion) and budgets (actions/tests/wall-clock) = −5 percentage points; 15 actions / 3 source revisions (repo)
    Joint pass/fail decisions and liveness interpretation hinge on this preregistered margin and sparse-completion budgets; authors note floor-limited weakness when baseline sound completion is near zero.
  • Task selection filters (30 HumanEval repairs; 27 with both states; 24 repo bugs ≤5 files/200 LOC) = 30 / 27 / 24 tasks as specified
    Estimands are conditional on repairable/selected tasks; absolute rates are not population samples of all bugs or agents.
axioms (5)
  • domain assumption Executable pass/fail on visible+challenge+hidden (or repo fail-to-pass/pass-to-pass/full) suites is the correctness label Y.
    Section 4.1 and 4.6; finite suites are incomplete specifications and share HumanEval+ construction.
  • domain assumption Task clusters are the independent sampling units; seeds are nested replicates.
    Section 4.4 inference design; underpins bootstrap/sign-flip CIs and Holm claims.
  • standard math q_{t+1}-q_t = (1-q_t)α_e - q_t β_e transition accounting for expected correctness change.
    Section 3 definitions; standard two-state transition identity.
  • domain assumption Common-state frozen wrong/correct programs identify evidence-alignment contrasts without post-treatment risk-set bias from free-running trajectories.
    Section 4.3; core identification strategy for RQ2.
  • ad hoc to paper Mechanically enforceable obligations (hash-bound evidence, typed Keep/Patch/Escalate, LKG checkpoint, fresh recertification) are separable from competence and calibrated dependence.
    Sections 6–7 contract derivation; supported as interface decomposition, not as a fitted multiplicative law.
invented entities (3)
  • Evidence-bound typed loop contract (state-bound envelopes, typed revision actions, LKG ledger, risk/dependence-aware stopping) independent evidence
    purpose: Make admission, preservation, and completion obligations explicit and partially enforceable around existing agents/verifiers.
    Derived in Section 6 from observed failure modes; obligations table maps mechanisms to residual risks.
  • StateSeal admission broker / reference implementation independent evidence
    purpose: Executable specification binding verifier evidence to code-state digests, checkpoints, and auditable receipts.
    Section 6.2; authors state it is conformance-only (35 failure-injection cases), not an effectiveness trial.
  • Effective evidence response contrasts R_W(e), R_C(e) no independent evidence
    purpose: Operationalize joint informativeness×responsiveness of evidence on wrong vs correct starts versus matched control.
    Section 3; explicitly not a separable structural factorization.

pith-pipeline@v1.2.0-grok45-kimik3 · 22198 in / 4106 out tokens · 85555 ms · 2026-07-31T10:46:15.604087+00:00 · methodology

0 comments
read the original abstract

Generate--test--revise loops are common in coding agents, but repetition alone provides no reliability guarantee. We study the gap between finding a correct patch and retaining, verifying, and submitting it. A sealed five-seed study over 30 HumanEval repairs produces 900 three-revision trajectories. Under forced revision, current correctness with current traces falls from 0.820 after one revision to 0.673 after two, although ever-correct rises to 0.847. Two common-state studies use 2,430 branches from identical frozen programs to remove post-treatment risk-set bias. In a prespecified 14B replication, stale traces harm 34/135 correct starts versus 4/135 with current traces, a 22.2-point increase (task-cluster 95\% CI $[8.9,37.0]$, exact Holm $p=0.0337$). A prospective 540-rollout policy eliminates observed correct-start harm but reduces wrong-start repair and fails its joint criterion. Repository experiments over 24 bugs and four coder stacks expose floor effects and component heterogeneity without Holm-significant effects. We therefore separate admission, preservation, grounded certification, competence, and liveness. We derive an evidence-bound typed loop contract and instantiate its mechanically enforceable subset in a reference implementation that binds verifier evidence to exact code states, preserves verified checkpoints, and emits auditable admission receipts. The implementation is an executable specification and conformance artifact, not evidence of improved repair competence or calibrated verifier dependence.

Figures

Figures reproduced from arXiv: 2607.24604 by Jianwei Yang, Qiang Yang, Xueping Gao.

Figure 1
Figure 1. Figure 1: Left: retrospective selective risk at matched coverage. Cross-family diversity does not ensure verifier quality. Right: [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: 7B correct-to-incorrect outcomes are clustered, not [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 2
Figure 2. Figure 2: Assignment outcomes from identical frozen states. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Prespecified repository-agent contract effects (ON minus OFF). Bars are task-cluster bootstrap 95% intervals over 24 [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references · 5 canonical work pages · 5 internal anchors

  1. [1]

    Islem Bouzenia and Michael Pradel. 2025. Understanding Software Engineer- ing Agents: A Study of Thought-Action-Result Trajectories. In40th IEEE/ACM International Conference on Automated Software Engineering. IEEE, 2846–2857. doi:10.1109/ASE63991.2025.00234

    arXiv 2025

  2. [2]

    José Pablo Cambronero, Michele Tufano, Sherry Shi, Renyao Wei, Grant Uy, Runx- iang Cheng, Chin-Jung Liu, Shiying Pan, Satish Chandra, and Patrick Rondon

  3. [3]

    Mnemosyne: Agentic Transaction Processing for Validating and Repairing AI-generated Workflows

    Edward Y. Chang, Longling Geng, and Emily J. Chang. 2026. Mnemosyne: Agentic Transaction Processing for Validating and Repairing AI-Generated Workflows. arXiv preprint arXiv:2607.00269(2026). doi:10.48550/arXiv.2607.00269

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2607.00269 2026

  4. [4]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, et al . 2021. Evaluating Large Language Models Trained on Code.arXiv preprint arXiv:2107.03374(2021). doi:10. 48550/arXiv.2107.03374

    Pith/arXiv arXiv doi:10.48550/arxiv.2107.03374 2021

  5. [5]

    Dekun Dai, MingWei Liu, Anji Li, Jialun Cao, Yanlin Wang, Chong Wang, Xin Peng, and Zibin Zheng. 2025. FeedbackEval: A Benchmark for Evaluating Large Language Models in Feedback-Driven Code Repair Tasks.arXiv preprint arXiv:2504.06939(2025). doi:10.48550/arXiv.2504.06939

    doi:10.48550/arxiv.2504.06939 2025

  6. [6]

    Yonatan Geifman and Ran El-Yaniv. 2019. SelectiveNet: A Deep Neural Network with an Integrated Reject Option. InProceedings of the 36th International Con- ference on Machine Learning (Proceedings of Machine Learning Research, Vol. 97). PMLR, 2151–2159. https://proceedings.mlr.press/v97/geifman19a.html

    2019

  7. [7]

    Thibaud Gloaguen, Niels Mündler, Mark Niklas Mueller, Veselin Raychev, and Martin Vechev. 2026. Coding Agents Don’t Know When to Act. InSecond Work- shop on Agents in the Wild: Safety, Security, and Beyond. https://openreview.net/ forum?id=CVg4Honh4Y

    2026

  8. [8]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, et al . 2024. DeepSeek-Coder: When the Large Language Model Meets Programming—The Rise of Code Intelligence. arXiv preprint arXiv:2401.14196(2024). doi:10.48550/arXiv.2401.14196

    Pith/arXiv arXiv doi:10.48550/arxiv.2401.14196 2024

  9. [9]

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024. Large Language Models Cannot Self- Correct Reasoning Yet. InInternational Conference on Learning Representations. https://openreview.net/forum?id=IkmD3fKBPQ

    2024

  10. [10]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. InInternational Conference on Learning Rep- resentations. https://openreview.net/forum?id=VTF8yNQM66

    2024

  11. [11]

    Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. 2024. When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self- Correction of LLMs.Transactions of the Association for Computational Linguistics 12 (2024), 1417–1440. doi:10.1162/tacl_a_00713

    doi:10.1162/tacl_a_00713 2024

  12. [12]

    Elliot Myunghoon Kim, Avi Garg, Kenny Peng, and Nikhil Garg. 2025. Correlated Errors in Large Language Models. InProceedings of the 42nd International Confer- ence on Machine Learning (Proceedings of Machine Learning Research, Vol. 267). PMLR, 30038–30066. https://proceedings.mlr.press/v267/kim25e.html

    2025

  13. [13]

    Cuong Chi Le, Aashish Yadavally, Minh Le-Anh, and Tien N. Nguyen. 2026. Benchmarking Code Improvement with Progressive, Adaptive, and Interactive Feedback.arXiv preprint arXiv:2607.01360(2026). doi:10.48550/arXiv.2607.01360

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2607.01360 2026

  14. [14]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. InAdvances in Neural Information Pro- cessing Systems, Vol. 36. Neural Information Processing Systems Foundation, Inc., 21558–21572. doi:10.52202/075280-0943

    doi:10.52202/075280-0943 2023

  15. [15]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al

  16. [16]

    Qianru Meng, Xiao Zhang, Zhaochun Ren, and Joost Visser. 2026. EviACT: An Evidence-to-Action Framework for Agentic Program Repair.arXiv preprint arXiv:2605.27238(2026). doi:10.48550/arXiv.2605.27238

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2605.27238 2026

  17. [17]

    Open Policy Agent Contributors. 2026. Open Policy Agent Documentation. Accessed 2026-07-24. https://www.openpolicyagent.org/docs

    2026

  18. [18]

    Bowman, and Shi Feng

    Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. LLM Evaluators Recognize and Favor Their Own Generations. InAdvances in Neural Information Processing Systems, Vol. 37. Neural Information Processing Systems Foundation, Inc., 68772–68802. doi:10.52202/079017-2197

    doi:10.52202/079017-2197 2024

  19. [19]

    Andrey Podivilov, Vadim Lomshakov, Sergey Savin, Matvei Startsev, Roman Pozharskiy, Maksim Parshin, and Sergey Nikolenko. 2026. AgentLens: Production- Assessed Trajectory Reviews for Coding Agent Evaluation.arXiv preprint arXiv:2607.06624(2026). doi:10.48550/arXiv.2607.06624

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2607.06624 2026

  20. [20]

    Qwen Team. 2024. Qwen2.5 Technical Report.arXiv preprint arXiv:2412.15115 (2024). doi:10.48550/arXiv.2412.15115

    Pith/arXiv arXiv doi:10.48550/arxiv.2412.15115 2024

  21. [21]

    Priyam Sahoo, Gaurav Mittal, Xiaomin Li, Shengjie Ma, Benjamin Steenhoek, Pingping Lin, and Yu Hu. 2026. AgentLens: Revealing the Lucky Pass Problem in SWE-Agent Evaluation.arXiv preprint arXiv:2605.12925(2026). doi:10.48550/ arXiv.2605.12925

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2605.12925 2026

  22. [22]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language Agents with Verbal Reinforcement Learn- ing. InAdvances in Neural Information Processing Systems, Vol. 36. Neural Informa- tion Processing Systems Foundation, Inc., 8634–8652. doi:10.52202/075280-0377

    doi:10.52202/075280-0377 2023

  23. [23]

    Supply-chain Levels for Software Artifacts. 2026. SLSA v1.2 Provenance. Accessed 2026-07-24. https://slsa.dev/spec/v1.2/provenance

    2026

  24. [24]

    Santiago Torres-Arias, Hammad Afzali, Trishank Karthik Kuppusamy, Reza Curt- mola, and Justin Cappos. 2019. in-toto: Providing Farm-to-Table Guarantees for Bits and Bytes. In28th USENIX Security Symposium. USENIX Association, 1393–

    2019

  25. [25]

    Gladys Tyen, Hassan Mansoor, Victor Carbune, Peter Chen, and Tony Mak. 2024. LLMs Cannot Find Reasoning Errors, but Can Correct Them Given the Error Location. InFindings of the Association for Computational Linguistics: ACL 2024. Association for Computational Linguistics, 13894–13908. doi:10.18653/v1/2024. findings-acl.826

    doi:10.18653/v1/2024 2024

  26. [26]

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2025. De- mystifying LLM-Based Software Engineering Agents.Proceedings of the ACM on Software Engineering2, FSE (2025), 801–824. doi:10.1145/3715754

    doi:10.1145/3715754 2025

  27. [27]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-Computer Inter- faces Enable Automated Software Engineering. InAdvances in Neural Information Processing Systems, Vol. 37. Neural Information Processing Systems Foundation, Inc., 50528–50652. doi:10.52202/079017-1601

    doi:10.52202/079017-1601 2024

  28. [28]

    Yunxiang Zhang, Muhammad Khalifa, Lajanugen Logeswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, and Lu Wang. 2024. Small Language Models Need Strong Verifiers to Self-Correct Reasoning. InFindings of the Association for Computational Linguistics: ACL 2024. Association for Computational Linguistics, 15637–15653. doi:10.18653/v1/2024.findings-acl.924

    doi:10.18653/v1/2024.findings-acl.924 2024

  29. [29]

    Bingchen Zhao, Dhruv Srikanth, Yuxiang Wu, and Zhengyao Jiang. 2026. SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents.arXiv preprint arXiv:2605.21384(2026). doi:10.48550/arXiv.2605.21384

    Pith/arXiv arXiv doi:10.48550/arxiv.2605.21384 2026

  30. [1410]

    https://www.usenix.org/conference/usenixsecurity19/presentation/torres- arias

  31. [2023]

    InAdvances in Neural Information Processing Systems, Vol

    Self-Refine: Iterative Refinement with Self-Feedback. InAdvances in Neural Information Processing Systems, Vol. 36. Neural Information Processing Systems Foundation, Inc., 46534–46594. doi:10.52202/075280-2019

    doi:10.52202/075280-2019 2019

  32. [2026]

    InProceedings of the IEEE/ACM 48th International Conference on Software Engineering: Software Engineering in Practice

    Abstain and Validate: A Dual-LLM Policy for Reducing Noise in Agentic Program Repair. InProceedings of the IEEE/ACM 48th International Conference on Software Engineering: Software Engineering in Practice. Association for Computing Machinery, 118–129. doi:10.1145/3786583.3786858 Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts ...

    arXiv 2026