Pith. sign in

REVIEW 3 major objections 6 minor 62 references

DiDPO: Diff-in-Diff Policy Optimization for Coding Agent Training

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DiDPO claims that RL credit for coding agents should be assigned at the level of functional sub-diffs inside code edits, and reports gains of 4.2–4.9 points over state-grouping baselines.

desk verdict DiDPO has a genuinely new idea for local credit assignment in coding-agent RL, and the internally controlled experiments show plausible gains, but the theory overclaims and missing error bars make the headline margins hard to trust. read the letter →

arxiv 2608.07147 v1 pith:GDHHNTTH submitted 2026-08-07 cs.AI

Xucong Wang , Zhe Zhao , Liheng Yu , Di Wu , Xiaofeng Cao , Pengkun Wang This is my paper · ORCID
classification cs.AI
keywords diff-in-diffpolicyoptimizationcodingagentscreditassignmentreinforcementlearningwithverifiablerewardscodediffsgroup-relativeadvantagesubmodularanchorselectionlong-horizongeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DiDPO claims that the right unit of credit for training coding agents is not the whole trajectory, the step, or the environment state, but the functional sub-diff inside a code change. It builds those units by matching similar sub-diffs across sampled rollouts, scoring candidate anchors with a groupability score that balances how much code an anchor represents with how many rollouts support it, and then averaging group-relative returns to get a diff-level advantage. This advantage is added to the trajectory-level advantage and projected onto response tokens, with no critic and no extra rollouts. If the claim is right, RL training for coding agents can localize feedback to individual edits, which is what the experiments report: on Qwen2.5-Coder-7B, DiDPO averages 48.4% across eight coding benchmarks, 4.2 points above GiGPO and more than 10% relative above comparable methods, narrowing the gap to much larger models.

What carries the argument

The load-bearing mechanism is the pairing of a groupability score with dynamic sub-diff anchors. The score is $GS(c) = (1-e^{-\bar L(c)})(1-e^{-(n(c)-1)})$, where $\bar L(c)$ is the average normalized size of the anchor's occurrences and $n(c)$ is the number of supporting occurrences; the saturating factors keep semantic scope and group mass from dominating each other. Anchor selection maximizes the total covered score over sub-diffs, a cardinality-constrained submodular maximization solved greedily with the classical $(1-1/e)$ guarantee. Selected anchors induce advantage groups over aligned sub-diffs, and the diff-level advantage is the normalized difference between a sub-diff's step return and the group mean, combined through $\hat A_{i,l} = A_E(\tau^{(i)}) + \lambda A_D(a_l^{(i)})$ and trained with the clipped objective of Eq. (14). The theoretical part treats sub-diff alignment as a metric correspondence with distortion error $\epsilon$, proving a local-contrast bias of $O(L\epsilon)$ and an MSE bound of $O(L^2\epsilon^2) + O(\sigma_\xi^2/m)$, with an $O(\sigma_\xi^2)$ contamination term for episode-level broadcasting.

What would settle it

On tasks with pairs of rollouts sharing a textually identical sub-diff but differing in whether that edit actually determines test outcomes, log DiDPO's assigned diff-level advantages: if the identical sub-diff receives the same advantage despite opposite causal roles, the exchangeability assumption fails; ablating the edit in each context would quantify the bias.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that code diffs produced by coding agents are divisible credit units: a single action at one step contains several functionally distinct sub-diffs, and these sub-diffs recur across rollouts even when whole diffs do not. DiDPO formalizes recurrence by aligning sub-diffs through a similarity threshold, selecting anchors through a groupability score, and grouping occurrences of the same anchor into advantage groups. It then defines a diff-level advantage by normalizing the step return against the average return within the group, and combines it with the trajectory-level advantage in a clipped PPO objective. The paper's central empirical assertion is that this edit-level credit improves long-horizon coding performance on both a 7B and a 4B backbone, with the largest gains on competition-style and interview-style tasks, at roughly 2.3% added training cost.

Load-bearing premise

The method assumes that sub-diffs that look similar across rollouts are the same kind of causal change, and that irrelevant trajectory noise averages out within each group; if the same edit helps in one context and hurts in another, the local advantage is biased.

Editorial extensions

If this is right

  • On Qwen2.5-Coder-7B, DiDPO raises the eight-benchmark average from 44.2 (GiGPO) to 48.4, and on Qwen3.5-4B from 53.7 to 58.6.
  • The gain is concentrated on long-horizon tasks: APPS Interview improves by 10.4 points over GiGPO on the 7B backbone, and USACO jumps to 15.6 versus GRPO's 6.8.
  • The method needs no value critic and no additional environment rollouts; measured training overhead over GRPO is about 2.3 percent.
  • Ablations make each component load-bearing: removing the episode-level advantage drops APPS from 31.3 to 10.4, removing the diff-level advantage to 23.8, and removing sub-diff decomposition to 25.0.
  • The saturating groupability score is not incidental: it outperforms an additive score (31.3 vs 24.4 on APPS) and an LLM-based judge (21.5).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same anchor-and-group machinery should transfer to other edit-heavy agent domains, such as notebook cells, document revisions, or configuration patches, where one action contains multiple functional changes; the paper only demonstrates code.
  • Editorial inference: the theory predicts that the similarity threshold and group mass should co-adapt, since more diverse rollouts make large low-bias groups harder to form; a testable extension would decay the matching threshold over training rather than fixing it at 0.8.
  • Editorial inference: if sub-diff recurrence really carries credit, then sampling strategies that deliberately generate variant rollouts around the same edit should increase group mass and speed learning, which is a natural but unexplored extension.
  • Editorial inference: a failure mode not covered by the paper is sparse overlap: on tasks where successful solutions share almost no code text, anchor groups shrink toward size one and DiDPO should degrade toward GRPO; measuring the correlation between group size and gain across tasks would bound when the method helps.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces DiDPO, a critic-free policy optimization method for coding agents that decomposes code diffs into sub-diffs, matches similar sub-diffs across rollouts using a groupability score, forms advantage groups, and computes diff-level advantages in addition to trajectory-level advantages. The method is evaluated on eight code-generation benchmarks with two backbones, with ablations and an efficiency analysis, and the authors release an open-source codebase. The central claim is that DiDPO improves coding-agent RL by providing fine-grained, local credit to individual edits without extra rollouts or a learned critic.

Significance. If the empirical gains are reproducible, the paper makes a useful contribution by extracting credit signals from the internal structure of code diffs, an underexplored axis for agentic RL. The comparisons are internally controlled: all RL baselines start from the same SFT checkpoint and share the environment and reward. The paper also provides theoretical bias-variance bounds and releases a codebase, which are concrete assets. However, the central quantitative claims currently rest on tables without error bars and on a theorem whose key decomposition is not validated; the significance is therefore conditional on those points being addressed.

major comments (3)
  1. [Section 5.2, Tables 1 and 2; Appendix E] The main tables report point estimates only, while Appendix E states that 'Validation accuracy is averaged over 5 runs' but provides no standard deviation or confidence interval. With evaluation subsets as small as 21 USACO Platinum problems and 53 ICPC Hard problems, differences of one or two points cannot be distinguished from noise; the abstract's 'over 10%' claim and the 4.2-4.9 point margins over GiGPO therefore need error bars or formal statistical tests. This is load-bearing because the paper's primary case is empirical superiority.
  2. [Section 4.5, Theorem 4.2 and Eq. 12] The theorem assumes R_i = r(s_i) + xi_i with zero-mean non-causal noise, but R_t in Eq. 12 is a discounted return over future steps, so the term r(s_i) would have to include all future rewards caused by the sub-diff. The assumption that xi_i is zero-mean is not established: later rewards often depend on later actions, not on noise from earlier steps, and a group in Eq. 12 can contain matched sub-diffs from different timesteps with different future horizons. Without a justification or an empirical check, the bias-variance bound does not establish that AD estimates the local causal contribution.
  3. [Section 4.2-4.3, Eq. 8-12] The method treats lexically and embedding-similar sub-diffs as exchangeable units, but identical-looking edits can have opposite effects depending on the surrounding code state, the order of edits, and whether later steps compensate. The paper does not test this exchangeability assumption. A controlled test, for example a synthetic coding environment where sub-diff effects are known or a leave-one-out style analysis on matched sub-diffs, is needed before the +7.5 APPS-point marginal gain attributed to AD in Table 3 can be interpreted as a credit-assignment signal rather than a grouping artifact.
minor comments (6)
  1. [Abstract] The abstract states that DiDPO 'exceeds comparable methods by over 10%', but Table 1 shows smaller average margins of 4.2 and 4.9 points over GiGPO; the specific comparison supporting the 10% claim should be identified.
  2. [Figure 5 (right)] The plot lacks axis labels and the '~2.3%' annotation is not defined; the paper should describe what is being measured and provide labeled axes so the overhead claim can be verified.
  3. [Section 5.1] The filtering of PRIME training data is described only as 'filter out those that belong to the validation set or do not conform to long-horizon code generation patterns'; the concrete filtering rules or a quantitative description of their effect should be reported.
  4. [Section 4.2 and Eq. 9] The claim that the groupability score 'optimally balances' semantic scope and group mass is an assertion; s0, g0, and K are free hyperparameters, so the selection or tuning procedure for these values should be described.
  5. [Section 4.4 and Eq. 11-13] The notation a(i)_l for response tokens and a(i)_{t,m} for sub-diff actions should be explicitly mapped, since Eq. 13 combines them without a formal definition of how a response token span maps to a sub-diff action.
  6. [Section 5.3, Table 3 (right)] The ablation labels are clear, but the claim that the components' combination 'exceeds additive contribution' is not quantified; the paper should report the sum of individual gains alongside the full-model result.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DiDPO's sub-diff credit assignment is empirically evaluated on held-out external benchmarks, and its theoretical bounds are conditional on explicit assumptions rather than derived from fitted targets.

full rationale

The paper's central mechanism is a credit-assignment estimator built from code-diff structure. The derivation chain in Eqs. 5-12 defines sub-diff candidates, anchor selection via the groupability score, and a diff-level advantage as a group-relative contrast over rollout returns. None of these quantities is fitted to test-set outcomes: the groupability score, similarity threshold, and lambda are design and hyperparameter choices validated during training, not constants inferred from the evaluation benchmarks. The empirical claims are comparisons on held-out external benchmarks with deterministic test execution, so the reported gains over GiGPO and GRPO are genuine experimental findings rather than predictions forced by construction. Theorems 4.1 and 4.2 are stated as conditional bounds under explicit Lipschitz and zero-mean-noise assumptions; they do not define the estimator's output as its input, and they do not rely on the paper's own prior results. The few self-citations in related work (e.g., refs. [34], [46], [47], [48]) appear in contextual surveys and are not load-bearing for the central claim. Therefore no step in the paper reduces, by definition or by self-citation, to its own inputs.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

DiDPO introduces no physical or metaphysical entities. Its inventions are algorithmic: sub-diff anchors, the groupability score, and the group-relative diff advantage. These rest on several hand-chosen hyperparameters and domain assumptions about similarity and noise, listed above.

free parameters (5)
  • lambda = 1.2
    Weight for the diff-level advantage term in Eq. 13; chosen from sensitivity analysis on validation, reported in Appendix E.
  • eta (sub-diff similarity threshold) = 0.8
    Matching threshold in Eq. 8; set by hand in Appendix E with no sensitivity analysis reported.
  • s0 (length scale in groupability score) = 8.0
    Added length scale in the implementation of the groupability score, reported in Appendix E; chosen by hand.
  • g0 (support scale in groupability score) = 8.0
    Added support scale in the implementation of the groupability score, reported in Appendix E; chosen by hand.
  • K (max anchors per task group) = 64
    Cardinality budget for anchor selection in Eq. 10; set in Appendix E.
assumptions (5)
  • standard math Greedy maximization of a monotone submodular function gives a (1-1/e) approximation.
    Used to justify anchor selection in Appendix B, citing Nemhauser et al. [36].
  • domain assumption Sub-diff similarity at threshold eta captures functional or causal equivalence of code changes.
    Required for grouping in Eq. 8 and advantage computation in Eq. 11-12; not empirically validated.
  • domain assumption Step return decomposes as R_i = r(s_i) + xi_i with xi_i zero-mean non-causal noise.
    Assumed in Theorem 4.2 to prove variance reduction; no evidence that non-causal trajectory effects are zero-mean.
  • domain assumption Local reward r(s) is L-Lipschitz with respect to the correspondence distance Delta.
    Assumed in Theorem 4.1; no estimate of L or verification of the Lipschitz condition.
  • ad hoc to paper The groupability score in Eq. 9 optimally balances semantic scope and group mass.
    The exponential saturating form and the scales are chosen by the authors rather than derived; alternative additive or LLM-judge forms underperform in Table 3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DiDPO: Diff-in-Diff Policy Optimization for Coding Agent Training." pith.science (2026). https://pith.science/paper/GDHHNTTH

@misc{pith2026260807147,
  author       = {Pith},
  title        = {Pith review of: DiDPO: Diff-in-Diff Policy Optimization for Coding Agent Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GDHHNTTH}},
  note         = {Machine review of arXiv:2608.07147}
}
read the original abstract

Reinforcement learning with Verifiable Reward (RLVR) has emerged as a powerful paradigm for training coding agents, where the execution feedback from compilation and tests provides objective verification. However, unlike agent tasks, coding agents face a unique and finer-grained credit assignment challenge: at each step, coding actions simultaneously pack varying changes into different regions of a code version, which makes the contribution of independent change indistinguishable. Existing RLVR methods mostly leverage the outcome reward or step-level reward, which fails to dive into a code diff and makes unique properties of coding actions invisible to training. In this paper, we propose Diff-in-Diff Policy Optimization (DiDPO), a critic-free RL method that constructs fine-grained credit units directly from the structure of code diffs. DiDPO organizes multi-turn coding interactions into multiple thought--action steps and discovers code diffs across sampled trajectories. It then selects anchors by aggregating highly similar sub-diffs split from each whole diff by our ``groupability score'', which provides the splitting schema that optimally balances the semantic scope of anchors and the group mass they may form. Finally these anchors form advantage groups and project the diff-level advantage back to individual response tokens. Experiments on long-horizon coding and reasoning benchmarks show that DiDPO significantly outperforms strong agentic RL baselines. On Qwen2.5-7B-Coder, DiDPO exceeds comparable methods by over 10\% and narrows the gap with far larger models, offering a principled framework for fine-grained credit assignment in coding agent training. We also open-source verl-code, an agentic rl codebase that supports various RL methods and coding benchmarks.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 31 canonical work pages

  1. [1]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Do- han, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

  2. [2]

    A general theo- retical paradigm to understand learning from hu- man preferences

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theo- retical paradigm to understand learning from hu- man preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447–

    work page

  3. [3]

    A course in metric geometry, volume 33

    Dmitri Burago, Yuri Burago, Sergei Ivanov, et al. A course in metric geometry, volume 33. American Mathematical Society Providence, 2001

    work page 2001

  4. [4]

    Minimax-m1: Scaling test- time compute efficiently with lightning attention

    Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, et al. Minimax-m1: Scaling test- time compute efficiently with lightning attention. arXiv preprint arXiv:2506.13585, 2025

    arXiv 2025

  5. [5]

    Codet: Code generation with generated tests.arXiv preprint arXiv:2207.10397, 2022

    Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests.arXiv preprint arXiv:2207.10397, 2022

    arXiv 2022

  6. [6]

    Reinforcement learning for long-horizon interactive llm agents

    Kevin Chen, Marco Cusumano-Towner, Brody Hu- val, Aleksei Petrenko, Jackson Hamburger, Vladlen Koltun, and Philipp Krähenbühl. Reinforcement learning for long-horizon interactive llm agents. arXiv preprint arXiv:2502.01600, 2025

    arXiv 2025

  7. [7]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

  8. [8]

    Teaching large language models to self-debug

    Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. InInternational Conference on Learning Representations, volume 2024, pages 8746–8825, 2024

    work page 2024

Show all 62 references
  1. [9]

    Mem0: Building production-ready ai agents with scalable long-term memory

    Prateek Chhikara, Dev Khant, Saket Aryan, Taran- jeet Singh, and Deshraj Yadav. Mem0: Building production-ready ai agents with scalable long-term memory. arXiv preprint arXiv:2504.19413, 2025

    2025 arXiv

  2. [10]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom Brown, Mil- jan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

    2017

  3. [11]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias 9 Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  4. [12]

    Process rein- forcement through implicit rewards.arXiv preprint arXiv:2502.01456, 2025

    Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Yuchen Zhang, Jiacheng Chen, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, et al. Process rein- forcement through implicit rewards.arXiv preprint arXiv:2502.01456, 2025

    2025 arXiv

  5. [13]

    Memp: Exploring agent procedural memory

    Runnan Fang, Yuan Liang, Xiaobin Wang, Jialong Wu, Shuofei Qiao, Pengjun Xie, Fei Huang, Huajun Chen, and Ningyu Zhang. Memp: Exploring agent procedural memory. InFindings of the Association for Computational Linguistics: ACL 2026, pages 17490–17502, 2026

    2026

  6. [14]

    Group-in-group policy optimization for llm agent training

    Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for llm agent training. AdvancesinNeuralInformationProcessing Systems, 38:46375–46408, 2026

    2026

  7. [15]

    Incoder: A generative model for code infilling and synthesis

    Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen- tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999, 2022

    2022 arXiv

  8. [16]

    Deepseek-r1: In- centivizing reasoning capability in llms via reinforce- ment learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: In- centivizing reasoning capability in llms via reinforce- ment learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  9. [17]

    Skywork open reasoner 1 technical report

    Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxi- ang Zhang, Jiacheng Xu, Wei Shen, et al. Skywork open reasoner 1 technical report. arXiv preprint arXiv:2505.22312, 2025

    2025 arXiv

  10. [18]

    Measuring coding challenge competence with apps

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938, 2021

    2021 arXiv

  11. [19]

    Cogagent: A visual language model for gui agents

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14281–14290, 2024

    2024

  12. [20]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Day- iheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder techni- cal report. arXiv preprint arXiv:2409.12186, 2024

    2024 arXiv

  13. [21]

    Live- codebench: Holistic and contamination free eval- uation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Live- codebench: Holistic and contamination free eval- uation of large language models for code. arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  14. [22]

    Cure: Code-aware neural machine translation for auto- matic program repair

    Nan Jiang, Thibaud Lutellier, and Lin Tan. Cure: Code-aware neural machine translation for auto- matic program repair. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), pages 1161–1173. IEEE, 2021

    2021

  15. [23]

    Self-planning code generation with large language models

    Xue Jiang, Yihong Dong, Lecheng Wang, Zheng Fang, Qiwei Shang, Ge Li, Zhi Jin, and Wenpin Jiao. Self-planning code generation with large language models. ACMTransactionsonSoftwareEngineering and Methodology, 33(7):1–30, 2024

    2024

  16. [24]

    Coderl+: Improving code generation via reinforcement with execution semantics alignment

    Xue Jiang, Yihong Dong, Mengyang Liu, Hongyi Deng, Tian Wang, Yongding Tao, Rongyu Cao, Binhua Li, Zhi Jin, Wenpin Jiao, et al. Coderl+: Improving code generation via reinforcement with execution semantics alignment. arXiv preprint arXiv:2510.18471, 2025

    arXiv

  17. [25]

    Swe-bench: Can language models re- solve real-world github issues? In International Conference on Learning Representations, volume 2024, pages 54107–54157, 2024

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models re- solve real-world github issues? In International Conference on Learning Representations, volume 2024, pages 54107–54157, 2024

    2024

  18. [26]

    Defects4j: A database of existing faults to enable controlled testing studies for java programs

    René Just, Darioush Jalali, and Michael D Ernst. Defects4j: A database of existing faults to enable controlled testing studies for java programs. In Proceedings ofthe2014internationalsymposium on software testing and analysis, pages 437–440, 2014

    2014

  19. [27]

    Ds- 1000: A natural and reliable benchmark for data science code generation

    Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds- 1000: A natural and reliable benchmark for data science code generation. InInternational Conference on Machine Learning, pages 18319–18345. P...

    2023

  20. [28]

    Coderl: Mastering code generation through pretrained mod- els and deep reinforcement learning

    Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Sil- vio Savarese, and Steven Chu Hong Hoi. Coderl: Mastering code generation through pretrained mod- els and deep reinforcement learning. Advances in Neural Information Processing Systems, 35:21314– 21328, 2022

    2022

  21. [29]

    A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each

    Claire Le Goues, Michael Dewey-Vogt, Stephanie Forrest, and Westley Weimer. A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each. In2012 34th international conference on software engineering (ICSE), pages 3–13. IEEE, 2012

    2012

  22. [30]

    Meta- harness: End-to-end optimization of model harnesses

    Yoonho Lee, Roshen Nair, Qizheng Zhang, Kang- wook Lee, Omar Khattab, and Chelsea Finn. Meta- harness: End-to-end optimization of model harnesses. arXiv preprint arXiv:2603.28052, 2026. 10

    2026 arXiv

  23. [31]

    Starcoder: may the source be with you!arXiv preprint arXiv:2305.06161, 2023

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you!arXiv preprint arXiv:2305.06161, 2023

    2023 arXiv

  24. [32]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Har- rison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InInternationalConference on Learning Representations, volume 2024, pages 39578–39601, 2024

    2024

  25. [33]

    Automatic patch gen- eration by learning correct code

    Fan Long and Martin Rinard. Automatic patch gen- eration by learning correct code. In Proceedings of the 43rd annual ACM SIGPLAN-SIGACT symposium on principles of programming languages, pages 298–312, 2016

    2016

  26. [34]

    Skillclaw: Let skills evolve collectively with agentic evolver

    Ziyu Ma, Shidong Yang, Yuxiang Ji, Xucong Wang, Yong Wang, Yiming Hu, Tongwen Huang, and Xiangxiang Chu. Skillclaw: Let skills evolve collectively with agentic evolver. arXiv preprint arXiv:2604.08377, 2026

    2026 arXiv

  27. [35]

    Gromov–wasserstein distances and the metric approach to object matching

    Facundo Mémoli. Gromov–wasserstein distances and the metric approach to object matching. Foundations of computational mathematics, 11(4): 417–487, 2011

    2011

  28. [36]

    An analysis of approxima- tions for maximizing submodular set functions—i

    George L Nemhauser, Laurence A Wolsey, and Marshall L Fisher. An analysis of approxima- tions for maximizing submodular set functions—i. Mathematical programming, 14(1):265–294, 1978

    1978

  29. [37]

    Training language models to follow in- structions with human feedback

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow in- structions with human feedback. arXiv preprint arXiv:2203.02155, 2022

    2022 arXiv

  30. [38]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Ste- fano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2023

    2023 arXiv

  31. [39]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  32. [40]

    Can language models solve olympiad programming? arXiv preprint arXiv:2404.10952, 2024

    Quan Shi, Michael Tang, Karthik Narasimhan, and Shunyu Yao. Can language models solve olympiad programming? arXiv preprint arXiv:2404.10952, 2024

    2024 arXiv

  33. [41]

    Reflexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems, 36:8634–8652, 2023

    2023

  34. [42]

    Alfworld: Aligning text and embodied environments for interactive learning.arXiv preprint arXiv:2010.03768, 2020

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning.arXiv preprint arXiv:2010.03768, 2020

    2010 arXiv

  35. [43]

    Ex- ecverify: White-box rl with verifiable stepwise re- wards for code execution reasoning

    Lingxiao Tang, He Ye, Zhaoyang Chu, Muyang Ye, Zhongxin Liu, Xiaoxue Ren, and Lingfeng Bao. Ex- ecverify: White-box rl with verifiable stepwise re- wards for code execution reasoning. InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Vo...

    2026

  36. [44]

    Ex- ecutable code actions elicit better llm agents

    Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Ex- ecutable code actions elicit better llm agents. In Forty-first International Conference on Machine Learning, 2024

    2024

  37. [45]

    Open- hands: An open platform for ai software developers as generalist agents

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Open- hands: An open platform for ai software developers as generalist agents. In International Conference on Learning Representations, volume...

    2025

  38. [46]

    Lightweight self-knowledge distillation with multi- source information fusion

    Xucong Wang, Pengchao Han, and Lei Guo. Lightweight self-knowledge distillation with multi- source information fusion. arXiv preprint arXiv:2305.09183, 2023

    2023 arXiv

  39. [47]

    Multi-label self knowledge distillation

    Xucong Wang, Pengkun Wang, Shurui Zhang, Miao Fang, and Yang Wang. Multi-label self knowledge distillation. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 21330– 21338, 2025

    2025

  40. [48]

    Role-agent: Bootstrapping llm agents via dual- role evolution

    Xucong Wang, Ziyu Ma, Shidong Yang, Tongwen Huang, Pengkun Wang, Yong Wang, and Xiangxiang Chu. Role-agent: Bootstrapping llm agents via dual- role evolution. arXiv preprint arXiv:2606.10917, 2026

    2026 arXiv

  41. [49]

    Ojbench: A competi- tion level code benchmark for large language models

    Zhexu Wang, Yiping Liu, Yejie Wang, Wenyang He, Bofei Gao, Muxi Diao, Yanxu Chen, Kelin Fu, Flood Sung, Zhilin Yang, et al. Ojbench: A competi- tion level code benchmark for large language models. arXiv preprint arXiv:2506.16395, 2025

    2025

  42. [50]

    Ra- gen: Understanding self-evolution in llm agents via multi-turn reinforcement learning

    Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, et al. Ra- gen: Understanding self-evolution in llm agents via multi-turn reinforcement learning. arXiv preprint arXiv:2504.20073, 2025

    2025 arXiv

  43. [51]

    Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824– 24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, 11 Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824– 24837, 2022

    2022

  44. [52]

    Swe-rl: Advancing llm reasoning via reinforce- ment learning on open software evolution.Advances inNeuralInformationProcessing Systems, 38:78500– 78525, 2026

    Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida Wang. Swe-rl: Advancing llm reasoning via reinforce- ment learning on open software evolution.Advances inNeuralInformationProcessing Systems,...

    2026

  45. [53]

    Agentless: Demystifying llm- based software engineering agents.arXiv preprint arXiv:2407.01489, 2024

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying llm- based software engineering agents.arXiv preprint arXiv:2407.01489, 2024

    2024 arXiv

  46. [54]

    Leetcodedataset: A temporal dataset for robust eval- uation and efficient training of code llms

    Yunhui Xia, Wei Shen, Yan Wang, Jason Klein Liu, Huifeng Sun, Siyue Wu, Jian Hu, and Xiaolong Xu. Leetcodedataset: A temporal dataset for robust eval- uation and efficient training of code llms. arXiv preprint arXiv:2504.14655, 2025

    2025 arXiv

  47. [55]

    Icpc-eval: Probing the frontiers of llm reasoning with competitive pro- gramming contests.Advancesin Neural Information Processing Systems, 38, 2026

    Shiyi Xu, Hu Yiwen, Yingqian Min, Zhipeng Chen, Xin Zhao, and Ji-Rong Wen. Icpc-eval: Probing the frontiers of llm reasoning with competitive pro- gramming contests.Advancesin Neural Information Processing Systems, 38, 2026

    2026

  48. [56]

    Swe-agent: Agent-computer inter- faces enable automated software engineering

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. Swe-agent: Agent-computer inter- faces enable automated software engineering. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  49. [57]

    Webshop: Towards scalable real-world web interaction with grounded language agents

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. AdvancesinNeuralInformationProcessing Systems, 35:20744–20757, 2022

    2022

  50. [58]

    React: Syn- ergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Syn- ergizing reasoning and acting in language models. In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022

    2022

  51. [59]

    Dapo: An open- source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open- source llm reinforcement learning system at scale. AdvancesinNeuralInformationProcessing Systems, 38:113222–113244, 2026

    2026

  52. [60]

    Group sequence pol- icy optimization

    Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence pol- icy optimization. arXiv preprint arXiv:2507.18071, 2025

    2025 arXiv

  53. [61]

    A syntax-guided edit decoder for neural pro- gram repair

    Qihao Zhu, Zeyu Sun, Yuan-an Xiao, Wenjie Zhang, Kang Yuan, Yingfei Xiong, and Lu Zhang. A syntax-guided edit decoder for neural pro- gram repair. In Proceedings of the 29th ACM joint meeting on European software engineering conference and symposium on the foundations of softw...

    2021

  54. [62]

    Reported subsets

    Siyuan Zhu, Chao Yu, Rongxin Yang, Zongkai Liu, Jinjun Hu, Qiwen Chen, and Yibo Zhang. Gagpo: Generalized advantage grouped policy optimization. arXiv preprint arXiv:2605.13217, 2026. 12 A Proofs for Theoretical Foundations A.1 Proof of Theorem 1 Proof A.1 Fix one prompt and c...

    2026 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.