Boosting Open-Source LLMs for Program Repair via Reasoning Transfer and LLM-Guided Reinforcement Learning

Jacques Klein; Tegawend\'e F. Bissyand\'e; Xunzhu Tang

REVIEW 5 major objections 6 minor 1 cited by

Boosting Open-Source LLMs for Program Repair via Reasoning Transfer and LLM-Guided Reinforcement Learning

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Repairity claims that filtering a closed model's test-passing reasoning traces, fine-tuning an open model on them, and then reinforcing with that closed model's preferences brings open-source program repair to near-parity with…

desk verdict Promising recipe, but the SWE-bench parity claim rests on split arithmetic the paper never pins down. read the letter →

arxiv 2506.03921 v1 pith:K6PMQB5J submitted 2025-06-04 cs.SE

Xunzhu Tang , Jacques Klein , Tegawendé F. Bissyandé This is my paper

classification cs.SE

keywords automatedprogramrepairlargelanguagemodelsreasoningtransferknowledgedistillationreinforcementlearningfromAIfeedbacksupervisedfine-tuningQwen2.5-CoderSWE-benchVerified

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Repairity claims that open-source LLMs can be brought to near-parity with closed frontier models on program repair without using the closed model's weights or training data. The recipe is to extract step-by-step reasoning traces from Claude-Sonnet3.7, keep only traces whose repaired code passes the test suite, fine-tune Qwen2.5-Coder-32B-Instruct on those traces, and then run reinforcement learning with a reward model trained on Claude's pairwise preferences between candidate repairs. On SWE-bench Verified the method raises the resolved rate from 38.2% to 62.7%, and across four benchmarks it cuts the average gap with Claude-Sonnet3.7 from 10.05 to 1.35 percentage points. If these numbers hold, organizations that need privacy, transparency, or local deployment could get near-frontier repair quality from a model they can inspect and host.

What carries the argument

The load-bearing mechanism is the RL-LF loop, Reinforcement Learning with LLM Feedback: a reward model trained on pairwise preferences that Claude-Sonnet3.7 expresses between candidate repairs, used as the reward signal for PPO fine-tuning of the supervised-fine-tuned model. The other half is the filtered reasoning-trace SFT dataset, built by keeping only Claude outputs whose generated code passes all tests; the reasoning traces are the medium through which bug identification and repair planning are transferred before reinforcement learning refines them. The pairing of demonstration learning from traces with preference optimization from LLM judgments is what the paper says closes the gap.

What would settle it

A clean re-run with fully disjoint splits and a single shared harness would settle it: if the resolved rate falls materially below 62.7% on SWE-bench Verified, or if Claude's 62.3% default-mode score was produced under different scaffolding, the claim of parity is not supported.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that an open 32B model can be taught to repair code almost as well as a much larger closed model by transferring not just correct outputs but the reasoning that produces them, then reinforcing that reasoning with preferences supplied by the same closed model acting as judge. Repairity filters teacher traces by test-passing correctness, fine-tunes the student on the surviving input-trace-output triples, and then applies PPO with a CodeLlama-7b reward model trained on Claude-Sonnet3.7's pairwise judgments of candidate repairs. The headline numbers are a 24.5-point absolute jump on SWE-bench Verified (38.2% to 62.7%, slightly above Claude-Sonnet3.7's default-mode score of 62.3%) and an average gap reduction from 10.05 to 1.35 percentage points across BigCodeBench, MBPP, SWE-bench Verified, and Defects4J. Ablations attribute the larger share of the gain to the reinforcement-learning stage rather than to reasoning-trace supervised fine-tuning alone.

Load-bearing premise

The headline gap-closing numbers depend on the evaluation split being disjoint from the 20% subset used for supervised fine-tuning and from the validation set used to train the reward model, and on the Claude baseline being measured with the same patch-application and scaffolding harness as Repairity.

Editorial extensions

If this is right

If Repairity's numbers hold, a 32B open-weights model can match a frontier closed model on several code benchmarks, so near-parity does not require a larger open model or access to closed weights.
The largest gain, +24.5 absolute points on SWE-bench Verified, occurs on the most complex task, so this kind of reasoning transfer should be tried first where bug localization and multi-step repair dominate.
The ablations imply that the LLM-judged reinforcement stage, not the reasoning traces alone, drives most of the improvement, so further gains should come from better reward models and preference data rather than more supervised fine-tuning data.
Cross-benchmark transfer, with 32.7% Pass@1 on full BigCodeBench after SWE-bench-only training versus 30.8% for the base model, means the method can improve held-out code tasks even when it is not fine-tuned on them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper, the reported 'exceeds Claude default' result on SWE-bench is only meaningful if the Claude baseline used the same scaffolding and patch-application harness; the paper lists two Claude scores, default and with scaffold, so a same-harness rerun could change the comparison.
Beyond the paper, because the teacher and judge are the same closed model, Repairity's open result still depends on a proprietary API; a fully self-sustaining open pipeline would need the improved open model itself to generate the next round's traces and preferences.
Beyond the paper, the reward model's reported 85% preference accuracy suggests the closed judge's quality preferences are learnable, so the same RL-LF recipe could transfer to other quality-sensitive code tasks such as test generation or code review where pairwise judgments can be elicited.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague — This paper is a composition of three known techniques — filtered reasoning-trace distillation, SFT, and RL with an LLM judge — applied to program repair. That is not a knock; the pipeline is clearly described, hyperparameters are concrete, and the ablation data suggest both the reasoning-trace SFT and the RL-LF stage add something. If the reported numbers hold, the result is practically useful: a 32B open model near Claude 3.7 on SWE-bench is something people want. That is why the paper deserves a serious look.

The soft spots are mostly in the numbers, and one is load-bearing. The abstract and Section 5 compare Repairity's SWE-bench Verified 62.7% against Claude 3.7's default 62.3%, and claim the gap closed from 10.05% to 1.35%. But Section 3.2.2 says only 20% of the benchmark was used for SFT, Section 4.4 describes a 70/15/15 split into training/validation/test, and Section 5.4 says the cross-benchmark experiment was fine-tuned on all 500 SWE-bench data points. Those statements do not fit together. If the test set is 15% of the 20% subset, that is about 75 instances, and the Claude baseline was computed on all 500. If the test set is the remaining 80%, the Claude number is still on all 500. Either way the 62.7 vs 62.3 comparison is not controlled, and the same ambiguity affects the other benchmarks because the paper never states which instances each baseline ran on. This is not a minor footnote; it is the basis of the 'closing 98% of the gap' claim.

Other issues are real but smaller. Section 4.4 promises significance testing; no error bars or significance results appear. Section 4.3 lists two ablations (direct-output fine-tuning and SFT without filtering); the results section never reports them. The 20% subset selection is not described, so selection bias is possible. The 'novel' language in the introduction and Section 3.5 oversells the composition, but that is typical and not the main problem.

The core idea is plausible and the improvements are large enough that the direction is likely right. I would not accept the quantitative claims as established, but the paper is worth refereeing — a good referee can force the authors to pin down the split arithmetic, rerun the Claude baselines on the exact test instances, and report the missing ablations and error bars. That is a paper-shaped object.

Referee Report

5 major / 6 minor

Summary. The paper proposes Repairity, a three-stage method to improve open-source LLMs for program repair by (1) extracting and correctness-filtering reasoning traces from a closed-source teacher model (Claude-Sonnet3.7), (2) supervised fine-tuning an open-source student (Qwen2.5-Coder-32B-Instruct) on those traces, and (3) optimizing the student with PPO using a reward model trained on preference judgments from the same closed-source model. The authors report large gains across BigCodeBench, SWE-bench Verified, MBPP, and Defects4J, including a 24.5-point absolute improvement on SWE-bench Verified, and claim near-parity with Claude-Sonnet3.7. Ablations are promised for reasoning-trace learning and correctness filtering. The central empirical claim is that reasoning-trace distillation plus LLM-judge reinforcement learning can bring a 32B open model close to a frontier closed model.

Significance. If the reported results are valid, the contribution is practically significant: an open-weights 32B model reaching 62.7% on SWE-bench Verified would be a strong result for reproducible program repair, and the three-stage pipeline is simple enough to be adopted by others. The paper has several genuine strengths: evaluation uses external benchmark test suites rather than the teacher's judgments, the SFT and PPO objectives are stated explicitly, implementation details are given, and the authors commit to releasing artifacts. However, the current evidence does not yet support the headline claims. The evaluation-split arithmetic is ambiguous or contradictory, two explicitly promised ablations are missing, no statistical uncertainty is reported despite a claimed significance-testing procedure, and the gap-closing figures in the abstract are not reproducible from the tables. These issues are load-bearing because the main claims depend on controlled comparisons and on the ablations.

major comments (5)

[§3.2.2, §4.4, §5.4, Tables 2–3] The manuscript does not establish that the SWE-bench Verified evaluation set is disjoint from the SFT subset and from the reward-model/PPO validation set. Section 3.2.2 says the filtered dataset is limited to 20% of benchmark size; Section 4.4 says the collected data is split 70/15/15 into training, validation, and test, with the test set used for final evaluation; Section 5.4 says a model was fine-tuned on SWE-bench Verified (500 data points). Under the first reading, the main SWE-bench result of 62.7% is computed on at most 15% of a 20% subset (roughly 15 instances), and under the second reading the evaluation set overlaps the training set. In either case, the comparison with the cited Claude-Sonnet3.7 default of 62.3%, which is a full-benchmark number, is not controlled. Please specify exactly which instances are used for SFT, reward-model training, and final evaluation, and rerun Claude-Sonnet3.7 under the identical evaluation harness and split.
[§4.4, §5] Section 4.4 step 7 states that significance testing is performed 'to validate our findings,' but no confidence intervals, p-values, or significance-test outputs appear anywhere in Section 5. Several headline gains are based on small absolute differences (e.g., BigCodeBench 30.8 to 35.6 Pass@1; Defects4J Pass@1 64.0 to 66.5), so without uncertainty estimates the reader cannot judge whether these improvements are stable. Please report per-benchmark confidence intervals and significance tests, or explicitly state that the differences are not statistically significant.
[§4.3, §5.2] Section 4.3 lists two ablation variants: Direct Output Fine-tuning and SFT without Filtering. Section 5.2 reports only the base model, the model with Reasoning Trace learning, and the complete model with RT plus RL-LF. Without the direct-output ablation, the claim that reasoning traces, rather than additional SFT data, drive the gains is unsupported; without the no-filtering ablation, the claim that correctness filtering matters is unsupported. Please add these ablation results or remove the corresponding claims from the abstract and Section 5.2.
[Abstract, §5.1, Tables 2–5] The abstract's headline figures 'reducing the capability gap ... from 10.05% to 1.35%' are not reproducible from the tables under a stated aggregation. Using the numbers in Tables 2–5 and averaging the absolute gaps for Pass@1/Resolved/Accuracy, the gap changes from 8.25 to -0.63 if Claude-Sonnet3.7 default (62.3%) is used for SWE-bench, and from 10.25 to 1.38 if the scaffolded number (70.3%) is used. The text should state which Claude configuration and which metric aggregation produce the abstract figures; as written, the headline gap-closing claim is ambiguous and appears to mix the default and scaffolded Claude baselines.
[§3.4.2, §3.4.4] The reward model is trained on preference judgments from Claude-Sonnet3.7, the same model that generated the SFT reasoning traces, and this reward model is then used as the PPO reward in Eq. (14). This is not definitionally circular because final evaluation uses external test suites, but it is a bias risk: the RL stage may align the student to Claude's stylistic preferences rather than to functional repair quality. Please report agreement of the CodeLlama-7b reward model with held-out human preferences or with test-suite-based correctness, and/or ablate with a test-suite-based reward to show that the RL-LF gains are not an artifact of the specific judge.

minor comments (6)

[§4.1, References] The SWE-bench Verified citation [26] points to 'SweBench: Benchmarking Large Language Models for Swedish,' which is a different benchmark; the correct SWE-bench citation should be used.
[§4.1, Table 1] The MBPP sample count is 974 in Table 1 but 964 in the prose of Section 4.1; these numbers should be reconciled.
[§6.2] There is an unresolved placeholder '[?]' in the limitation 'perhaps by identifying and focusing on the most informative parts of reasoning traces [?]'; a reference or explanation should be added.
[§3.3.2, §3.6, §4.5] The SFT training epochs are stated as 3 in Section 3.3.2 and Section 3.6, but Section 4.5 says 'SFT Model Training: Approximately 3 hours on 8×H100 GPUs with 5 epochs.' Please make the epoch count consistent.
[§3.3.2, §3.6] The SFT sequence length is given as 'Maximum of 4096 tokens' in Section 3.3.2 but as '2048-token context windows' in Section 3.6; please clarify which value was used.
[§3.4.1] The phrase 'differs from standard RLHF1' should be formatted as a footnote marker or written out as 'RLHF'; the current inline '1' is confusing.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Repairity's gains are measured on held-out external benchmark test suites, and the teacher-as-judge dependency is a methodological bias risk rather than a definitional reduction.

full rationale

The paper's central claims are empirical and are validated against held-out portions of external benchmarks (BigCodeBench, SWE-bench Verified, MBPP, Defects4J) using test-case-based metrics such as Pass@1, Resolved, Accuracy, and BLEU (Section 4.1-4.2, Tables 2-5). The SFT objective in Eq. (7) maximizes the likelihood of teacher reasoning traces and repaired code, and the RL-LF stage in Eqs. (12)-(14) optimizes a reward model trained on Claude preference labels, but neither objective defines the evaluation metric. Evaluation is performed on disjoint held-out sets per the stated procedure in Section 4.4, steps 2 and 7. The fact that Claude-Sonnet3.7 is both the reasoning-trace teacher and the preference judge raises a legitimate concern about aligning the student to one closed model's style, but this is not circular: the benchmark tests can falsify the improvement claims independently of the training signal. There are no load-bearing self-citations; the cited prior work on chain-of-thought, distillation, and RLAF is external. The ambiguity between the '20% of benchmark size' training subset (Section 3.2.2), the 70/15/15 data split (Section 4.4), and the statement that the model was fine-tuned on all 500 SWE-bench Verified points (Section 5.4) is a reproducibility and baseline-comparability risk, not a circular derivation. No prediction is equivalent to its input by construction.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central results rest on several domain assumptions: benchmark correctness is a valid repair signal, Claude's preferences are a good reward signal, the selected subset and split are representative and leakage-free, and published baseline numbers are protocol-comparable. The only new constructs are training stages, not natural entities, so no entities with independent evidence are introduced.

free parameters (5)

SFT subset size = 20% of each benchmark
Chosen by the authors in Section 3.2.2 without describing the selection procedure; directly determines training data composition.
PPO KL coefficient beta = 0.1
Set by hand in Section 3.6; controls how far the policy can drift from the SFT model in Eq. 14.
LoRA rank and alpha = rank=4, alpha=16
Hand-selected adaptation hyperparameters in Section 3.6; no sensitivity analysis.
Reward model architecture = CodeLlama-7b
Chosen as the reward model in Section 3.6; no comparison to other reward models or sizes.
Candidate repairs per problem for preferences = k=3
Sets the size of the preference dataset in Section 3.4.2; no analysis of how k affects reward model quality.

assumptions (5)

standard math Bradley-Terry preference model and PPO/GAE update rules are valid for training the policy.
Used in Sections 3.4.3 to 3.4.5 without proof; accepted ML machinery.
domain assumption Passing the provided test suites is a sufficient measure of repair correctness.
Used to filter SFT data (Eq. 5) and to compute Pass@1 and Resolved in Section 4.2.
domain assumption Claude-Sonnet3.7's pairwise preferences align with human judgments of repair quality and provide a stable training signal.
Core to RL-LF reward data collection in Section 3.4.2; no human agreement study is reported.
domain assumption The 20% benchmark subset and the 70/15/15 split are representative and contamination-free.
Section 3.2.2 and Section 4.4; the selection procedure is not described, so this is an unverified premise.
domain assumption Published baseline numbers in Tables 2-5 are protocol-comparable with Repairity's evaluation.
Section 5.1 uses external model scores directly; scaffolding and harness details for SWE-bench are not reported for Repairity.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Boosting Open-Source LLMs for Program Repair via Reasoning Transfer and LLM-Guided Reinforcement Learning." pith.science (2026). https://pith.science/paper/K6PMQB5J

@misc{pith2026250603921,
  author       = {Pith},
  title        = {Pith review of: Boosting Open-Source LLMs for Program Repair via Reasoning Transfer and LLM-Guided Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K6PMQB5J}},
  note         = {Machine review of arXiv:2506.03921}
}

read the original abstract

Several closed-source LLMs have consistently outperformed open-source alternatives in program repair tasks, primarily due to their superior reasoning capabilities and extensive pre-training. This paper introduces Repairity, a novel three-stage methodology that significantly narrows this performance gap through reasoning extraction and reinforcement learning. Our approach: (1) systematically filters high-quality reasoning traces from closed-source models using correctness verification, (2) transfers this reasoning knowledge to open-source models via supervised fine-tuning, and (3) develops reinforcement learning with LLM-based feedback to further optimize performance. Empirical evaluation across multiple program repair benchmarks demonstrates that Repairity improves the performance of Qwen2.5-Coder-32B-Instruct, a base open source LLM, by 8.68\% on average, reducing the capability gap with Claude-Sonnet3.7, a state-of-the-art closed-source model, from 10.05% to 1.35%. Ablation studies confirm that both reasoning extraction and LLM-guided reinforcement learning contribute significantly to these improvements. Our methodology generalizes effectively to additional code-related tasks, enabling organizations to leverage high-quality program repair capabilities while maintaining the customizability, transparency, and deployment flexibility inherent to open-source models.

Figures

Figures reproduced from arXiv: 2506.03921 by the authors.

**Figure 1.** Overview of the Repairity 3.3.3 Ablation: Direct Output Fine-tuning. To validate the importance of reasoning traces, we conduct an ablation study with a variant model MDO 𝑂 that is fine-tuned only on direct outputs without reasoning: LDO = − ∑︁ (𝑥𝑖 ,𝑟𝑖 ,𝑦ˆ𝑖 ) ∈DSFT log 𝑃M𝑂 (𝑦ˆ𝑖 | 𝑥𝑖) (8) This allows us to isolate the specific contribution of reasoning trace learning to model performance. 3.4 Step 3: Reinforcement … view at source ↗

**Figure 2.** Example model outputs. We pick a input from Big [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

DPO-F+: Aligning Code Repair Feedback with Developers' Preferences
cs.SE 2025-11 reject novelty 5.0 of 10

DPO-f+ fine-tunes LLMs with reward-augmented DPO to generate developer-profile-aware code-repair feedback, claiming gains over DPO that rest on LLM-judge evaluations.

Reference graph

Works this paper leans on

65 extracted references · 35 canonical work pages · cited by 1 Pith paper

[1]

Anthropic. 2023. Claude: A new AI assistant from Anthropic. Retrieved from https://www.anthropic.com/claude

work page 2023
[2]

Anthropic. 2025. Claude 3.7 Sonnet: Hybrid Reasoning AI. https://www.anthropic. com/news/claude-3-7-sonnet. Accessed: March 14, 2025

work page 2025
[3]

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program Synthesis with Large Language Models. InarXiv preprint arXiv:2108.07732

arXiv 2021
[4]

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. Constitutional AI: Harmlessness from AI Feedback.arXiv preprint arXiv:2212.08073(2022)

arXiv 2022
[5]

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258(2021)

arXiv 2021
[6]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners.Advances in neural information processing systems33 (2020), 1877–1901

2020
[7]

Tuan-Dung Bui, Duc-Thieu Luu-Van, Thanh-Phat Nguyen, Thu-Trang Nguyen, Son Nguyen, and Hieu Dinh Vo. 2024. Rambo: Enhancing rag-based repository- level method body completion.arXiv preprint arXiv:2409.15204(2024)

work page Pith review arXiv 2024
[8]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al

work page

Show all 65 references

[9]

Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. 2022. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.arXiv preprint arXiv:2211.12588(2022)

2022 arXiv
[10]

Xinyun Chen, Jerry Tworek, Mira Murati Openai, and Wojciech Zaremba. 2023. Teaching large language models to self-debug.arXiv preprint arXiv:2304.05128 (2023)

2023 arXiv
[11]

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep Reinforcement Learning from Human Preferences. InAd- vances in Neural Information Processing Systems. 4299–4307

2017
[12]

Amirhossein Dakhel, Majid Majdinasab, Amin Nikanjam, and Foutse Khomh
[13]

Rotem Dror, Gal Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitch- hiker’s guide to testing statistical significance in natural language processing. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics. 1383–1392

2018
[14]

Zhiwei Fan, Xinyuan Li, Han Yu, Yuan Xu, Chang Liu, Ting-Yi Liu, and Lingming Wang. 2023. Automated Program Repair with Large Language Models.arXiv preprint arXiv:2307.07359(2023)

2023 arXiv
[15]

Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. The false promise of imitating proprietary LLMs.arXiv preprint arXiv:2305.15717(2023)

2023 arXiv
[16]

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al . 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948(2025)

2025 arXiv
[17]

2009.The elements of statistical learning: data mining, inference, and prediction

Trevor Hastie, Robert Tibshirani, and Jerome Friedman. 2009.The elements of statistical learning: data mining, inference, and prediction. Springer Science & Business Media

2009
[18]

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the Knowledge in a Neural Network.arXiv preprint arXiv:1503.02531(2015)

2015 arXiv
[19]

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. LoRA: Low-Rank Adaptation of Large Language Models. InInternational Conference on Learning Representations

2021
[20]

Yaojie Hu, Qiang Zhou, Qihong Chen, Xiaopeng Li, Linbo Liu, Dejiao Zhang, Amit Kachroo, Talha Oz, and Omer Tripp. 2025. QualityFlow: An Agentic Work- flow for Program Synthesis Controlled by LLM Quality Checks.arXiv preprint arXiv:2501.17167(2025)

2025 arXiv
[21]

Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2. 5-coder technical report.arXiv preprint arXiv:2409.12186(2024)

2024 arXiv
[22]

Md Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. 2024. Map- coder: Multi-agent code generation for competitive problem solving.arXiv preprint arXiv:2405.11403(2024)

2024 arXiv
[23]

Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. Openai o1 system card.arXiv preprint arXiv:2412.16720(2024)

2024 arXiv
[24]

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Deven- dra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Marie- Anne Lachaux, Naiming Gu, et al. 2023. Mistral 7B.arXiv preprint arXiv:2310.06825 (2023)

2023 arXiv
[25]

Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. TinyBERT: Distilling BERT for natural language un- derstanding. InFindings of the Association for Computational Linguistics: EMNLP

2020
[26]

Alexander Jimenez-Sanchez, Alex Fargher, Anton Axelsson, Amanda Felländer, James Watten, Oscar Rudberg, Arvid Kahn, Ariel Calota, Daniel Gillblad, Anders Holst, et al. 2023. SweBench: Benchmarking Large Language Models for Swedish. InarXiv preprint arXiv:2312.16639

2023 arXiv
[27]

Chen Jin, Wenda Lee, Sruti Patel, Jillian Li, Nitya Mathur, Abhinav Mehrotra, Aniruddha Sathish, Aditya Kanade, Dokyun Lee, and Ce Min. 2023. Program of Thoughts: Towards Reasoning Scaffolds for Program Synthesis. InProceedings of the 37th IEEE/ACM International Conference on ...

2023
[28]

René Just, Darioush Jalali, and Michael D Ernst. 2014. Defects4J: A database of existing faults to enable controlled testing studies for Java programs.Proceedings of the 2014 International Symposium on Software Testing and Analysis(2014), 437–440

2014
[29]

Bowen Kim, Daun Kim, Kihyuk Lee, Seong Joon Hwang, and Jinwoo Choi. 2021. Sequence-level knowledge distillation for dense prediction tasks. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7259–7268

2021
[30]

Dongsun Kim, Jaechang Nam, Jaewoo Song, and Sunghun Kim. 2013. Automatic patch generation learned from human-written patches. InProceedings of the 35th International Conference on Software Engineering (ICSE). IEEE, 802–811

2013
[31]

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners.arXiv preprint arXiv:2205.11916(2022)

2022 arXiv
[32]

Wei Lai, Zekun Tian, Hanmer Yu, Xuangui Yan, Xia Chen, and Pengfei Yin. 2023. BigCodeBench: A Systematic Evaluation of Large Language Models for Code Generation. InProceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS)

2023
[33]

Claire Le Goues, ThanhVu Nguyen, Stephanie Forrest, and Westley Weimer. 2012. GenProg: A generic method for automatic software repair.IEEE Transactions on Software Engineering38, 1 (2012), 54–72

2012
[34]

Harrison Lee, Buck Shlegeris, Eric Chan, Roger Grosse, and John X Morris. 2023. RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. arXiv preprint arXiv:2309.00267(2023)

2023 arXiv
[35]

Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. Competition-level code generation with alphacode.Science378, 6624 (2022), 1092–1097. Conference acronym ’XX, June 03–...

2022
[36]

Martin Monperrus. 2018. Automatic software repair: A bibliography.ACM Computing Surveys (CSUR)51, 1 (2018), 1–24

2018
[37]

Hoang Duong Thien Nguyen, Dawei Qi, Abhik Roychoudhury, and Satish Chan- dra. 2013. SemFix: Program repair via semantic analysis. InProceedings of the 35th International Conference on Software Engineering (ICSE). IEEE, 772–781

2013
[38]

Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. CodeGen: An open large language model for code with multi-turn program synthesis. InInternational Conference on Learning Representations (ICLR)

2023
[39]

OpenAI. 2023. GPT-4 Technical Report.arXiv preprint arXiv:2303.08774(2023)

2023 arXiv
[40]

OpenAI. 2025. OpenAI o3-mini System Card. https://openai.com/index/o3-mini- system-card/

2025
[41]

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training Language Models to Follow Instructions with Human Feedback. In Advances in Neural Information Processing Syste...

2022
[42]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. InProceedings of the 40th annual meeting of the Association for Computational Linguistics. 311–318

2002
[43]

Qwen Team. 2023. Qwen Technical Report.arXiv preprint arXiv:2309.16609 (2023)

2023 arXiv
[44]

Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deep- speed: System optimizations enable training deep learning models with over 100 billion parameters. InProceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. ...

2020
[45]

Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Tan, Yossi Liu, Rémi Lebret, Rob Fergus, and Yann LeCun. 2023. Code Llama: Open foundation models for code.arXiv preprint arXiv:2308.12950(2023)

2023 arXiv
[46]

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distil- BERT, a distilled version of BERT: smaller, faster, cheaper and lighter. InNeurIPS 2019 Workshop on Energy Efficient Machine Learning and Cognitive Computing

2019
[47]

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov
[48]

Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize from human feedback. InAdvances in Neural Information Processing Systems

2020
[49]

Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020. MobileBERT: a compact task-agnostic BERT for resource-limited devices. InProceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics. 2158–2170

2020
[50]

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288(2023)

2023 arXiv
[51]

Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou
[52]

Zhiruo Wang, Yunhao Bai, Dinglan Song, Pengfei Yin, Po-Sen Huang, Panupong Pasupat, Yiran Wang, Duen Horng Chau, Aditya Parameswaran, and Percy Liang
[53]

Chunqiu Steven Wei, Ce Min, Thai Farid, and Charles Jin. 2023. Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program Repair. InProceedings of the 31st ACM Joint European Software Engi- neering Conference and Symposium on the Founda...

2023
[54]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of Thought Prompting Elicits Reasoning in Large Language Models. InAdvances in Neural Information Processing Systems, Vol. 35. 24824–24837

2022
[55]

Ting Yuan, Yixuan Li, Junjie Wu, Qi Zhao, and Hongyu Liu. 2023. No more manual tests? Evaluating and improving ChatGPT for unit test generation.Proceedings of the ACM on Programming Languages7, OOPSLA1 (2023), 1–30

2023
[56]

Xingdi Yuan, Xiaoyu Wang, Caglar Wang, Kunal Aggarwal, Gokhan Tur, Lu Hou, Nan Deng, and Hoifung Poon. 2023. Improving code generation by training with natural language feedback.arXiv preprint arXiv:2303.16749(2023)

2023 arXiv
[57]

CodeDistill: Learning to Distill Code Generation Tasks to Large Language Models.arXiv preprint arXiv:2312.00876(2023)

2023 arXiv
[58]

Shun Zhang, Michael Ahn, Shuai Jiang, Ashwin Kumar, Bailin Wang, Zhihong Pang, Tengyang Xia, Pengcheng Yin, Shivanshu Mudgal, Marc Crawford, et al
[59]

Zhiqiang Zhao, Yiling Yuan, Chen Huang, and Wei Zhao. 2023. ChatRepair: Automating Bug Fixing via Large Language Models. InProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering. 1–12

2023
[62]

Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. Repocoder: Repository-level code com- pletion through iterative retrieval and generation.arXiv preprint arXiv:2303.12570 (2023)

2023 arXiv
[64]

Planning with Large Language Models for Code Generation.arXiv preprint arXiv:2305.02309(2023)

2023 arXiv
[2017]

Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347 (2017)

2017 arXiv
[2020]

InAdvances in Neural Information Processing Systems, Vol

MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. InAdvances in Neural Information Processing Systems, Vol. 33. 5776–5788
[2021]

InAdvances in Neural Information Processing Systems

Evaluating Large Language Models Trained on Code. InAdvances in Neural Information Processing Systems
[2023]

InProceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering

GitHub Copilot AI Pair Programmer: Asset or Liability?. InProceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering. 385–396

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Anthropic. 2023. Claude: A new AI assistant from Anthropic. Retrieved from https://www.anthropic.com/claude

work page 2023

[2] [2]

Anthropic. 2025. Claude 3.7 Sonnet: Hybrid Reasoning AI. https://www.anthropic. com/news/claude-3-7-sonnet. Accessed: March 14, 2025

work page 2025

[3] [3]

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program Synthesis with Large Language Models. InarXiv preprint arXiv:2108.07732

arXiv 2021

[4] [4]

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. Constitutional AI: Harmlessness from AI Feedback.arXiv preprint arXiv:2212.08073(2022)

arXiv 2022

[5] [5]

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258(2021)

arXiv 2021

[6] [6]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners.Advances in neural information processing systems33 (2020), 1877–1901

2020

[7] [7]

Tuan-Dung Bui, Duc-Thieu Luu-Van, Thanh-Phat Nguyen, Thu-Trang Nguyen, Son Nguyen, and Hieu Dinh Vo. 2024. Rambo: Enhancing rag-based repository- level method body completion.arXiv preprint arXiv:2409.15204(2024)

work page Pith review arXiv 2024

[8] [8]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al

work page