Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Empirical Evaluation of Large Language Models in Automated Program Repair

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Model specialization can beat raw scale in automated program repair, and most correct patches appear early in sampling.

desk verdict Useful, broad empirical study of open LLMs for APR; claims hold up in the large, but the unblinded manual correctness oracle and post-hoc 30-patch budget need careful reporting before I'd trust the small differences. read the letter →

arxiv 2506.13186 v1 pith:WHRQVUFM submitted 2025-06-16 cs.SE

Jiajun Sun , Fengjie Li , Xinzhu Qi , Hongyu Zhang , Jiajun Jiang This is my paper
classification cs.SE
keywords automatedprogramrepairlargelanguagemodelsempiricalstudypatchgenerationpromptengineeringmodelspecializationrateearlycandidateranking
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is an empirical study asking whether modern open-source LLMs can repair real and algorithmic bugs, and which factors decide success. It evaluates four models from 7B to 33B parameters on six benchmarks in Java, C/C++, and Python, generating over 600,000 patches. Its central claims are that code-specialized models can outperform larger general-purpose models, that repair success does not grow linearly with model size, that correct patches concentrate in the earliest generated candidates, and that prompt content strongly changes outcomes. A sympathetic reader cares because these findings bear directly on how to build cost-effective APR systems: smaller fine-tuned models, early-stopped sampling, and prompt design may matter more than raw scale.

What carries the argument

The load-bearing machinery is the evaluation protocol itself rather than a single mathematical object. Each bug is given perfect fault localization, and each model is prompted to produce candidate patches; patches that pass the dataset's test suite are labeled plausible, then the first two authors manually judge whether each plausible patch is semantically equivalent to the developer's ground-truth patch, producing the repair rate and precision numbers. The second key instrument is the cumulative rank distribution of the first correct patch, which supports the early-candidate finding: by plotting where the first correct patch appears across all bugs, the paper shows diminishing returns after roughly 30 candidates and uses that to cap generation budgets. Prompt templates (basic structure, one-shot and two-shot examples, and an analysis-augmented prompt) are the third mechanism, used to isolate how context changes repair behavior.

What would settle it

Take a random sample of plausible patches from these datasets, have independent annotators who are blind to the generating model classify each as correct or incorrect, and compare against the paper's labels; if agreement falls below a conventional threshold (for instance Cohen's kappa below 0.6) or the relative ordering of models changes, the paper's repair-rate comparisons collapse.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a set of empirical regularities about LLM-based program repair. Across the six datasets, DeepSeek-Coder (33B) usually repairs the most bugs, but the 7B CodeLlama beats the 13B general-purpose LLaMA on every dataset, and StarCoder (15.5B) stays close to 33B DeepSeek-Coder. Repair rates on algorithmic-assignment bugs are far higher than on enterprise-grade project bugs; all models perform notably worse on the C/C++ BugsCpp benchmark. Correct patches appear early: most are found within the top 30 generations, and for several model/dataset combinations all correct patches are within the first 7. Prompt engineering matters: adding one repair example substantially improves weaker models, while adding LLM-generated bug analysis helps weak models but hurts the strongest model, which is misled by flawed diagnoses. The paper concludes that model specialization, early candidate selection, and prompt validation offer practical leverage relative to simply scaling model size.

Load-bearing premise

The entire measurement system rests on the two authors' manual judgment that a plausible patch is semantically equivalent to the developer's fix, and the paper reports no inter-rater reliability, blinding, or independent adjudication for that judgment.

Editorial extensions

If this is right

  • APR pipelines can generate around 30 candidate patches per bug instead of 200 with little loss in repair effectiveness.
  • Smaller code-specialized models such as 7B CodeLlama can replace larger general-purpose models, cutting inference cost without sacrificing repair rate.
  • Combining multiple open-source LLMs can increase total coverage, since each model fixes some bugs that the others cannot.
  • Prompt construction is a first-order design decision: adding a worked repair example improves weak models, while supplying LLM-generated bug analysis helps weak models but can mislead strong ones, so diagnosis content should be validated before use.
  • Repair effectiveness varies strongly by language and benchmark type, so evaluations that rely only on Java enterprise bugs may misrepresent how well an LLM repairs in practice.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to treat the early-candidate concentration as evidence for adaptive sampling: verify candidates as they are generated and stop once the estimated probability of finding a new correct patch is negligible.
  • The complementarity result suggests a portfolio strategy: running a small cheap model for 30 samples and a large model only on bugs the small model cannot fix may match the large model's repair rate at a fraction of the cost.
  • The claimed harm of bug analysis for strong models implies a testable extension: filtering or fact-checking LLM-generated diagnoses before inclusion should recover the losses and may further improve strong models.
  • Because the correctness oracle is manual and unreported in reliability, re-running the comparison with an automated semantic or test-based verdict could substantially change the numerical repair rates, even if the qualitative trends persist.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper reports a large empirical study of four open-source LLMs (CodeLlama-7B, LLaMA-2-13B, StarCoder-15.5B, DeepSeek-Coder-33B) for automated program repair across seven datasets spanning Java, C/C++, and Python, and covering enterprise-grade bugs and algorithmic assignments. The authors generate up to 200 patches per bug (or 30, depending on the research question), validate plausible patches via test suites, manually judge semantic equivalence to ground-truth patches, and then analyze repair rates, precision, rank positions of correct patches, bug-length effects, and prompt strategies. The central findings are that code-specialized models can outperform larger general-purpose models, that repair performance does not scale linearly with model size, that correct patches tend to appear early, and that prompt design significantly affects repair outcomes.

Significance. If the findings are robust, the paper provides valuable evidence on specialization vs. scale in LLM-based APR and practical guidance on cost-aware patch sampling. The experimental scale (over 600,000 patches), the breadth of datasets and languages, and the promise to release artifacts are notable strengths. However, the central claims rest on a manual correctness oracle with no reported reliability, and on a generation budget that was chosen after observing early-patch behavior; these issues affect the measurement foundation of every quantitative conclusion. The paper likely merits publication after substantial revision.

major comments (4)
  1. [Section III-E] The correctness oracle is the measurement foundation of the entire study. Section III-E states that a patch is “correct” only if the first two authors manually judge it semantically equivalent to the ground-truth developer patch. Section V-C adds only that the authors performed “a comprehensive manual analysis” and released the results. No inter-rater reliability metric (e.g., Cohen's kappa), no blinded protocol, and no independent adjudication are reported. Because the annotators know which model produced each patch and what the study hypothesizes, the C/P counts in Tables IV, V, and VI, the precision values, and the rank distributions in Figure 4 could be systematically biased. For example, if plausible patches from CodeLlama are judged more leniently than similar patches from LLaMA, the headline specialization-over-scale finding would be an artifact of the oracle. The paper should report an agreement measure on a sample, blind the annotators to the source model, or have a third annotator adjudicate disagreements. Without this, every quantitative conclusion is contingent on an unvalidated measurement instrument.
  2. [Section III-E and IV-B] The patch generation budget is inconsistent across research questions and was set based on the data being analyzed. Section III-E says that 200 patches per bug were generated for Defects4J and BugsCpp, but after a preliminary analysis (Section IV-B) showed that correct patches appear within the top 30, the budget was reduced to 30 for the algorithmic-assignment datasets. Consequently, the repair rates in Table V (RQ2) are achieved with a 30-patch budget while those in Table IV (RQ1) use a 200-patch budget, so the comparison between enterprise-grade and algorithmic bugs—a recurring theme in the abstract and conclusion—is confounded. Additionally, using the same experiments to both discover the “early patch” phenomenon and evaluate the models under the reduced budget introduces selection bias. The paper should adopt a fixed budget across all RQs, or validate the 30-patch cutoff on an independent set of model/dataset combinations, before claiming that 30 patches recover most of the repair capability.
  3. [Sections IV-A and IV-B] All model comparisons are reported without any statistical significance testing, confidence intervals, or error bars. Some differences are large (e.g., CodeLlama 40 correct vs. LLaMA 19 on Defects4J v1.2), but several are small (e.g., DeepSeek 44 vs. StarCoder 42 on the same dataset; precision 60.0% vs. 61.5% on Defects4J v2.0). Since each bug produces a binary outcome (fixed/not fixed) for each model, a simple paired test such as McNemar's test across models, or a bootstrap interval on repair rate, would give a principled basis for the claims of “complementarity” and “diminishing returns.” The paper currently treats all count differences as meaningful, which is not justified for a study whose stated goal is to provide actionable guidance.
  4. [Sections III-D and IV-D] The bug analysis used in the RQ4 prompt conditions is generated by DeepSeek-Coder, one of the four models under evaluation, and is then fed to all models, including DeepSeek-Coder itself. Because the analysis source is a single model, any peculiarity of DeepSeek-Coder's analysis style (e.g., an inclination toward condition-related explanations) will differentially affect the other models, and DeepSeek-Coder may benefit from self-consistency with its own outputs. The paper reports that analysis improves weak models but hurts DeepSeek-Coder, but this pattern could be partly an artifact of the analysis generator. The authors should either use analyses from a neutral source (e.g., a different LLM or human-written explanations) or at least measure the accuracy of the generated analyses independently and report whether the observed effects persist when controlling for analysis quality.
minor comments (5)
  1. [Abstract and Table I] The abstract says “six benchmarks” but Table I actually lists seven datasets (Defects4J v1.2, Defects4J v2.0, BugsCpp, IntroClass-C, IntroClass-Java, ConDefects-Java, ConDefects-Py). Please correct the count.
  2. [Section VI and IV-B] The conclusion states that models “performed better on Java datasets than on C/C++ datasets,” but on the IntroClass pair the repair rate for C is higher than for Java for all four models (e.g., DeepSeek 45.5% vs. 42.1%). The precision is higher for Java, so the claim should be qualified to refer to precision or to specific datasets.
  3. [Section IV-A] The statement that CodeLlama “demonstrates substantially better repair capabilities” than LLaMA should be restricted to repair count/rate, since CodeLlama's precision is lower than LLaMA's on several algorithmic datasets (e.g., IntroClass-C 90.8% vs. 94.1%).
  4. [Figure 6] Figure 6 contains garbled labels in the provided version (e.g., sequences of “/uni0000…” tokens); the figure should be re-rendered so that the operation names (e.g., “InfixExpression”) are legible.
  5. [Section III-E] The definition of “plausible” as passing “all test cases” should specify whether every dataset provides a complete test suite; for the algorithmic datasets, it is unclear if the withheld tests are used or if the original public tests are sufficient.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; the study is a measured benchmark comparison, with only a minor non-load-bearing self-citation.

full rationale

This is an empirical study, not a derivation, so there is no equation-level input/output identity to inspect. Repair rates, precision, and patch-rank statistics are measured against external benchmarks (Defects4J, BugsCpp, IntroClass, ConDefects) with fixed open-source models, and the headline comparison (CodeLlama outperforming LLaMA despite smaller size) comes from RQ1's 200-patch runs, not from any fitted parameter. The only self-referential elements are: (1) reference [72], an overlapping-authors paper cited only to justify the prompt template and the initial 200-patch sampling budget; this does not carry any of the paper's conclusions, and (2) the RQ4 bug-analysis texts generated by DeepSeek-Coder and then fed to all models, including DeepSeek-Coder itself. That is a methodological confound, but it does not make any reported quantity equal to its input by construction. In particular, the early-patch finding is based on rank distributions from full 200-patch samples (Figure 4) rather than on the later 30-patch budget, and the 30-patch budget was adopted only after that observation. The manual semantic-equivalence oracle is a validity and reliability threat, not a circular dependency. Therefore no circular step meets the evidentiary bar.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new entities are introduced. The main ledger entries are two experiment-design choices that function like fitted parameters: the 30-patch budget selected from preliminary observations, and the choice of DeepSeek-Coder as the analysis generator for all models. The axioms are standard assumptions of benchmark-based empirical software engineering.

free parameters (3)
  • patch_generation_budget (30 per bug) = 30
    Chosen after observing that most correct patches appear within the top 30 (Section IV-B); this post-hoc choice shapes the RQ2 repair rates and the 'early patches' conclusion.
  • bug analysis source (DeepSeek-Coder) = DeepSeek-Coder-33B-instruct
    The analysis used in the 'bug analysis' prompt condition is generated once by one of the evaluated models (Section III-D), with no independent check of analysis quality; this may bias comparisons.
  • decoding hyperparameters (temperature, top-p)
    Not reported in the paper; any comparison across models assumes identical or unspecified sampling settings, which affects patch diversity and correctness.
assumptions (4)
  • domain assumption Benchmark test suites are sufficient to classify a patch as plausible when all tests pass.
    Used throughout Section III-E; if tests are inadequate, 'plausible' overstates correctness.
  • domain assumption Manual semantic-equivalence judgment by the first two authors is an accurate correctness oracle.
    Section III-E; no inter-rater reliability or blinded verification is reported.
  • domain assumption The four selected LLMs are representative of modern open-source LLMs.
    The paper generalizes from CodeLlama, LLaMA, StarCoder, and DeepSeek-Coder to 'LLMs' in its conclusions.
  • domain assumption Data leakage does not systematically bias the relative comparisons.
    Section V-A discusses leakage but performs no contamination check.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Empirical Evaluation of Large Language Models in Automated Program Repair." pith.science (2026). https://pith.science/paper/WHRQVUFM

@misc{pith2026250613186,
  author       = {Pith},
  title        = {Pith review of: Empirical Evaluation of Large Language Models in Automated Program Repair},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WHRQVUFM}},
  note         = {Machine review of arXiv:2506.13186}
}
read the original abstract

The increasing prevalence of software bugs has made automated program repair (APR) a key research focus. Large language models (LLMs) offer new opportunities for APR, but existing studies mostly rely on smaller, earlier-generation models and Java benchmarks. The repair capabilities of modern, large-scale LLMs across diverse languages and scenarios remain underexplored. To address this, we conduct a comprehensive empirical study of four open-source LLMs, CodeLlama, LLaMA, StarCoder, and DeepSeek-Coder, spanning 7B to 33B parameters, diverse architectures, and purposes. We evaluate them across two bug scenarios (enterprise-grades and algorithmic), three languages (Java, C/C++, Python), and four prompting strategies, analyzing over 600K generated patches on six benchmarks. Key findings include: (1) model specialization (e.g., CodeLlama) can outperform larger general-purpose models (e.g., LLaMA); (2) repair performance does not scale linearly with model size; (3) correct patches often appear early in generation; and (4) prompts significantly affect results. These insights offer practical guidance for designing effective and efficient LLM-based APR systems.

Figures

Figures reproduced from arXiv: 2506.13186 by the authors.

Figure 1
Figure 1. Prompt structure [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 3
Figure 3. Unique bugs fixed by LLMs on Defects4J v1.2 & v2.0 [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 2
Figure 2. A bug on BugsCpp similar to Defects4J Moreover, we analyzed the reasons behind the sharp drop in LLMs’ repair performance on the BugsCpp dataset. This drop can be attributed to the limited cross-language generalizability of LLMs. As shown in [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Distribution of Correct Patch Rankings Finding 4: Increasing the number of candidate patches shows diminishing returns in terms of repair effectiveness. In our experiments, generating 30 candidate patches achieves a better balance between repair success rate and comput…
Figure 5
Figure 5. Figure 5: Length distribution of successfully repaired and unre [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Top 10 modification operations in Java, C, and Python [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: An example bug abc229-b from ConDefects-Java dataset [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Unique bugs fixed by LLMs with and without Bug [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Semantic Drift in Bug Resolution: How Behavioral Signals Propagate from Reports to Tests and Patches

    cs.SE 2026-07 conditional novelty 7.0 of 10

    Behavioral signals in bug reports propagate only partially into tests and fixes; alignment is measurable but representation-dependent, and LLM judges are systematically optimistic versus human ratings.

Reference graph

Works this paper leans on

79 extracted references · 56 canonical work pages · cited by 1 Pith paper

  1. [1]

    The debugging mindset: Understanding the psychology of learning strategies leads to effective problem-solving skills

    D. H. O’Dell, “The debugging mindset: Understanding the psychology of learning strategies leads to effective problem-solving skills.”Queue, vol. 15, no. 1, pp. 71–90, 2017

    work page 2017

  2. [2]

    Genprog: A generic method for automatic software repair,

    C. Le Goues, T. Nguyen, S. Forrest, and W. Weimer, “Genprog: A generic method for automatic software repair,”Ieee transactions on software engineering, vol. 38, no. 1, pp. 54–72, 2011

    work page 2011

  3. [3]

    Nopol: Automatic repair of conditional statement bugs in java programs,

    J. Xuan, M. Martinez, F. Demarco, M. Clement, S. L. Marcote, T. Durieux, D. Le Berre, and M. Monperrus, “Nopol: Automatic repair of conditional statement bugs in java programs,”IEEE Transactions on Software Engineering, vol. 43, no. 1, pp. 34–55, 2016

    work page 2016

  4. [4]

    S3: syntax- and semantic-guided repair synthesis via programming by examples,

    X.-B. D. Le, D.-H. Chu, D. Lo, C. Le Goues, and W. Visser, “S3: syntax- and semantic-guided repair synthesis via programming by examples,” in Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, 2017, pp. 593–604

    work page 2017

  5. [5]

    Staged program repair with condition synthesis,

    F. Long and M. Rinard, “Staged program repair with condition synthesis,” inProceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, 2015, pp. 166–178

    work page 2015

  6. [6]

    Angelix: Scalable multiline program patch synthesis via symbolic analysis,

    S. Mechtaev, J. Yi, and A. Roychoudhury, “Angelix: Scalable multiline program patch synthesis via symbolic analysis,” inProceedings of the 38th international conference on software engineering, 2016, pp. 691– 701

    work page 2016

  7. [7]

    Astor: A program repair library for java,

    M. Martinez and M. Monperrus, “Astor: A program repair library for java,” inProceedings of the 25th international symposium on software testing and analysis, 2016, pp. 441–444

    work page 2016

  8. [8]

    History driven program repair,

    X.-B. D. Le, D. Lo, and C. L. Goues, “History driven program repair,”2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER), vol. 1, pp. 213–224, 2016. [Online]. Available: https://api.semanticscholar.org/CorpusID:8844190

    work page 2016

Show all 79 references
  1. [9]

    Automatic patch generation by learning correct code,

    F. Long and M. Rinard, “Automatic patch generation by learning correct code,” inProceedings of the 43rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, 2016, pp. 298– 312

    2016

  2. [10]

    Leveraging syntax-related code for automated program repair,

    Q. Xin and S. P. Reiss, “Leveraging syntax-related code for automated program repair,” in2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2017, pp. 660–670

    2017

  3. [11]

    Precise condition synthesis for program repair,

    Y . Xiong, J. Wang, R. Yan, J. Zhang, S. Han, G. Huang, and L. Zhang, “Precise condition synthesis for program repair,” in2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). IEEE, 2017, pp. 416–426

    2017

  4. [12]

    Automatic inference of code transforms for patch generation,

    F. Long, P. Amidon, and M. Rinard, “Automatic inference of code transforms for patch generation,” inProceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, 2017, pp. 727–739

    2017

  5. [13]

    Towards practical program repair with on-demand candidate generation,

    J. Hua, M. Zhang, K. Wang, and S. Khurshid, “Towards practical program repair with on-demand candidate generation,” inProceedings of the 40th international conference on software engineering, 2018, pp. 12–23

    2018

  6. [14]

    Context-aware patch generation for better automated program repair,

    M. Wen, J. Chen, R. Wu, D. Hao, and S.-C. Cheung, “Context-aware patch generation for better automated program repair,” inProceedings of the 40th international conference on software engineering, 2018, pp. 1–11

    2018

  7. [15]

    Shaping program repair space with existing patches and similar code,

    J. Jiang, Y . Xiong, H. Zhang, Q. Gao, and X. Chen, “Shaping program repair space with existing patches and similar code,” inProceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis, 2018, pp. 298–309

    2018

  8. [16]

    Tbar: Revisiting template-based automated program repair,

    K. Liu, A. Koyuncu, D. Kim, and T. F. Bissyand ´e, “Tbar: Revisiting template-based automated program repair,” inProceedings of the 28th ACM SIGSOFT international symposium on software testing and analysis, 2019, pp. 31–42

    2019

  9. [17]

    Avatar: Fixing semantic bugs with fix patterns of static analysis violations,

    ——, “Avatar: Fixing semantic bugs with fix patterns of static analysis violations,” in2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 2019, pp. 1–12

    2019

  10. [18]

    Practical program repair via bytecode mutation,

    A. Ghanbari, S. Benton, and L. Zhang, “Practical program repair via bytecode mutation,” inProceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, 2019, pp. 19–30

    2019

  11. [19]

    Inferring program transfor- mations from singular examples via big code,

    J. Jiang, L. Ren, Y . Xiong, and L. Zhang, “Inferring program transfor- mations from singular examples via big code,” in2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2019, pp. 255–266

    2019

  12. [20]

    The plastic surgery hypothesis in the era of large language models,

    C. S. Xia, Y . Ding, and L. Zhang, “The plastic surgery hypothesis in the era of large language models,” in2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2023, pp. 522–534

    2023

  13. [21]

    Sequencer: Sequence-to-sequence learning for end- to-end program repair,

    Z. Chen, S. Kommrusch, M. Tufano, L.-N. Pouchet, D. Poshyvanyk, and M. Monperrus, “Sequencer: Sequence-to-sequence learning for end- to-end program repair,”IEEE Transactions on Software Engineering, vol. 47, no. 9, pp. 1943–1959, 2019

    1943

  14. [22]

    Coconut: combining context-aware neural translation models using ensemble for program repair,

    T. Lutellier, H. V . Pham, L. Pang, Y . Li, M. Wei, and L. Tan, “Coconut: combining context-aware neural translation models using ensemble for program repair,” inProceedings of the 29th ACM SIGSOFT international symposium on software testing and analysis, 2020, pp. 101–114

    2020

  15. [23]

    Dlfix: Context-based code transformation learning for automated program repair,

    Y . Li, S. Wang, and T. N. Nguyen, “Dlfix: Context-based code transformation learning for automated program repair,” inProceedings of the ACM/IEEE 42nd international conference on software engineering, 2020, pp. 602–614

    2020

  16. [24]

    A syntax-guided edit decoder for neural program repair,

    Q. Zhu, Z. Sun, Y .-a. Xiao, W. Zhang, K. Yuan, Y . Xiong, and L. Zhang, “A syntax-guided edit decoder for neural program repair,” inProceedings of the 29th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, ...

    2021

  17. [25]

    Cure: Code-aware neural machine translation for automatic program repair,

    N. Jiang, T. Lutellier, and L. Tan, “Cure: Code-aware neural machine translation for automatic program repair,” in2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 2021, pp. 1161–1173

    2021

  18. [26]

    Neural program repair with execution-based backpropagation,

    H. Ye, M. Martinez, and M. Monperrus, “Neural program repair with execution-based backpropagation,” inProceedings of the 44th international conference on software engineering, 2022, pp. 1506–1518

    2022

  19. [27]

    Tare: Type-aware neural program repair,

    Q. Zhu, Z. Sun, W. Zhang, Y . Xiong, and L. Zhang, “Tare: Type-aware neural program repair,” in2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 1443–1455

    2023

  20. [28]

    Vulrepair: a t5-based automated software vulnerability repair,

    M. Fu, C. Tantithamthavorn, T. Le, V . Nguyen, and D. Phung, “Vulrepair: a t5-based automated software vulnerability repair,” inProceedings of the 30th ACM joint european software engineering conference and symposium on the foundations of software engineering, 2022, pp. 935–947

    2022

  21. [29]

    Less training, more repairing please: revisiting automated program repair via zero-shot learning,

    C. Xia and L. Zhang, “Less training, more repairing please: revisiting automated program repair via zero-shot learning,”Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2022. [Online]. Availabl...

    2022

  22. [30]

    Prompting is all you need: Automated android bug replay with large language models,

    S. Feng and C. Chen, “Prompting is all you need: Automated android bug replay with large language models,” inProceedings of the 46th IEEE/ACM International Conference on Software Engineering, 2024, pp. 1–13

    2024

  23. [31]

    Unixcoder: Unified cross-modal pre-training for code representation,

    D. Guo, S. Lu, N. Duan, Y . Wang, M. Zhou, and J. Yin, “Unixcoder: Unified cross-modal pre-training for code representation,”arXiv preprint arXiv:2203.03850, 2022

    2022 arXiv

  24. [32]

    Codegen: An open large language model for code with multi-turn program synthesis,

    E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y . Zhou, S. Savarese, and C. Xiong, “Codegen: An open large language model for code with multi-turn program synthesis,”arXiv preprint arXiv:2203.13474, 2022

    2022 arXiv

  25. [33]

    Codet5+: Open code large language models for code understanding and generation,

    Y . Wang, H. Le, A. D. Gotmare, N. D. Bui, J. Li, and S. C. Hoi, “Codet5+: Open code large language models for code understanding and generation,”arXiv preprint arXiv:2305.07922, 2023

    2023 arXiv

  26. [34]

    Few-shot training llms for project-specific code-summarization,

    T. Ahmed and P. Devanbu, “Few-shot training llms for project-specific code-summarization,” inProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, 2022, pp. 1–5. 11

    2022

  27. [35]

    Can openai’s codex fix bugs? an evaluation on quixbugs,

    J. A. Prenner, H. Babii, and R. Robbes, “Can openai’s codex fix bugs? an evaluation on quixbugs,” inProceedings of the Third International Workshop on Automated Program Repair, 2022, pp. 69–75

    2022

  28. [36]

    Impact of code language models on automated program repair,

    N. Jiang, K. Liu, T. Lutellier, and L. Tan, “Impact of code language models on automated program repair,” in2023 IEEE/ACM 45th Interna- tional Conference on Software Engineering (ICSE). IEEE, 2023, pp. 1430–1442

    2023

  29. [37]

    Automated program repair in the era of large pre-trained language models,

    C. S. Xia, Y . Wei, and L. Zhang, “Automated program repair in the era of large pre-trained language models,” inProceedings of the 45th International Conference on Software Engineering (ICSE 2023). Association for Computing Machinery, 2023

    2023

  30. [38]

    Automated repair of programs from large language models,

    Z. Fan, X. Gao, M. Mirchev, A. Roychoudhury, and S. H. Tan, “Automated repair of programs from large language models,” in2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 1469–1481

    2023

  31. [39]

    Gamma: Revisiting template-based automated program repair via mask prediction,

    Q. Zhang, C. Fang, T. Zhang, B. Yu, W. Sun, and Z. Chen, “Gamma: Revisiting template-based automated program repair via mask prediction,” in2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2023, pp. 535–547

    2023

  32. [40]

    Thinkrepair: Self-directed automated program repair,

    X. Yin, C. Ni, S. Wang, Z. Li, L. Zeng, and X. Yang, “Thinkrepair: Self-directed automated program repair,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024, pp. 1274–1286

    2024

  33. [41]

    Keep the conversation going: Fixing 162 out of 337 bugs for $0.42 each using chatgpt,

    C. S. Xia and L. Zhang, “Keep the conversation going: Fixing 162 out of 337 bugs for $0.42 each using chatgpt,”arXiv preprint arXiv:2304.00385, 2023

    2023 arXiv

  34. [42]

    An empirical study on fine-tuning large language models of code for automated program repair,

    K. Huang, X. Meng, J. Zhang, Y . Liu, W. Wang, S. Li, and Y . Zhang, “An empirical study on fine-tuning large language models of code for automated program repair,” in2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2023, pp. 1162–1174

    2023

  35. [43]

    How far can we go with practical function-level program repair?

    J. Xiang, X. Xu, F. Kong, M. Wu, Z. Zhang, H. Zhang, and Y . Zhang, “How far can we go with practical function-level program repair?”arXiv preprint arXiv:2404.12833, 2024

    2024 arXiv

  36. [44]

    Codebert: A pre-trained model for programming and natural languages,

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jianget al., “Codebert: A pre-trained model for programming and natural languages,”arXiv preprint arXiv:2002.08155, 2020

    2002 arXiv

  37. [45]

    Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,

    Y . Wang, W. Wang, S. Joty, and S. C. Hoi, “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,”arXiv preprint arXiv:2109.00859, 2021

    2021 arXiv

  38. [46]

    Defects4j: A database of existing faults to enable controlled testing studies for java programs,

    R. Just, D. Jalali, and M. D. Ernst, “Defects4j: A database of existing faults to enable controlled testing studies for java programs,” in Proceedings of the 2014 international symposium on software testing and analysis, 2014, pp. 437–440

    2014

  39. [47]

    The manybugs and introclass benchmarks for automated repair of c programs,

    C. Le Goues, N. Holtschulte, E. K. Smith, Y . Brun, P. Devanbu, S. Forrest, and W. Weimer, “The manybugs and introclass benchmarks for automated repair of c programs,”IEEE Transactions on Software Engineering, vol. 41, no. 12, pp. 1236–1256, 2015

    2015

  40. [48]

    Quixbugs: A multi- lingual program repair benchmark set based on the quixey challenge,

    D. Lin, J. Koppel, A. Chen, and A. Solar-Lezama, “Quixbugs: A multi- lingual program repair benchmark set based on the quixey challenge,” inProceedings Companion of the 2017 ACM SIGPLAN international conference on systems, programming, languages, and applications: software for...

    2017

  41. [49]

    An overview of large ai models and their applications,

    X. Tu, Z. He, Y . Huang, Z.-H. Zhang, M. Yang, and J. Zhao, “An overview of large ai models and their applications,”Visual Intelligence, vol. 2, no. 1, pp. 1–22, 2024

    2024

  42. [50]

    The cost of training nlp models: A concise overview,

    O. Sharir, B. Peleg, and Y . Shoham, “The cost of training nlp models: A concise overview,”arXiv preprint arXiv:2004.08900, 2020

    2004 arXiv

  43. [51]

    Code llama: Open foundation models for code,

    B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, T. Remez, J. Rapinet al., “Code llama: Open foundation models for code,”arXiv preprint arXiv:2308.12950, 2023

    2023 arXiv

  44. [52]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosaleet al., “Llama 2: Open foundation and fine-tuned chat models,”arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  45. [53]

    Starcoder: may the source be with you!

    R. Li, L. B. Allal, Y . Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chimet al., “Starcoder: may the source be with you!”arXiv preprint arXiv:2305.06161, 2023

    2023 arXiv

  46. [54]

    Deepseek-coder: When the large language model meets programming – the rise of code intelligence,

    D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y . Wu, Y . K. Li, F. Luo, Y . Xiong, and W. Liang, “Deepseek-coder: When the large language model meets programming – the rise of code intelligence,” 2024. [Online]. Available: https://arxiv.org/abs/2401.14196

    2024 arXiv

  47. [55]

    Bugsc++: A highly usable real world defect benchmark for c/c++,

    G. An, M. Kwon, K. Choi, J. Yi, and S. Yoo, “Bugsc++: A highly usable real world defect benchmark for c/c++,” in2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2023, pp. 2034–2037

    2023

  48. [56]

    Introclassjava: A benchmark of 297 small and buggy java programs,

    T. Durieux and M. Monperrus, “Introclassjava: A benchmark of 297 small and buggy java programs,” Ph.D. dissertation, Universite Lille 1, 2016

    2016

  49. [57]

    Condefects: A new dataset to address the data leakage concern for llm-based fault localization and program repair,

    Y . Wu, Z. Li, J. M. Zhang, and Y . Liu, “Condefects: A new dataset to address the data leakage concern for llm-based fault localization and program repair,” 2023. [Online]. Available: https://arxiv.org/abs/2310.16253

    2023 arXiv

  50. [58]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” inProceedings of the 31st International Conference on Neural Information Processing Systems, ser. NIPS’17. Red Hook, NY , USA: Curran Associates I...

    2017

  51. [59]

    Scaling laws for neural language models,

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,”arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  52. [60]

    How to fine-tune bert for text classification?

    C. Sun, X. Qiu, Y . Xu, and X. Huang, “How to fine-tune bert for text classification?” inChina national conference on Chinese computational linguistics. Springer, 2019, pp. 194–206

    2019

  53. [61]

    Parameter-efficient fine-tuning of large-scale pre-trained language models,

    N. Ding, Y . Qin, G. Yang, F. Wei, Z. Yang, Y . Su, S. Hu, Y . Chen, C.-M. Chan, W. Chenet al., “Parameter-efficient fine-tuning of large-scale pre-trained language models,”Nature Machine Intelligence, vol. 5, no. 3, pp. 220–235, 2023

    2023

  54. [62]

    The power of scale for parameter- efficient prompt tuning,

    B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter- efficient prompt tuning,”arXiv preprint arXiv:2104.08691, 2021

    2021 arXiv

  55. [63]

    Visual prompt tuning,

    M. Jia, L. Tang, B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim, “Visual prompt tuning,” inEuropean conference on computer vision. Springer, 2022, pp. 709–727

    2022

  56. [64]

    Repairllama: Efficient repre- sentations and fine-tuned adapters for program repair,

    A. Silva, S. Fang, and M. Monperrus, “Repairllama: Efficient repre- sentations and fine-tuned adapters for program repair,”arXiv preprint arXiv:2312.15698, 2023

    2023 arXiv

  57. [65]

    Lora: Low-rank adaptation of large language models

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, W. Chenet al., “Lora: Low-rank adaptation of large language models.” ICLR, vol. 1, no. 2, p. 3, 2022

    2022

  58. [66]

    Language models are few-shot learners,

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B....

    2020

  59. [67]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. V . Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” in Advances in Neural Information Processing Systems, vol. 35,

  60. [68]

    Automatic chain of thought prompting in large language models,

    Z. Zhang, A. Zhang, M. Li, and A. Smola, “Automatic chain of thought prompting in large language models,” 2022. [Online]. Available: https://arxiv.org/abs/2210.03493

    2022 arXiv

  61. [69]

    Towards understanding chain-of-thought prompting: An empirical study of what matters,

    B. Wang, S. Min, X. Deng, J. Shen, Y . Wu, L. Zettlemoyer, and H. Sun, “Towards understanding chain-of-thought prompting: An empirical study of what matters,” inProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Tor...

    2023

  62. [70]

    Copiloting the copilots: Fusing large language models with completion engines for automated program repair,

    Y . Wei, C. S. Xia, and L. Zhang, “Copiloting the copilots: Fusing large language models with completion engines for automated program repair,” inProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering...

    2023

  63. [71]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Ch...

    2020

  64. [72]

    Hybrid automated program repair by combining large language models and program analysis,

    F. Li, J. Jiang, J. Sun, and H. Zhang, “Hybrid automated program repair by combining large language models and program analysis,”ACM Trans. Softw. Eng. Methodol., Jan. 2025, just Accepted

    2025

  65. [73]

    Atcoder,

    “Atcoder,” https://atcoder.jp

  66. [74]

    Benchmarking automated program repair: An extensive study on both real-world and artificial bugs,

    Y . Ouyang, J. Yang, and L. Zhang, “Benchmarking automated program repair: An extensive study on both real-world and artificial bugs,” in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024, pp. 440–452

    2024

  67. [75]

    A large-scale empirical review of patch correctness checking approaches,

    J. Yang, Y . Wang, Y . Lou, M. Wen, and L. Zhang, “A large-scale empirical review of patch correctness checking approaches,” inProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2023, pp. 1203– 1215

    2023

  68. [76]

    Fine-grained and accurate source code differencing,

    J.-R. Falleri, F. Morandat, X. Blanc, M. Martinez, and M. Monperrus, “Fine-grained and accurate source code differencing,” inProceedings of the 29th ACM/IEEE international conference on Automated software engineering, 2014, pp. 313–324

    2014

  69. [77]

    Hyperparameter optimiza- tion for ast differencing,

    M. Martinez, J.-R. Falleri, and M. Monperrus, “Hyperparameter optimiza- tion for ast differencing,”IEEE Transactions on Software Engineering, vol. 49, no. 10, pp. 4814–4828, 2023. 13

    2023

  70. [1901]

    Available: https://proceedings.neurips.cc/paper files/ paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf

    [Online]. Available: https://proceedings.neurips.cc/paper files/ paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf

    2020

  71. [2022]

    Available: https://proceedings.neurips.cc/paper/2022/ hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html

    [Online]. Available: https://proceedings.neurips.cc/paper/2022/ hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.