Pith. sign in

REVIEW 4 major objections 6 minor 80 references

Adding bug-report context to prompts markedly raises how often LLMs pinpoint faulty lines, with Gemini-2.5-Flash reaching 23.67% Top@5 on Defects4J.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 08:22 UTC pith:JY3PWM45

load-bearing objection Useful but leak-prone LLM fault localization benchmark study; the bug-report-context finding on Defects4J is exactly the result most exposed to training-data memorization. the 4 major comments →

arxiv 2510.20521 v2 pith:JY3PWM45 submitted 2025-10-23 cs.SE

Large Language Models for Fault Localization: An Empirical Study

Yingjian Xiao , Weiwei Gong , Jianjun Huang , Rongqun Hu , Hongwei Li , Anquan Jie , Boyang Yang This is my paper
classification cs.SE
keywords large language modelsfault localizationstatement-level localizationDefects4JHumanEval-Javaprompt engineeringchain-of-thoughtfew-shot learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks how well large language models can find the exact statement that causes a bug, a task that matters because automated program repair depends on accurate upstream localization. It compares four models—two open-weight, two closed—on two Java benchmarks, under zero-shot, few-shot, and chain-of-thought prompting, with and without the original bug report. The central finding is that supplying the bug report is the most reliable lever: it improves every model's localization accuracy on Defects4J by a large margin (for example, Gemini-2.5-Flash Top@5 rises from 6.08% to 23.67%). Few-shot examples help some models but show diminishing returns, and chain-of-thought helps mainly models with stronger native reasoning. The paper also documents wide differences in response time and API cost, arguing that model choice involves cost-performance trade-offs.

Core claim

The paper's core claim is that LLMs do show usable statement-level fault localization ability, but only when given enough context—specifically, the bug report—and that standard prompt-engineering tricks are secondary. On Defects4J, every tested model improved dramatically when the bug report was included; Gemini-2.5-Flash reached Top@5 of 23.67% and Pass@1 of 17.81%, ahead of GPT-4.1 mini (15.15% Top@5), Qwen2.5-coder-32B-Instruct (13.75%), and DeepSeek-V3 (11.56%). On HumanEval-Java, a synthetic benchmark, Gemini-2.5-Flash again led with Top@5 of 65.03%. The paper argues that a strict exact-match standard (all defective lines must be found) makes these numbers conservative, and that bug-rep

What carries the argument

The central object is the evaluation protocol: statement-level fault localization measured by Top@k and Pass@k, with success defined as exact match, meaning the model must identify all defective lines in the file. The protocol varies the input context (source code only vs. source code plus bug report) and prompt style (standard, few-shot, chain-of-thought) across identical 13-sample runs, and augments accuracy numbers with response-time and token-cost measurements. This setup is what isolates the effect of context and prompting from raw model capability.

Load-bearing premise

The evaluation assumes that the API models' behavior at test time matches their behavior now and that the test data (Defects4J and its bug reports) were not memorized during training, so the measured accuracy is a fair estimate of real localization skill.

What would settle it

Re-run the exact same prompts on a fresh set of Java bugs with cutoff dates after the models' training, including newly written bug reports and project code; if Top@5 on that fresh set falls close to the no-bug-report level (~6%), the bug-report gains would be largely memorization rather than reasoning. A cheaper check: run the same bug-report prompt on Defects4J with the report text scrambled or paraphrased; if localization accuracy collapses, the model is keying on superficial wording rather than causal content.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • LLM-based repair pipelines should feed the bug report into the localization stage; it is the single most effective enhancement tested.
  • Benchmarks that evaluate localization without bug-report context understate real-world performance and conflate model capability with prompt information.
  • Few-shot prompting should be tuned per model; two examples often suffice, and adding more can hurt.
  • Chain-of-thought prompting is not universally beneficial; it helps open-weight models on hard real-world defects but can degrade results on simpler synthetic bugs.
  • Cost and latency are decisive for adoption: closed models cost more per call, while open-weight models can be run in-house, and slower models like Gemini-2.5-Flash may not suit latency-critical workflows.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The reported gains may overstate true generalization if any of the tested models memorized Defects4J or its bug reports during training; running the same protocol on a post-cutoff benchmark would separate memorized patterns from genuine localization skill.
  • The exact-match criterion likely under-reports partial success; a graded metric that rewards locating some but not all defective lines would give a fuller picture of practical usefulness.
  • Because bug-report context helps so much, a natural extension is to test whether automatically generated bug summaries (e.g., from issue titles or stack traces) confer similar benefits when no human-written report exists.
  • The same protocol could be extended to multi-line and cross-module defects, where the paper notes current benchmarks are limited.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper reports an empirical comparison of four large language models—GPT-4.1 mini, Gemini-2.5-Flash, Qwen2.5-Coder-32B-Instruct, and DeepSeek-V3—for statement-level fault localization on the HumanEval-Java and Defects4J benchmarks. It evaluates standard zero-shot prompting, few-shot prompting (one-, two-, three-shot), and chain-of-thought prompting, reporting Top@k and Pass@k metrics, together with time and API-cost measurements. The main findings are that Gemini-2.5-Flash performs best overall, that adding bug-report context substantially improves Defects4J results, that few-shot prompting yields only marginal and inconsistent gains, and that chain-of-thought effects are model-dependent. The paper also compares against non-LLM baselines (PMD, LineDef) and discusses internal, construct, and external validity threats.

Significance. If the measurements are valid, the study provides a useful and timely matrix of model, prompting, and benchmark choices for LLM-based fault localization, including practical cost/efficiency trade-offs. The design is systematic: two benchmarks, four models, multiple prompt strategies, and honest reporting of threats. However, the paper's main quantitative conclusions are undermined by an unresolved data-leakage risk on Defects4J and by the absence of confidence intervals or significance tests for the RQ1–RQ3 comparisons. The study does not ship prompts, outputs, or scripts, which limits reproducibility. With additional leak-controlled evaluation and statistical grounding, the contribution could be solid; as it stands, the headline claims should be treated as provisional.

major comments (4)
  1. [§5, Threats to Construct Validity (fourth item); Tables 4, 6, 8] The paper explicitly acknowledges that data leakage 'cannot be completely excluded,' but this threat is load-bearing rather than residual. Defects4J v1.2.0 (2018) predates the knowledge cutoffs of all four models (2023-10 to 2025-01), and model training corpora almost certainly contain Defects4J code and issue texts. The central Defects4J results—especially the large bug-report-context gains, e.g., Gemini Top@5 from 6.08% to 23.67% in Table 4—are exactly the quantities that memorized report-to-fix associations would inflate. Because HumanEval-Java is leakage-controlled by design but has no bug reports, it cannot support the bug-report-context conclusion. Please add a leak-controlled evaluation (e.g., post-cutoff bugs such as GitBug-Java, or a withheld recent subset) or explicitly restrict the generalization claims to the leak-controlled part of the study.
  2. [§3.4 and Tables 3–8] The RQ1–RQ3 comparisons are reported as point estimates without confidence intervals or significance tests. For example, Table 4 shows GPT-4.1 mini with-bug-report Top@5 of 15.15% vs. Qwen's 13.75% and DeepSeek's 11.56%; Table 6 shows one-shot vs. zero-shot differences of a few percentage points. The text uses 'significantly' repeatedly (e.g., Finding 2 in §4.1), but no test or CI supports these claims. Scott-Knott is used only for time analysis (§4.4.1). Please add per-bug bootstrap CIs and/or paired tests with multiple-comparison correction for all headline comparisons.
  3. [§3.4, Pass@k definition and formula] The Pass@k formula is missing from the manuscript: the text says '其公式为' and then breaks off without showing the equation. The relation among n=13, c, k, Top@k, and Pass@k is not fully defined, and it is unclear how 13 rounds are converted into the n/c counts or how partial/overlapping statement matches are handled. Because all conclusions rest on these metrics, the exact generation protocol, candidate parsing, and matching procedure must be specified so the experiments can be reproduced and the numerical differences interpreted.
  4. [Reproducibility: prompts and artifacts] No prompt templates, few-shot examples, CoT instructions, model output samples, random seeds, or evaluation scripts are provided. Empirical claims about prompt-strategy effects are highly sensitive to prompt wording and output parsing. Please release the complete prompt set, the parsing code, and raw per-bug results. This is necessary to verify the rankings and to assess the magnitude of prompt-sensitivity.
minor comments (6)
  1. [§3.4] The Pass@k equation is omitted; please add the standard formula and define all symbols consistently.
  2. [References] Several references are duplicated: [6] and [18], [8] and [19], [9] and [24], [4] and [20]. Please consolidate.
  3. [Tables 9 and 10] Costs are reported in mixed currencies (USD for GPT/Gemini, RMB for Qwen/DeepSeek). Provide a normalized comparison in one currency or include the exchange rate used, since the cost rankings are otherwise unclear.
  4. [Figures 1 and 2] The x-axis labels include 'Zoro-Shot' typos; also make the boxplot definitions (success vs. failure groups) explicit.
  5. [Model naming] Model names are inconsistent: 'ChatGPT' vs. 'GPT-4.1 mini', 'CodeQwen' vs. 'Qwen2.5-coder-32b-instruct'. Use one canonical name per model throughout.
  6. [§4.2 Findings] The text says two-shot is generally best, but Table 6 shows DeepSeek-V3 performing best at three-shot on several metrics. Please reconcile or qualify this claim.

Circularity Check

0 steps flagged

No circularity: the paper is a measurement study over external benchmarks; no fitted parameter, definitional equivalence, or load-bearing self-citation reduces the claims to their inputs.

full rationale

This paper reports API-based measurements of four LLMs on HumanEval-Java and Defects4J. The headline claims — Gemini-2.5-Flash's Top@5 of 23.67% with bug-report context, and the assertion that bug-report context improves performance — are direct observations of model outputs under a fixed prompt template, not quantities derived from a fitted parameter or from an equation that contains the conclusion. There is no self-definitional construction: the evaluation metric (complete match of all defect lines) is independent of the models' outputs, and the paper does not fit a parameter and then 'predict' the same data. The only self-citation-adjacent element is reference [7], used to attribute the Pass@k metric definition; this is not load-bearing because the metric is a standard estimator and the measured numbers are external observations. The paper itself flags a data-leakage threat in Section 5 ('仍无法完全排除数据泄露的可能性'), but data leakage is a benchmark-contamination/construct-validity risk, not circularity: even if Defects4J appeared in training data, the reported numbers would still be outputs of a model, not a tautology. No uniqueness theorem, ansatz smuggled via citation, or renamed known result appears. The skeptical concern about memorization is worth investigating empirically, but it does not make the derivation chain circular. The correct circularity verdict is therefore 0.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

This is a black-box empirical benchmark. There are no free parameters fit by the central results and no invented theoretical entities. The load-bearing axioms are all measurement assumptions: benchmark ground truth, parsing fidelity, API access as a faithful proxy, and no memorization of Defects4J.

axioms (4)
  • domain assumption The evaluation benchmarks (HumanEval-Java, Defects4J v1.2.0) and their ground-truth faulty lines are correct and used as-is.
    All metrics are computed against these ground truths; no independent verification of the benchmark labels is attempted.
  • domain assumption LLM outputs are parsed reliably and matched to ground-truth lines using the stated code-matching procedure.
    Section 5 (construct validity) admits that format deviations can misclassify correct semantic localization as failure, so parsing fidelity is load-bearing for the reported percentages.
  • domain assumption The API-accessed models correspond to the named model cards and their outputs are the stable behavior of those models.
    The models are proprietary and accessed via APIs; the paper acknowledges external API service variability and model stochasticity as an internal-validity threat.
  • domain assumption The models' training data does not contain the exact Defects4J bug instances and bug reports in a form that inflates performance.
    The paper lists data leakage as an unremoved construct-validity threat; every Defects4J number in Tables 4, 6, 8 depends on this.

pith-pipeline@v1.3.0-alltime-deepseek · 11483 in / 6214 out tokens · 45721 ms · 2026-08-04T08:22:59.321807+00:00 · methodology

0 comments
read the original abstract

Large Language Models (LLMs) have demonstrated strong performance on code-related tasks, particularly in automated program repair. However, repair effectiveness often depends on accurate upstream fault localization, while the statement-level fault localization capability of LLMs remains insufficiently evaluated. This paper presents a systematic empirical study of LLMs for statement-level fault localization. We evaluate four representative LLMs, including two open-weight models, Qwen2.5-Coder-32B-Instruct and DeepSeek-V3, and two closed-source models, GPT-4.1 mini and Gemini-2.5-Flash, on HumanEval-Java and Defects4J. The evaluation covers different input contexts and prompt strategies, including Zero-shot, Few-shot, and Chain-of-Thought prompting. We further assess model performance from three complementary perspectives: Exact Match, Partial Match, and output consistency, and compare LLMs with representative non-LLM baselines, including PMD and LineDef, under the same source-code-only input setting. In addition, we analyze practical efficiency and cost in terms of end-to-end response time and token-based API cost. The results show that bug report context improves observed fault localization performance on Defects4J; Few-shot prompting improves performance in some cases but does not yield consistent gains; and Chain-of-Thought prompting shows mixed effects across models. Overall, this study reveals the strengths, limitations, and practical trade-offs of LLMs in statement-level fault localization, providing empirical evidence for model selection and application in software engineering practice.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

80 extracted references · 6 canonical work pages · 2 internal anchors

  1. [1]

    Constraint L ogic P rogramming, S elected R esearch. 1993

    1993

  2. [2]

    Abreu, Rui and Zoeteweij, Peter and Van Gemund, Arjan , year = 2006, pages =. An. 2006 12th. doi:10.1109/PRDC.2006.18 , isbn =

    doi:10.1109/prdc.2006.18 2006

  3. [4]

    Automated Type Annotation in Python Using Large Language Models

    Bharti, Varun and Jha, Shashwat and Kumar, Dhruv and Jalote, Pankaj , year = 2025, number =. Automated. doi:10.48550/arXiv.2508.00422 , archiveprefix =. 2508.00422 , primaryclass =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2508.00422 2025

  4. [5]

    Proceedings of the

    Bin Murtaza, Sardar and Mccoy, Aidan and Ren, Zhiyuan and Murphy, Aidan and Banzhaf, Wolfgang , year = 2024, pages =. Proceedings of the. doi:10.1145/3638530.3664174 , isbn =

    arXiv 2024

  5. [6]

    Usage of

    Bistarelli, Stefano and Fiore, Marco and Mercanti, Ivan and Mongiello, Marina , year = 2025, journal =. Usage of

    2025

  6. [7]

    and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and

    Brown, Tom and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared D. and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and. Language. Advances in Neural Information Processing Systems , volume =

  7. [8]

    Evaluating

    Chen, Mark and Tworek, Jerry and Jun, Heewoo and Yuan, Qiming and Pinto, Henrique Ponde de Oliveira and Kaplan, Jared and Edwards, Harri and Burda, Yuri and Joseph, Nicholas and Brockman, Greg and Ray, Alex and Puri, Raul and Krueger, Gretchen and Petrov, Michael and Khlaaf, Heidy and Sastry, Girish and Mishkin, Pamela and Chan, Brooke and Gray, Scott and...

    Pith/arXiv arXiv doi:10.48550/arxiv.2107.03374

  8. [9]

    Exploring

    Deng, Zehang and Ma, Wanlun and Han, Qing-Long and Zhou, Wei and Zhu, Xiaogang and Wen, Sheng and Xiang, Yang , year = 2025, journal =. Exploring

    2025

  9. [10]

    Evaluating

    Du, Xueying and Liu, Mingwei and Wang, Kaixin and Wang, Hanlin and Liu, Junwei and Chen, Yixuan and Feng, Jiayi and Sha, Chaofeng and Peng, Xin and Lou, Yiling , year = 2024, pages =. Evaluating. Proceedings of the. doi:10.1145/3597503.3639219 , isbn =

    arXiv 2024

  10. [11]

    doi:10.48550/arXiv.2506.10954 , archiveprefix =

    Guo, Lianghong and Wang, Yanlin and Li, Caihua and Yang, Pengyu and Chen, Jiachi and Tao, Wei and Zou, Yingtian and Tang, Duyu and Zheng, Zibin , year = 2025, number =. doi:10.48550/arXiv.2506.10954 , archiveprefix =. 2506.10954 , primaryclass =

    doi:10.48550/arxiv.2506.10954 2025

  11. [12]

    Analyzing the

    Haldar, Rajarshi and Hockenmaier, Julia , year = 2024, publisher =. Analyzing the. doi:10.48550/ARXIV.2404.08018 , copyright =

    Pith/arXiv arXiv doi:10.48550/arxiv.2404.08018 2024

  12. [13]

    Evaluating

    Jiang, Shengbei and Zhang, Jiabao and Chen, Wei and Wang, Bo and Zhou, Jianyi and Zhang, Jie , year = 2024, pages =. Evaluating. Proceedings of the 1st. doi:10.1145/3643795.3648390 , isbn =

    arXiv 2024

  13. [15]

    Jiang, Juyong and Wang, Fan and Shen, Jiasi and Kim, Sungju and Kim, Sunghun , year = 2024, publisher =. A. doi:10.48550/ARXIV.2406.00515 , copyright =

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.00515 2024

  14. [16]

    Sentiment Analysis of

    Jukiewicz, Marcin , year = 2025, publisher =. Sentiment Analysis of. doi:10.21203/rs.3.rs-7718099/v1 , archiveprefix =

    work page doi:10.21203/rs.3.rs-7718099/v1 2025

  15. [17]

    Proceedings of the 2014

    Just, Ren. Proceedings of the 2014. doi:10.1145/2610384.2628055 , isbn =

    arXiv 2014

  16. [18]

    Kang, Sungmin and An, Gabin and Yoo, Shin , year = 2024, journal =. A. doi:10.1145/3660771 , copyright =

    doi:10.1145/3660771 2024

  17. [19]

    Proceedings of the 62nd

    Khan, Mohammad Abdullah Matin and Bari, M Saiful and Do, Xuan Long and Wang, Weishi and Parvez, Md Rizwan and Joty, Shafiq , editor =. Proceedings of the 62nd

  18. [20]

    Kim, Yunho and Mun, Seokhyeon and Yoo, Shin and Kim, Moonzoo , year = 2019, journal =. Precise

    2019

  19. [21]

    Variational

    Liang, Wei and Hu, Yiyong and Zhou, Xiaokang and Pan, Yi and Wang, Kevin I-Kai , year = 2022, journal =. Variational

    2022

  20. [22]

    Context-Aware Prompting for

    Li, Yingling and Cai, Muxin and Chen, Junjie and Xu, Yang and Huang, Lei and Li, Jianping , year = 2025, journal =. Context-Aware Prompting for

    2025

  21. [23]

    , year = 2022, pages =

    Li, Yi and Wang, Shaohua and Nguyen, Tien N. , year = 2022, pages =. Proceedings of the 44th. doi:10.1145/3510003.3510177 , isbn =

    arXiv 2022

  22. [24]

    Proceedings of the 28th

    Li, Xia and Li, Wei and Zhang, Yuqun and Zhang, Lingming , year = 2019, pages =. Proceedings of the 28th. doi:10.1145/3293882.3330574 , isbn =

    arXiv 2019

  23. [25]

    Evaluating the

    Li, Fengjie and Jiang, Jiajun and Sun, Jiajun and Zhang, Hongyu , year = 2025, pages =. Evaluating the. 2025. doi:10.1109/ICSE-NIER66352.2025.00024 , copyright =

    arXiv 2025

  24. [26]

    Li, Yi and Wang, Shaohua and Nguyen, Tien , year = 2021, pages =. Fault. 2021. doi:10.1109/ICSE43902.2021.00067 , copyright =

    arXiv 2021

  25. [27]

    Li, Fengjie and Jiang, Jiajun and Sun, Jiajun and Zhang, Hongyu , year = 2025, journal =. Hybrid

    2025

  26. [28]

    Can Automated Program Repair Refine Fault Localization? A Unified Debugging Approach , shorttitle =

    Lou, Yiling and Ghanbari, Ali and Li, Xia and Zhang, Lingming and Zhang, Haotian and Hao, Dan and Zhang, Lu , year = 2020, pages =. Can Automated Program Repair Refine Fault Localization? A Unified Debugging Approach , shorttitle =. Proceedings of the 29th. doi:10.1145/3395363.3397351 , isbn =

    arXiv 2020

  27. [29]

    arXiv preprint arXiv:2503.21710 , year=

    Enhancing repository-level software repair via repository-aware knowledge graphs , author=. arXiv preprint arXiv:2503.21710 , year=

    arXiv

  28. [30]

    arXiv preprint arXiv:2506.23749 , year=

    A Survey of LLM-based Automated Program Repair: Taxonomies, Design Paradigms, and Applications , author=. arXiv preprint arXiv:2506.23749 , year=

    arXiv

  29. [31]

    arXiv preprint arXiv:2503.22512 , year=

    Unlocking LLM Repair Capabilities in Low-Resource Programming Languages Through Cross-Language Translation and Multi-Agent Refinement , author=. arXiv preprint arXiv:2503.22512 , year=

    arXiv

  30. [32]

    Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis , pages=

    Cref: An llm-based conversational software repair framework for programming tutors , author=. Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis , pages=. 2024 , publisher=

    2024

  31. [33]

    Luo, Wenqiang and Keung, Jacky and Yang, Boyang and Ye, He and Le Goues, Claire and Bissyand. When. ACM Transactions on Software Engineering and Methodology , pages =

  32. [34]

    Expert Systems with Applications , volume =

    Majd, Amirabbas and. Expert Systems with Applications , volume =

  33. [35]

    Querying Large Automotive Software Models: Agentic vs. Direct LLM Approaches

    Mazur, Lukasz and Petrovic, Nenad and Miranda, James Pontes and Radermacher, Ansgar and Rasche, Robert and Knoll, Alois , year = 2025, number =. Querying. doi:10.48550/arXiv.2506.13171 , archiveprefix =. 2506.13171 , primaryclass =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2506.13171 2025

  34. [36]

    ACM Transactions on Software Engineering and Methodology , volume =

    A Model for Spectra-Based Software Diagnosis , author =. ACM Transactions on Software Engineering and Methodology , volume =

  35. [37]

    Evaluating and

    Pearson, Spencer and Campos, Jos. Evaluating and. 2017. doi:10.1109/ICSE.2017.62 , publisher =

    doi:10.1109/icse.2017.62 2017

  36. [38]

    doi:10.1109/TSE.2022.3144348 , copyright =

    Pornprasit, Chanathip and Tantithamthavorn, Chakkrit Kla , year = 2023, journal =. doi:10.1109/TSE.2022.3144348 , copyright =

    arXiv 2023

  37. [39]

    Qin, Yihao and Wang, Shangwen and Lou, Yiling and Dong, Jinhao and Wang, Kaixin and Li, Xiaoling and Mao, Xiaoguang , year = 2025, journal =

    2025

  38. [40]

    Deductive

    Rukmono, Satrio Adi and Ochoa, Lina and Chaudron, Michel , year = 2024, pages =. Deductive. Proceedings of the 2024. doi:10.1145/3639476.3639776 , isbn =

    arXiv 2024

  39. [41]

    Sasaki, Yui and Higo, Yoshiki and Matsumoto, Shinsuke and Kusumoto, Shinji , year = 2020, pages =. 2020. doi:10.1109/ICSME46990.2020.00076 , publisher =

    arXiv 2020

  40. [42]

    Refactoring

    Shirafuji, Atsushi and Oda, Yusuke and Suzuki, Jun and Morishita, Makoto and Watanobe, Yutaka , year = 2023, pages =. Refactoring. 2023 30th. doi:10.1109/APSEC60848.2023.00025 , publisher =

    arXiv 2023

  41. [43]

    Silva, Andr. 2025. doi:10.1109/LLM4Code66737.2025.00006 , publisher =

    arXiv 2025

  42. [44]

    Proceedings of the

    Sufian, Abu and Ghosh, Anirudha and Barman, Debaditya and Leo, Marco and Distante, Cosimo and Li, Baihua , year = 2023, pages =. Proceedings of the

    2023

  43. [45]

    and Ni, Lionel M

    Wang, Yaqing and Yao, Quanming and Kwok, James T. and Ni, Lionel M. , year = 2021, journal =. Generalizing from a

    2021

  44. [46]

    and Zhou, Denny , year = 2022, journal =

    Wei, Jason and Wang, Xuezhi and Schuurmans, Dale and Bosma, Maarten and Ichter, Brian and Xia, Fei and Chi, Ed and Le, Quoc V. and Zhou, Denny , year = 2022, journal =. Chain-of-

    2022

  45. [47]

    Eric and Gao, Ruizhi and Li, Yihao and Abreu, Rui and Wotawa, Franz , year = 2016, journal =

    Wong, W. Eric and Gao, Ruizhi and Li, Yihao and Abreu, Rui and Wotawa, Franz , year = 2016, journal =. A. doi:10.1109/TSE.2016.2521368 , copyright =

    arXiv 2016

  46. [48]

    and Papadakis, Mike and Harman, Mark and Liu, Yong , year = 2023, publisher =

    Wu, Yonghao and Li, Zheng and Zhang, Jie M. and Papadakis, Mike and Harman, Mark and Liu, Yong , year = 2023, publisher =. Large. doi:10.48550/ARXIV.2308.15276 , copyright =

    Pith/arXiv arXiv doi:10.48550/arxiv.2308.15276 2023

  47. [50]

    doi:10.48550/arXiv.2503.02951 , archiveprefix =

    Xu, Zhangchen and Liu, Yang and Yin, Yueqin and Zhou, Mingyuan and Poovendran, Radha , year = 2025, number =. doi:10.48550/arXiv.2503.02951 , archiveprefix =. 2503.02951 , primaryclass =

    Pith/arXiv arXiv doi:10.48550/arxiv.2503.02951 2025

  48. [51]

    and Alon, Uri and Neubig, Graham and Hellendoorn, Vincent Josua , year = 2022, pages =

    Xu, Frank F. and Alon, Uri and Neubig, Graham and Hellendoorn, Vincent Josua , year = 2022, pages =. A Systematic Evaluation of Large Language Models of Code , booktitle =. doi:10.1145/3520312.3534862 , isbn =

    arXiv 2022

  49. [52]

    Chain-of-

    Yang, Guang and Zhou, Yu and Chen, Xiang and Zhang, Xiangyu and Zhuo, Terry Yue and Chen, Taolue , year = 2024, journal =. Chain-of-

    2024

  50. [53]

    Yang, Aidan Z. H. and Le Goues, Claire and Martins, Ruben and Hellendoorn, Vincent , year = 2024, pages =. Large. Proceedings of the. doi:10.1145/3597503.3623342 , isbn =

    arXiv 2024

  51. [54]

    Yao, Shunyu and Yu, Dian and Zhao, Jeffrey and Shafran, Izhak and Griffiths, Tom and Cao, Yuan and Narasimhan, Karthik , year = 2023, volume =. Tree of. Advances in

    2023

  52. [55]

    Journal of Software: Evolution and Process , volume =

    Multi-graph Learning-based Software Defect Location , author =. Journal of Software: Evolution and Process , volume =

  53. [56]

    Proceedings of the 33rd

    Yin, Xin and Ni, Chao and Wang, Shaohua and Li, Zhenhao and Zeng, Limin and Yang, Xiaohu , year = 2024, pages =. Proceedings of the 33rd. doi:10.1145/3650212.3680359 , isbn =

    arXiv 2024

  54. [57]

    Context-Based

    Zhang, Lehuan and Guo, Shikai and Guo, Yi and Li, Hui and Chai, Yu and Chen, Rong and Li, Xiaochen and Jiang, He , year = 2025, journal =. Context-Based

    2025

  55. [58]

    Zheng, Tianming and Liu, Haojun and Xu, Hang and Chen, Xiang and Yi, Ping and Wu, Yue , year = 2024, journal =. Few-

    2024

  56. [59]

    Research on the

    Zhu, Zongxu and Du, Shuxin and Qiu, Yi , year = 2024, pages =. Research on the. 2024. doi:10.1109/ICAIT62580.2024.10807982 , publisher =

    arXiv 2024

  57. [60]

    Empirical

    Liu, Yangtao and Liu, Hengyuan and Yang, Zezhong and Li, Zheng and Liu, Yong , year = 2024, pages =. Empirical. 2024. doi:10.1109/QRS62785.2024.00027 , publisher =

    arXiv 2024

  58. [61]

    doi:10.1007/s10515-025-00549-x , url =

    Assessing the Effectiveness of Recent Closed-Source Large Language Models in Fault Localization and Automated Program Repair , author =. doi:10.1007/s10515-025-00549-x , url =

    work page doi:10.1007/s10515-025-00549-x

  59. [62]

    doi:10.48550/ARXIV.2403.16362 , copyright =

    Qin, Yihao and Wang, Shangwen and Lou, Yiling and Dong, Jinhao and Wang, Kaixin and Li, Xiaoling and Mao, Xiaoguang , year = 2024, publisher =. doi:10.48550/ARXIV.2403.16362 , copyright =

    Pith/arXiv arXiv doi:10.48550/arxiv.2403.16362 2024

  60. [63]

    Automated

    Fan, Zhiyu and Gao, Xiang and Mirchev, Martin and Roychoudhury, Abhik and Tan, Shin Hwei , year = 2023, pages =. Automated. 2023. doi:10.1109/ICSE48619.2023.00128 , publisher =

    arXiv 2023

  61. [64]

    , year = 2017, journal =

    Assiri, Fatmah Yousef and Bieman, James M. , year = 2017, journal =. Fault Localization for Automated Program Repair: Effectiveness, Performance, Repair Correctness , shorttitle =

    2017

  62. [65]

    and Jones, Shawn and Biswas, Ayan and Alexandrov, Boian and O'Malley, Daniel , year = 2024, pages =

    Bhattarai, Manish and Santos, Javier E. and Jones, Shawn and Biswas, Ayan and Alexandrov, Boian and O'Malley, Daniel , year = 2024, pages =. Enhancing. 2024. doi:10.1109/HPEC62836.2024.10938485 , publisher =

    arXiv 2024

  63. [66]

    Generating

    Xu, Jingyi and Le, Hieu and Samaras, Dimitris , year = 2023, pages =. Generating. Proceedings of the

    2023

  64. [67]

    Qiu, Shaojian and Huang, Huihao and Luo, Jianxiang and Kuang, Yingjie and Luo, Haoyu , year = 2024, pages =. 2024. doi:10.1109/SANER60148.2024.00036 , publisher =

    arXiv 2024

  65. [68]

    Assessing the

    Vallecillos Ruiz, Fernando and Grishina, Anastasiia and Hort, Max and Moonen, Leon , year = 2025, journal =. Assessing the

    2025

  66. [69]

    ICSE , pages=

    Template-Guided Program Repair in the Era of Large Language Models , author=. ICSE , pages=. 2025 , publisher =

    2025

  67. [70]

    Practitioners' Expectations on Automated Fault Localization , booktitle =

    Kochhar, Pavneet Singh and Xia, Xin and Lo, David and Li, Shanping , year = 2016, pages =. Practitioners' Expectations on Automated Fault Localization , booktitle =. doi:10.1145/2931037.2931051 , isbn =

    arXiv 2016

  68. [71]

    Trust Enhancement Issues in Program Repair , booktitle =

    Noller, Yannic and Shariffdeen, Ridwan and Gao, Xiang and Roychoudhury, Abhik , year = 2022, pages =. Trust Enhancement Issues in Program Repair , booktitle =. doi:10.1145/3510003.3510040 , isbn =

    arXiv 2022

  69. [72]

    doi:10.32657/10356/173910 , school =

    Neural Code Generation for Robust Automatic Program Repair , author =. doi:10.32657/10356/173910 , school =. 10356/173910 , eprinttype =

    work page doi:10.32657/10356/173910

  70. [73]

    Liu, Kui and Koyuncu, Anil and Bissyand. You. 2019 12th. doi:10.1109/ICST.2019.00020 , publisher =

    arXiv 2019

  71. [74]

    Kulal, Sumith and Pasupat, Panupong and Chandra, Kartik and Lee, Mina and Padon, Oded and Aiken, Alex and Liang, Percy S , year = 2019, volume =

    2019

  72. [75]

    Code-Line-Level Bugginess Identification:

    Guo, Zhaoqiang and Liu, Shiran and Liu, Xutong and Lai, Wei and Ma, Mingliang and Zhang, Xu and Ni, Chao and Yang, Yibiao and Li, Yanhui and Chen, Lin and Zhou, Guoqiang and Zhou, Yuming , year = 2023, journal =. Code-Line-Level Bugginess Identification:

    2023

  73. [76]

    A Longitudinal Study of Static Analysis Warning Evolution and the Effects of

    Trautsch, Alexander and Herbold, Steffen and Grabowski, Jens , year = 2020, journal =. A Longitudinal Study of Static Analysis Warning Evolution and the Effects of

    2020

  74. [77]

    IEEE Transactions On Software Engineering , volume =

    Line-Level Defect Prediction by Capturing Code Contexts with Graph Convolutional Networks , author =. IEEE Transactions On Software Engineering , volume =

  75. [78]

    and Arana, Jasper Meynard and Casin, Ethan Robert and Monterola, Christopher and Tan, Daniel Stanley and Valenzuela, Jesus Felix B

    Carandang, Kristine Ann M. and Arana, Jasper Meynard and Casin, Ethan Robert and Monterola, Christopher and Tan, Daniel Stanley and Valenzuela, Jesus Felix B. and Alis, Christian , year = 2025, pages =. Are. Proceedings of the 63rd

    2025

  76. [79]

    Bug Detection in

    Amankwah, Richard and Chen, Jinfu and Song, Heping and Kudjo, Patrick Kwaku , year = 2023, journal =. Bug Detection in. doi:10.1002/spe.3181 , copyright =

    work page doi:10.1002/spe.3181 2023

  77. [80]

    , year = 2026, eprint =

    Hannan, Md Abdul and Ni, Ronghao and Zhang, Chi and Jia, Limin and Mangal, Ravi and Pasareanu, Corina S. , year = 2026, eprint =. On the. doi:10.14722/last-x.2026.23025 , archiveprefix =

    arXiv 2026

  78. [81]

    , title =

    More, Riddhi and Bradbury, Jeremy S. , title =. 2025 , note =

    2025

  79. [82]

    2025 , howpublished =

    OpenAI , title =. 2025 , howpublished =

    2025

  80. [83]

    2026 , howpublished =

    OpenAI , title =. 2026 , howpublished =

    2026