Pith. sign in

REVIEW 4 major objections 4 minor 53 references

Improving LLM-Based Fault Localization with External Memory and Project Context

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MemFL claims that injecting project-specific external memory—static project and class summaries plus iteratively refined debugging guidance—into LLM prompts lets a simple three-step pipeline localize 12.7% more buggy methods at Top-1 than…

desk verdict MemFL is a clean prompt-engineering pipeline with a confounded headline comparison: dynamic memory is built from ground-truth patches, so the 12.7% gain over zero-shot AutoFL conflates few-shot supervision with external memory. read the letter →

arxiv 2506.03585 v1 pith:EOOLIBU7 submitted 2025-06-04 cs.SE

Inseok Yeo , Duksan Ryu , Jongmoon Baik This is my paper
classification cs.SE
keywords faultlocalizationlargelanguagemodelsexternalmemoryknowledgeeditingproject-specificcontextsoftwaredebuggingDefects4Jpromptengineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that LLM-based fault localization fails on complex projects mainly because the model is asked to debug without project-specific context, and that this context can be supplied cheaply as external memory instead of through fine-tuning or elaborate agent workflows. MemFL prepends two kinds of memory to the prompts of a three-step LLM pipeline: static memory, containing a project summary and per-class summaries, and dynamic memory, containing debugging guidance distilled from earlier localization attempts on a small batch of training bugs. On the Defects4J benchmark, MemFL with GPT-4o-mini reports 12.7% more Top-1 hits than AutoFL and 24.5% more than SoapFL, with the margin growing to 27.6% over AutoFL on the hardest project, Closure, while using about 21% of AutoFL's execution time and 33% of its API cost. The paper further claims that a few training bugs and one or two refinement iterations are enough to build useful dynamic memory, and that the Code Condensation step contributes the largest accuracy gain.

What carries the argument

The load-bearing mechanism is a two-component external memory injected as ordinary text into the LLM prompt: static memory (a project summary plus per-class summaries generated once from the codebase) supplies enduring context, while dynamic memory (per-step debugging guidance distilled from a small batch of training bugs) is refined by comparing each step's outputs against bug reports over a fixed number of iterations. The procedure is organized as three steps—Bug Review Generation, Code Condensation, and Fault Confirmation—and the efficiency claim comes from replacing tool-use and multi-agent orchestration with this memory. The performance claim rests on dynamic memory carrying reusable 'how to debug this project' guidance rather than bug-specific answers, and on Code Condensation shrinking the search space before the final ranking step; the ablation study identifies this condensation step as the single largest contributor to accuracy.

What would settle it

Generate dynamic memory from deliberately corrupted bug reports—for example, reports with shuffled or fabricated root causes—and run MemFL on the test folds: if Top-1 accuracy stays near the reported 178 of 350, the dynamic-memory component is not doing the explanatory work claimed. A second check is to audit the generated dynamic memory for method names, class names, or line numbers that match held-out bugs, or to evaluate with entire projects held out from memory construction; either observation would show that the gains come from memorization rather than from transferable project-specific guidance.

Watch

Extended reading notes

Core claim

MemFL's central claim is that project knowledge, written out as plain-text memory and concatenated into each LLM prompt, is what separates effective from ineffective LLM fault localization, and that a deliberately simple three-step procedure can beat considerably more complex agents and tool-use systems. The Fault Localization stage runs Bug Review Generation, Code Condensation, and Fault Confirmation, with Code Condensation hierarchically narrowing candidates by method-level coverage rate, class summaries, and source-code review so that the final ranking step sees a small relevant slice of code. The External Memory Generation stage builds static summaries automatically, then distills dynamic memory from a handful of pseudo-training bugs over up to three refinement passes; the paper reports batch size five with three iterations as the best balance of stability and cost. Evaluated on 350 real Java bugs from Defects4J, MemFL with GPT-4o-mini locates 178 bugs at Top-1 versus 158 for AutoFL and 143 for SoapFL, and with GPT-4.1-mini it reaches 214, surpassing the learning-based and spectrum-based baselines as well.

Load-bearing premise

The evaluation assumes that debugging guidance distilled from a small batch of training bugs transfers to held-out bugs in the same project, and that neither the static summaries nor the dynamic guidance encode the identities or locations of the test bugs.

Editorial extensions

If this is right

  • If the reported gains hold, a lightweight three-step pipeline with injected memory can outperform substantially more complex multi-agent and tool-use systems on method-level fault localization, implying that workflow complexity is not what buys accuracy.
  • On context-heavy projects such as Closure, where existing LLM baselines drop sharply, MemFL's Top-1 advantage over AutoFL is 27.6%, suggesting that project-specific memory yields its largest payoff exactly where generic LLM reasoning is weakest.
  • The average cost of 17.4 seconds and $0.0033 per bug with GPT-4o-mini makes per-bug LLM fault localization cheap enough to run repeatedly during debugging, not just as a one-shot analysis.
  • Dynamic memory appears to saturate quickly: one to two refinement iterations capture most of the benefit, and a batch of five training bugs yields stable guidance, so the memory-building overhead is modest relative to the localization run itself.
  • The ablation order—Code Condensation, then Bug Review Generation, then Dynamic Memory—indicates that hierarchical search-space reduction, not context injection alone, carries much of the accuracy gain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Going beyond the paper: because the memory is plain text and the pipeline is model-agnostic, the same static-plus-dynamic memory recipe should transfer to other repository-scale LLM tasks—program repair and vulnerability triage are named by the authors as future work, and nothing in the mechanism is fault-localization-specific.
  • The paper's 5-fold split keeps every test bug in the same project as training bugs; a stricter project-level holdout would test whether dynamic memory built from one project helps another, which is the stronger form of the 'project-specific knowledge generalizes' claim.
  • A natural stress test for the mechanism is to read the final dynamic memory: if the distilled guidance consists of reusable heuristics, the memory is doing what the paper claims; if it contains method names or line numbers of particular bugs, part of the gain is memorization rather than transferable guidance.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes MemFL, an LLM-based fault localization technique that augments prompts with two forms of external memory: a static memory containing project-level and class-level summaries, and a dynamic memory that is iteratively refined from previous fault localization attempts. The pipeline is simplified into three steps—Bug Review Generation, Code Condensation, and Fault Confirmation—and is evaluated on 350 Defects4J bugs against LLM-based baselines (SoapFL and AutoFL), learning-based baselines (GRACE, DeepFL), and a spectrum-based baseline (Ochiai). The authors claim that MemFL with GPT-4o-mini localizes 12.7% more bugs at Top-1 than existing LLM-based methods while using only 21% of the execution time and 33% of the API cost, with larger gains on the Closure project and further gains using GPT-4.1-mini. The paper also reports an ablation study and an analysis of dynamic-memory generation policies.

Significance. If the central claim is established, MemFL would be a practically valuable result: it shows that lightweight prompt-level injection of project context can improve LLM fault localization while substantially reducing cost and runtime. The paper has several strengths: evaluation on 350 real-world bugs, use of 5-fold cross-validation, explicit cost and time measurements, an ablation study, and comparison with multiple baseline families. However, the main result is not currently established because the dynamic memory is constructed from ground-truth buggy and patched methods of a small training batch from the same project, while the LLM baselines are evaluated zero-shot; this confounds external memory with few-shot supervised adaptation. Additional issues include a mixed-model baseline for the Math project, hyperparameter selection on the same benchmark, and the absence of variance or significance reporting. These are fixable with additional experiments and reporting, but they are load-bearing for the paper's headline claim.

major comments (4)
  1. [Section IV-D and Section V-C] The central comparison in Table II does not isolate external memory as a mechanism. As described in Section IV-D ('Dynamic Memory Generation'), the dynamic memory is built by first selecting a small batch of bugs and creating 'detailed bug reports' that include 'the source and patched versions of the buggy methods'; the LLM is then re-prompted with these reports to refine the memory. In contrast, AutoFL and SoapFL are evaluated without any such supervision (Section V-C). Therefore the reported 12.7% Top-1 advantage over AutoFL may be due to MemFL receiving few-shot, ground-truth-labeled examples from the target project rather than to the memory structure itself. Table VI ('w/o 5-Fold Cross Validation') only varies how the folds are assigned for the same supervised construction, so it does not resolve this confound. A matched baseline that is given the same training-bug bug reports and ground-truth method pairs without the memory framework, or a content analysis showing that the generated dynamic memory contains no bug- or class-location-specific information, is needed to support the claim that external memory, rather than supervised exposure, drives the improvement.
  2. [Section V-C and Table II] The SoapFL results for the Math project are obtained with a different model than the rest of the comparison. The footnote to Table II states that the underlined SoapFL results (including Math) were evaluated using GPT-3.5-turbo-0613, while MemFL and AutoFL use GPT-4o-mini. Consequently, any aggregate comparison that includes SoapFL—such as the 24.5% Top-1 improvement over SoapFL reported in Section VI-A—mixes model generations and is not a fair same-model comparison. The Math row should be re-run with GPT-4o-mini, or the aggregate claims should be recomputed excluding it.
  3. [Section V-B and Section VI-B] The dynamic-memory hyperparameters (batch size and number of iterations) are selected on the same benchmark that is used to report the main result. Figure 7 compares batch sizes and iteration counts over the Defects4J projects, and the 'optimal' configuration (batch size 5, three iterations) is then used in Table II. Although 5-fold cross-validation reduces per-bug leakage between training and test folds, selecting the configuration on the same 350-bug benchmark can still inflate the expected result, and the paper reports no per-fold variance or significance testing. It is therefore unclear whether the 19-bug Top-1 difference over AutoFL is stable. Reporting per-fold ranges or a paired significance test, and ideally holding out the configuration search, would address this concern.
  4. [Section VI-A and Table II] The abstract's claim that MemFL 'localized 12.7% more bugs than current LLM-based methods' is not qualified as applying only to Top-1 accuracy. At Top-5, MemFL (244) is below AutoFL (253) on the same GPT-4o-mini comparison, as shown in Table II. The RQ1 discussion correctly states that MemFL improves Top-1 and Top-3 while being 'comparable' at Top-5, but the headline claim should be reworded to specify Top-1 accuracy, and the Top-5 decrement should be discussed explicitly, since it suggests the method may improve ranking precision without improving recall.
minor comments (4)
  1. [Table I] The project metrics in Table I appear to be misaligned or corrupted (e.g., the Chart row reads '9650 2205 26.6'), making the columns difficult to interpret. Please reformat the table and verify the values.
  2. [Table VI] In Table VI, the Top-5 value for 'w/o 5-Fold Cross Validation' is printed as '230245'; this appears to be a missing space between '230' and '245.'
  3. [Section IV-C] The class reduction step is described as selecting only classes ranked within the top 60 by method-level coverage rate and 'retaining over 98% of the bugs,' but no supporting evidence or distribution is provided. Please clarify how this threshold was chosen and substantiate the retention claim.
  4. [Section I and Section IV-B] There are a few typographical errors, including 'Furthremore' in the introduction and 'fault localization on a small training dataset to to build dynamic memory' in Section IV-B.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MemFL's central result is an empirical held-out comparison; dynamic memory from pseudo-training bugs is an input to the method, not a renamed prediction of the test bugs.

full rationale

MemFL's claimed contribution is an empirical fault-localization result on Defects4J, not a formal derivation that reduces to its own inputs. Static memory is generated from project source and class summaries (Section IV-D1), and dynamic memory is built from a small pseudo-training subset of bugs in the same project (Section IV-D2), using ground-truth buggy/patched method pairs. The evaluation uses 5-fold cross-validation (Section V-B): dynamic memory is generated on four folds and tested on a held-out fold, so the Top-1 numbers in Table II are held-out predictions rather than fits to the target bugs. No equation in the paper defines the predicted buggy method as a direct function of the dynamic memory in a way that would make the result true by construction. There is no load-bearing self-citation: the baseline methods (SoapFL, AutoFL) and knowledge-editing background are external prior work. The paper's own leakage check (Table VI) reports nearly identical Top-1 with and without 5-fold cross-validation, and its limitation discussion (Section VII-B.1) acknowledges possible LLM training-data overlap without making that overlap load-bearing. The absence of a matched-supervision baseline for AutoFL/SoapFL and the selection of batch size/iterations on the same benchmark are legitimate evaluation-validity concerns, but they are not circularity under the definitions used here: a confound or fairness question is not a derivation that is equivalent to its inputs by construction. Therefore, the paper's central claim is not circular.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No new physical, mathematical, or conceptual entities are introduced; external memory is a software artifact assembled from LLM-generated text. The main burdens are the free parameters and the assumption that project-level guidance transfers across bugs without leaking target-bug information.

free parameters (3)
  • dynamic memory batch size = 5
    Selected as optimal from Figure 7 on the Defects4J benchmark; performance varies with batch size, so this choice influences the reported gain.
  • dynamic memory generation iterations = 3
    Selected from Figure 7 after performance stabilization on the same benchmark used for final evaluation.
  • class filtering threshold = top 60 classes by method-level coverage
    Hand-chosen cutoff adopted from SoapFL-inspired reduction; the authors justify it by stating it retains over 98% of bugs on Defects4J, which is a fit to the benchmark.
assumptions (5)
  • domain assumption Defects4J is a representative benchmark for real-world Java fault localization.
    All performance claims are based on 350 Java bugs from five Defects4J projects, Section V-B.
  • domain assumption LLM outputs are stable enough for single-run comparison.
    No repeated runs or confidence intervals are reported for the LLM-based results, Section VI.
  • domain assumption LLM-generated summaries and debugging guidance are faithful and useful.
    Static and dynamic memory are generated by the LLM itself; if summaries are inaccurate, the whole method underperforms, Section IV-D.
  • domain assumption Method-level coverage data correctly ranks classes and retains the faulty class.
    Code Condensation first filters to classes ranked in the top 60 by coverage rate r_c, Section IV-C2.
  • domain assumption Dynamic memory learned from training bugs does not leak test-bug-specific information.
    5-fold cross-validation is used, but static memory is built once per project and hyperparameters are chosen on the whole benchmark, Section VII-A.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving LLM-Based Fault Localization with External Memory and Project Context." pith.science (2026). https://pith.science/paper/EOOLIBU7

@misc{pith2026250603585,
  author       = {Pith},
  title        = {Pith review of: Improving LLM-Based Fault Localization with External Memory and Project Context},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EOOLIBU7}},
  note         = {Machine review of arXiv:2506.03585}
}
read the original abstract

Fault localization, the process of identifying the software components responsible for failures, is essential but often time-consuming. Recent advances in Large Language Models (LLMs) have enabled fault localization without extensive defect datasets or model fine-tuning. However, existing LLM-based methods rely only on general LLM capabilities and lack integration of project-specific knowledge, resulting in limited effectiveness, especially for complex software. We introduce MemFL, a novel approach that enhances LLM-based fault localization by integrating project-specific knowledge via external memory. This memory includes static summaries of the project and dynamic, iterative debugging insights gathered from previous attempts. By leveraging external memory, MemFL simplifies debugging into three streamlined steps, significantly improving efficiency and accuracy. Iterative refinement through dynamic memory further enhances reasoning quality over time. Evaluated on the Defects4J benchmark, MemFL using GPT-4o-mini localized 12.7% more bugs than current LLM-based methods, achieving this improvement with just 21% of the execution time (17.4 seconds per bug) and 33% of the API cost (0.0033 dollars per bug). On complex projects, MemFL's advantage increased to 27.6%. Additionally, MemFL with GPT-4.1-mini outperformed existing methods by 24.4%, requiring only 24.7 seconds and 0.0094 dollars per bug. MemFL thus demonstrates significant improvements by effectively incorporating project-specific knowledge into LLM-based fault localization, delivering high accuracy with reduced time and cost.

Figures

Figures reproduced from arXiv: 2506.03585 by the authors.

Figure 1
Figure 1. FL performance comparison in Closure and other [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Motivating Example III. CHALLENGES AND MOTIVATION Although LLM-based fault localization techniques have shown promising results without requiring large error datasets, they still face notable challenges. These techniques primarily rely on readily available inputs such as source code, error mes￾sages, test cases, method documentation, and static analysis. While useful, these inputs alone do not fully utilize an LLM’s… view at source ↗
Figure 3
Figure 3. Overview of Fault Localization Static Memory is generated prior to the fault localization process and remains unchanged. It consists of two parts: a project summary and a class summary. The project summary provides a high-level overview of the project’s purpose and architecture, offering broader contextual information about the target project. The class summary delivers more granular information detailing the functi… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Overview of External Memory Generation trace, error message, and failing test code, along with the source and patched versions of the buggy methods and the associated class summaries. This information is provided to the LLM, which produces a detailed, structured bug re…
Figure 5
Figure 5. Figure 5: Overlap Analysis an outstanding 27.6% and 48% increase in Top-1 accuracy compared to baselines. Overlap Analysis [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Debugging Time comparison overall performance. As shown in [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Top@1 Accuracy over Dynamic memory generation [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 26 canonical work pages

  1. [1]

    Where is the bug and how is it fixed? an experiment with practitioners,

    M. B ¨ohme, E. O. Soremekun, S. Chattopadhyay, E. Ugherughe, and A. Zeller, “Where is the bug and how is it fixed? an experiment with practitioners,” inProceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ser. ESEC/FSE 2017. New York, NY , USA: Association for Computing Machinery, 2017, p. 117–128. [Online]. Available: https:...

    arXiv 2017

  2. [2]

    Spectrum-based software fault localization: A survey of techniques, advances, and challenges,

    H. A. de Souza, M. L. Chaim, and F. Kon, “Spectrum-based software fault localization: A survey of techniques, advances, and challenges,” arXiv preprint arXiv:1607.04347, 2016

    arXiv 2016

  3. [3]

    Ask the mutants: Mutating faulty programs for fault localization,

    S. Moon, Y . Kim, M. Kim, and S. Yoo, “Ask the mutants: Mutating faulty programs for fault localization,” in2014 IEEE Seventh Interna- tional Conference on Software Testing, Verification and Validation, 2014, pp. 153–162

    work page 2014

  4. [4]

    Systematic analysis of learning-based software fault localization,

    Y . Zou, H. Li, D. Li, M. Zhao, and Z. Chen, “Systematic analysis of learning-based software fault localization,” in2024 10th International Symposium on System Security, Safety, and Reliability (ISSSR), 2024, pp. 478–489

    work page 2024

  5. [5]

    A practical evaluation of spectrum-based fault localization,

    R. Abreu, P. Zoeteweij, R. Golsteijn, and A. J. C. van Gemund, “A practical evaluation of spectrum-based fault localization,”J. Syst. Softw., vol. 82, no. 11, p. 1780–1792, Nov. 2009. [Online]. Available: https://doi.org/10.1016/j.jss.2009.06.035

    doi:10.1016/j.jss.2009.06.035 2009

  6. [6]

    Impact of large language models of code on fault localization,

    S. Ji, S. Lee, C. Lee, H. Im, and Y .-S. Han, “Impact of large language models of code on fault localization,” 08 2024

    work page 2024

  7. [7]

    Boosting coverage-based fault localization via graph-based representation learning,

    Y . Lou, Q. Zhu, J. Dong, X. Li, Z. Sun, D. Hao, L. Zhang, and L. Zhang, “Boosting coverage-based fault localization via graph-based representation learning,” inProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2021. New York, NY , USA: Association ...

    arXiv 2021

  8. [8]

    Deepfl: integrating multiple fault diagnosis dimensions for deep fault localization,

    X. Li, W. Li, Y . Zhang, and L. Zhang, “Deepfl: integrating multiple fault diagnosis dimensions for deep fault localization,” inProceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA 2019. New York, NY , USA: Association for Computing Machinery, 2019, p. 169–180. [Online]. Available: https://doi.org/10.114...

    arXiv 2019

Show all 53 references
  1. [9]

    A survey on deep learning tools dealing with data scarcity: definitions, challenges, solutions, tips, and applications,

    L. Alzubaidi, J. Bai, A. Al-Sabaawi, J. I. Santamar ´ıa, A. S. Albahri, B. S. N. Al-dabbagh, M. A. Fadhel, M. Manoufali, J. Zhang, A. H. Al-timemy, Y . Duan, A. Abdullah, L. Farhan, Y . Lu, A. Gupta, F. Albu, A. Abbosh, and Y . Gu, “A survey on deep learning tools dealing with...

    2023

  2. [10]

    Data collection and quality challenges in deep learning: A data-centric ai perspective,

    S. E. Whang, Y . Roh, H. Song, and J.-G. Lee, “Data collection and quality challenges in deep learning: A data-centric ai perspective,”The VLDB Journal, vol. 32, no. 4, pp. 791–813, 2023

    2023

  3. [11]

    Perceptions, expectations, and challenges in defect prediction,

    Z. Wan, X. Xia, A. E. Hassan, D. Lo, J. Yin, and X. Yang, “Perceptions, expectations, and challenges in defect prediction,”IEEE Transactions on Software Engineering, vol. 46, no. 11, pp. 1241–1266, 2020

    2020

  4. [12]

    Data quality for software vulnerability datasets,

    R. Croft, M. A. Babar, and M. M. Kholoosi, “Data quality for software vulnerability datasets,” in2023 IEEE/ACM 45th International Confer- ence on Software Engineering (ICSE). IEEE, 2023, pp. 121–133

    2023

  5. [13]

    Large language models: A survey,

    S. Minaee, T. Mikolov, N. Nikzad, M. Chenaghlu, R. Socher, X. Amatriain, and J. Gao, “Large language models: A survey,” 2025. [Online]. Available: https://arxiv.org/abs/2402.06196

    2025 arXiv

  6. [14]

    Large language models for test-free fault localization,

    A. Z. Yang, C. Le Goues, R. Martins, and V . Hellendoorn, “Large language models for test-free fault localization,” inProceedings of the 46th IEEE/ACM International Conference on Software Engineering, 2024, pp. 1–12

    2024

  7. [15]

    Large language models in fault localisation,

    Y . Wu, Z. Li, J. M. Zhang, M. Papadakis, M. Harman, and Y . Liu, “Large language models in fault localisation,”arXiv preprint arXiv:2308.15276, 2023

    2023 arXiv

  8. [16]

    Soapfl: A standard operating procedure for llm-based method-level fault localization,

    Y . Qin, S. Wang, Y . Lou, J. Dong, K. Wang, X. Li, and X. Mao, “Soapfl: A standard operating procedure for llm-based method-level fault localization,”IEEE Transactions on Software Engineering, vol. 51, no. 4, pp. 1173–1187, 2025

    2025

  9. [17]

    A quantitative and qualitative evaluation of llm-based explainable fault localization,

    S. Kang, G. An, and S. Yoo, “A quantitative and qualitative evaluation of llm-based explainable fault localization,”Proc. ACM Softw. Eng., vol. 1, no. FSE, Jul. 2024. [Online]. Available: https://doi.org/10.1145/3660771

    2024 doi

  10. [18]

    Comprehending Test Code: An Empirical Study ,

    C. S. Yu, C. Treude, and M. Aniche, “ Comprehending Test Code: An Empirical Study ,” in2019 IEEE International Conference on Software Maintenance and Evolution (ICSME). Los Alamitos, CA, USA: IEEE Computer Society, Oct. 2019, pp. 501–512. [Online]. Available: https://doi.ieeec...

    2019

  11. [19]

    Using contextual knowledge in interactive fault localization,

    F. Horv ´ath, A. Besz ´edes, B. Vancsics, G. Balogh, L. Vid ´acs, and T. Gyim ´othy, “Using contextual knowledge in interactive fault localization,”Empirical Softw. Engg., vol. 27, no. 6, Nov. 2022. [Online]. Available: https://doi.org/10.1007/s10664-022-10190-x

    2022 doi

  12. [20]

    Knowledge editing for large language models: A survey,

    S. Wang, Y . Zhu, H. Liu, Z. Zheng, C. Chen, and J. Li, “Knowledge editing for large language models: A survey,”ACM Comput. Surv., vol. 57, no. 3, Nov. 2024. [Online]. Available: https://doi.org/10.1145/3698590

    2024 doi

  13. [21]

    Defects4j: a database of existing faults to enable controlled testing studies for java programs,

    R. Just, D. Jalali, and M. D. Ernst, “Defects4j: a database of existing faults to enable controlled testing studies for java programs,” in Proceedings of the 2014 International Symposium on Software Testing and Analysis, ser. ISSTA 2014. New York, NY , USA: Association for Com...

    2014

  14. [22]

    On the accuracy of spectrum-based fault localization,

    R. Abreu, P. Zoeteweij, and A. J. C. van Gemund, “On the accuracy of spectrum-based fault localization,” inProceedings of the Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION, ser. TAICPART-MUTATION ’07. USA: IEEE Computer Society, 2007, p. 89–98

    2007

  15. [23]

    An evaluation of similarity coefficients for software fault localization,

    R. Abreu, P. Zoeteweij, and A. J. C. v. Gemund, “An evaluation of similarity coefficients for software fault localization,” inProceedings of the 12th Pacific Rim International Symposium on Dependable Computing, ser. PRDC ’06. USA: IEEE Computer Society, 2006, p. 39–46. [Online...

    2006 doi

  16. [24]

    Spectrum-based multi- ple fault localization,

    R. Abreu, P. Zoeteweij, and A. J. van Gemund, “Spectrum-based multi- ple fault localization,” in2009 IEEE/ACM International Conference on Automated Software Engineering, 2009, pp. 88–99

    2009

  17. [25]

    Visualization of test information to assist fault localization,

    J. Jones, M. Harrold, and J. Stasko, “Visualization of test information to assist fault localization,” inProceedings of the 24th International Conference on Software Engineering. ICSE 2002, 2002, pp. 467–477

    2002

  18. [26]

    A model for spectra- based software diagnosis,

    L. Naish, H. J. Lee, and K. Ramamohanarao, “A model for spectra- based software diagnosis,”ACM Trans. Softw. Eng. Methodol., vol. 20, no. 3, Aug. 2011. [Online]. Available: https://doi.org/10.1145/2000791. 2000795

    2011 doi

  19. [27]

    The dstar method for effective software fault localization,

    W. E. Wong, V . Debroy, R. Gao, and Y . Li, “The dstar method for effective software fault localization,”IEEE Transactions on Reliability, vol. 63, no. 1, pp. 290–308, 2014

    2014

  20. [28]

    Effective fault localization using code coverage,

    W. E. Wong, Y . Qi, L. Zhao, and K.-Y . Cai, “Effective fault localization using code coverage,” in31st Annual International Computer Software and Applications Conference (COMPSAC 2007), vol. 1, 2007, pp. 449– 456

    2007

  21. [29]

    Evolving human competitive spectra-based fault localisation techniques,

    S. Yoo, “Evolving human competitive spectra-based fault localisation techniques,” inSearch Based Software Engineering, G. Fraser and J. Teixeira de Souza, Eds. Berlin, Heidelberg: Springer Berlin Hei- delberg, 2012, pp. 244–258

    2012

  22. [30]

    Metallaxis-fl: mutation-based fault localization,

    M. Papadakis and Y . Le Traon, “Metallaxis-fl: mutation-based fault localization,”Softw. Test. Verif. Reliab., vol. 25, no. 5–7, p. 605–628, Aug. 2015. [Online]. Available: https://doi.org/10.1002/stvr.1509

    2015 doi

  23. [31]

    Learning to construct better mutation faults,

    Z. Tian, J. Chen, Q. Zhu, J. Yang, and L. Zhang, “Learning to construct better mutation faults,” inProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’22. New York, NY , USA: Association for Computing Machinery,

  24. [32]

    Deepmutation: a neural mutation tool,

    M. Tufano, J. Kimko, S. Wang, C. Watson, G. Bavota, M. Di Penta, and D. Poshyvanyk, “Deepmutation: a neural mutation tool,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Companion Proceedings, ser. ICSE ’20. New York, NY , USA: Associati...

    2020

  25. [33]

    A survey on methods for explainability in deep learning models,

    K. Mazhar and P. Dwivedi, “A survey on methods for explainability in deep learning models,” inMachine Intelligence, Tools, and Applications, S. Dehuri, S.-B. Cho, V . P. Padhy, P. Shanmugam, and A. Ghosh, Eds. Cham: Springer Nature Switzerland, 2024, pp. 257–277

    2024

  26. [34]

    A survey of large language models,

    W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y . Hou, Y . Min, B. Zhang, J. Zhang, Z. Dong, Y . Du, C. Yang, Y . Chen, Z. Chen, J. Jiang, R. Ren, Y . Li, X. Tang, Z. Liu, P. Liu, J.-Y . Nie, and J.-R. Wen, “A survey of large language models,” 2025. [Online]. Available: https:...

    2025 arXiv

  27. [35]

    Codegen: An open large language model for code with multi-turn program synthesis,

    E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y . Zhou, S. Savarese, and C. Xiong, “Codegen: An open large language model for code with multi-turn program synthesis,” 2023. [Online]. Available: https://arxiv.org/abs/2203.13474

    2023 arXiv

  28. [36]

    Practitioners’ expectations on automated fault localization,

    P. S. Kochhar, X. Xia, D. Lo, and S. Li, “Practitioners’ expectations on automated fault localization,” inProceedings of the 25th International Symposium on Software Testing and Analysis, ser. ISSTA 2016. New York, NY , USA: Association for Computing Machinery, 2016, p. 165–17...

    2016

  29. [38]

    Automated program repair in the era of large pre-trained language models,

    C. S. Xia, Y . Wei, and L. Zhang, “Automated program repair in the era of large pre-trained language models,” inProceedings of the 45th International Conference on Software Engineering, ser. ICSE ’23. IEEE Press, 2023, p. 1482–1494. [Online]. Available: https://doi.org/10.1109...

    2023

  30. [39]

    Recall and learn: Fine-tuning deep pretrained language models with less forgetting,

    S. Chen, Y . Hou, Y . Cui, W. Che, T. Liu, and X. Yu, “Recall and learn: Fine-tuning deep pretrained language models with less forgetting,”

  31. [40]

    Editable neural networks,

    A. Sinitsin, V . Plokhotnyuk, D. Pyrkin, S. Popov, and A. Babenko, “Editable neural networks,” 2020. [Online]. Available: https://arxiv.org/ abs/2004.00345

    2020 arXiv

  32. [41]

    Locating and editing factual associations in gpt,

    K. Meng, D. Bau, A. Andonian, and Y . Belinkov, “Locating and editing factual associations in gpt,” 2023. [Online]. Available: https://arxiv.org/abs/2202.05262

    2023 arXiv

  33. [42]

    Editing common sense in transformers,

    A. Gupta, D. Mondal, A. Sheshadri, W. Zhao, X. Li, S. Wiegreffe, and N. Tandon, “Editing common sense in transformers,” 01 2023, pp. 8214–8232

    2023

  34. [43]

    Transformer-patcher: One mistake worth one neuron,

    Z. Huang, Y . Shen, X. Zhang, J. Zhou, W. Rong, and Z. Xiong, “Transformer-patcher: One mistake worth one neuron,” 2023. [Online]. Available: https://arxiv.org/abs/2301.09785

    2023 arXiv

  35. [44]

    Calibrating factual knowledge in pretrained language models,

    Q. Dong, D. Dai, Y . Song, J. Xu, Z. Sui, and L. Li, “Calibrating factual knowledge in pretrained language models,” 2022. [Online]. Available: https://arxiv.org/abs/2210.03329

    2022 arXiv

  36. [45]

    Memory-based model editing at scale,

    E. Mitchell, C. Lin, A. Bosselut, C. D. Manning, and C. Finn, “Memory-based model editing at scale,” 2022. [Online]. Available: https://arxiv.org/abs/2206.06520

    2022 arXiv

  37. [46]

    Memory-assisted prompt editing to improve gpt-3 after deployment,

    A. Madaan, N. Tandon, P. Clark, and Y . Yang, “Memory-assisted prompt editing to improve gpt-3 after deployment,” 2023. [Online]. Available: https://arxiv.org/abs/2201.06009

    2023 arXiv

  38. [47]

    Agentless: Demystifying llm-based software engineering agents,

    C. S. Xia, Y . Deng, S. Dunn, and L. Zhang, “Agentless: Demystifying llm-based software engineering agents,” 2024. [Online]. Available: https://arxiv.org/abs/2407.01489

    2024 arXiv

  39. [48]

    Closure compiler,

    Google, “Closure compiler,” https://github.com/google/ closure-compiler, 2024, accessed: 2025-03-01

    2024

  40. [49]

    Fault localization via efficient probabilistic modeling of program semantics,

    M. Zeng, Y . Wu, Z. Ye, Y . Xiong, X. Zhang, and L. Zhang, “Fault localization via efficient probabilistic modeling of program semantics,” inProceedings of the 44th International Conference on Software Engineering, ser. ICSE ’22. New York, NY , USA: Association for Computing M...

    2022

  41. [50]

    Dissection of a bug dataset: Anatomy of 395 patches from defects4j,

    V . Sobreira, T. Durieux, F. Madeiral, M. Monperrus, and M. de Almeida Maia, “Dissection of a bug dataset: Anatomy of 395 patches from defects4j,” in2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, Mar. 2018. [Online]. Av...

    2018

  42. [51]

    Are automated debugging techniques actually helping programmers?

    C. Parnin and A. Orso, “Are automated debugging techniques actually helping programmers?” inProceedings of the 2011 International Symposium on Software Testing and Analysis, ser. ISSTA ’11. New York, NY , USA: Association for Computing Machinery, 2011, p. 199–209. [Online]. Av...

    2011

  43. [52]

    Berrar,Cross-Validation, 01 2018

    D. Berrar,Cross-Validation, 01 2018

    2018

  44. [2020]

    Available: https://arxiv.org/abs/2004.12651

    [Online]. Available: https://arxiv.org/abs/2004.12651

    2004 arXiv

  45. [2023]

    Available: https://doi.org/10.1145/3551349.3556949

    [Online]. Available: https://doi.org/10.1145/3551349.3556949

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.