Pith. sign in

REVIEW 3 major objections 6 minor 68 references

FlexFL: Flexible and Effective Fault Localization with Open-Source Large Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FlexFL claims that a two-stage reduce-then-refine framework lets the open-source Llama3-8B model locate buggy methods more accurately than prior GPT-3.5-based systems, using bug reports, trigger tests, or both.

desk verdict Open-source LLM fault localization that mostly delivers; main caveat is unverified baseline margins and missing significance testing. read the letter →

arxiv 2411.10714 v2 pith:LH76ZATA submitted 2024-11-16 cs.SE

Chuyang Xu , Zhongxin Liu , Xiaoxue Ren , Gehao Zhang , Ming Liang , David Lo This is my paper · ORCID
classification cs.SE
keywords FaultLocalizationLargeLanguageModelsLLM-basedAgentOpen-SourceLLMDefects4JSpectrum-basedIR-basedFunctionCalling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that automated fault localization does not require proprietary, closed-source LLMs or a fixed type of bug information. The proposed framework, FlexFL, splits the task into two stages: a space-reduction stage that merges an LLM search agent (Agent4SR) with classical techniques (SBIR, Ochiai, BoostN) into a candidate list of 20 suspicious methods, and a localization-refinement stage in which a second agent (Agent4LR) reads the code of those candidates and ranks the most likely buggy methods. On Defects4J v2.0.0, FlexFL with the open-source Llama3-8B-Instruct model puts the true buggy method first for 350 of 835 bugs (529 within the top five), and on Defects4J v1.0 it finds 42 and 63 more bugs at Top-5 than AutoFL and AgentFL, both of which run on GPT-3.5. The framework is deliberately agnostic to input type, accepting bug reports, trigger tests, or both, and it grants function-calling capability to open-source chat models through prompt engineering and fuzzy name matching. If correct, the result means debugging agents can be self-hosted on mid-size models without sacrificing localization accuracy.

What carries the argument

The machinery that carries the argument is the agent pipeline shared by Agent4SR and Agent4LR, together with a normalization layer that adapts open-source chat models to tool use. Each agent runs a three-step loop: task assignment through a dynamic prompt that inserts whatever bug-related information exists; interaction with a fixed set of seven function calls (get_paths, get_classes_of_path, get_methods_of_class, get_code_snippet_of_method, find_class, find_method, exit) that allow repository exploration; and a structured summarization step producing the output schema Top_i: PathName.ClassName.MethodName(ArgTypeList). Two components make this usable with lightweight open-source models that lack native function-calling support: the prompt asks the model to emit exactly one line 'FunctionName(Argument)', which is parsed with regular expressions, and a postprocessing matcher resolves the model's often inaccurate names to real fully qualified method names by segment containment first and Levenshtein distance (threshold below 5) second. Agent4LR is deliberately restricted to get_code_snippet_of_method and exit, called by candidate-list index, so that its limited context is spent scrutinizing code rather than exploring the repository. The postprocessing step also cleans the final ranked list, turning a hallucinated method name into the closest existing method, which the case study of the bug Time-25 illustrates can be what puts the true buggy method at Top-1.

What would settle it

Take the set of Defects4J bugs that FlexFL currently localizes at Top-1 and rerun FlexFL with the true buggy method artificially removed from the 20-method candidate list before Agent4LR runs; if a substantial fraction are still Top-1 hits, the framework does not actually depend on the candidate list and the stated two-stage design is not the cause of the result, whereas if performance collapses, first-stage recall is confirmed as the ceiling and every missed bug can be traced to the candidate list.

Watch

Extended reading notes

Core claim

The central claim, stated on the paper's own terms, is that a two-stage reduce-then-refine architecture is what makes LLM-based fault localization work with modest open-source models. The first stage builds a 20-method candidate list by taking the top-5 suggestions from each of four complementary sources: Agent4SR, an LLM agent that searches the repository guided by bug reports and/or trigger tests; SBIR, a hybrid of spectrum and information retrieval; Ochiai, a spectrum-based technique; and BoostN, an information-retrieval technique. The second stage hands that list to Agent4LR, which calls a single function to fetch the code snippet of each candidate by index and returns a ranked Top-k of the most suspicious methods. Because the pipeline presumes no particular input type, FlexFL can localize from a bug report alone, from trigger tests alone, or from both, and the paper reports that using both together improves Top-1 by at least 31.6% over either alone. The empirical claim that carries the paper is the Defects4J comparison: 350 Top-1 localizations on v2.0.0, 93 Top-1 successes on bugs missed by all non-LLM baselines, and a 42/63-bug Top-5 advantage over the GPT-3.5-based AutoFL and AgentFL on v1.0, plus 19 of 28 Top-1 localizations on a post-training-cutoff GHRB subset used to address data-contamination concerns.

Load-bearing premise

The whole pipeline is only as good as its first stage: the buggy method must make it into the 20-method candidate list, meaning at least one of the four space-reduction techniques must rank it among its top suggestions, and if none does, the second stage cannot find it no matter how well the language model reasons.

Editorial extensions

If this is right

  • FlexFL with Llama3-8B-Instruct localizes the buggy method at Top-1 for 350 of 835 Defects4J v2.0.0 bugs and at Top-5 for 529, exceeding all evaluated non-LLM baselines (BoostN, Ochiai, SBIR) on every metric.
  • On Defects4J v1.0, FlexFL beats the GPT-3.5-based AutoFL and AgentFL by 42 and 63 more bugs at Top-5, and the FlexFL-GPT-3.5-1106 variant beats a reproduced AutoFL-GPT-3.5-1106 by 19 bugs at Top-5 at less than half the API cost.
  • The framework accepts bug reports, trigger tests, or both, and combining the two inputs raises Top-1 performance by at least 31.6% over using either input alone, so one pipeline covers bugs found by fuzzing, user reports, and developer tests.
  • The same two-stage pipeline transfers across open-source models (Llama3-8B, Qwen2-7B, Mistral-Nemo-12B) with comparable results, and repeating runs with stochastic sampling (FlexFL+Repetition) raises Top-5 localization to 558 bugs on Defects4J v2.0.0.
  • FlexFL is complementary to existing LLM-based localizers: plugging AutoFL-GPT-4's results into the first stage yields a variant that localizes 10 more bugs at Top-5 than AutoFL-GPT-4 alone, and FlexFL alone finds 93 Top-1 bugs that the non-LLM techniques miss.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The space-reduction stage's recall is the binding constraint on FlexFL's ceiling: whenever the buggy method appears nowhere in the 20-method candidate list, Agent4LR cannot recover it, so the highest-leverage improvement for this line of work is raising first-stage recall (for example, by adding more FL families, call-graph information, or search engines), a direction the paper gestures at in its
  • The single-line 'FunctionName(Argument)' protocol plus fuzzy name matching is a general recipe for granting tool use to any chat model, so the same agent pipeline could plausibly be lifted to other repository-scale software tasks such as automated program repair, test generation, or code review with little modification.
  • Because the function calls and the parser are the only language-dependent parts, a direct replication on a non-Java benchmark (for example, Python or TypeScript bugs) would test whether the reported gains come from the framework itself or from the artifact quality of the Java-based Defects4J benchmark.
  • An oracle-style experiment would separate the two stages' contributions: forcibly inserting the true buggy method into the candidate list for the bugs FlexFL currently misses would quantify how much of the remaining error is Agent4LR's reasoning versus first-stage recall.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes FlexFL, a two-stage fault localization framework for Java projects. In the space-reduction stage, it combines an LLM-based agent (Agent4SR) with three non-LLM techniques (BoostN, Ochiai, SBIR) to produce a candidate list of 20 suspicious methods. In the localization-refinement stage, a second agent (Agent4LR) double-checks the code snippets of the candidate methods and reranks them. FlexFL is designed to work with open-source LLMs via a prompt-based reason-act loop and a postprocessing step for matching imperfect LLM output to real code entities. On Defects4J v2.0.0, FlexFL with Llama3-8B achieves Top-1 350, Top-3 478, and Top-5 529, outperforming the non-LLM baselines. On the Defects4J v1.0 subset, the paper claims that FlexFL outperforms the GPT-3.5-based AutoFL and AgentFL by 42 and 63 bugs at Top-5, respectively, while also providing a reproduced GPT-3.5-turbo-1106 comparison. The paper includes ablations, a repetition strategy, a GHRB experiment for data-leakage control, and a replication package.

Significance. If the results hold, FlexFL is a significant contribution to LLM-based fault localization: it demonstrates that lightweight open-source LLMs can perform project-level localization, it handles both bug reports and trigger tests in a unified pipeline, and it includes a valuable control (the 'w/o buggy program' variant in Table 6) suggesting that memorization of Defects4J is not the main driver of performance. The Defects4J v2.0.0 results are large and internally consistent, and the GHRB subset provides a best-effort contamination check. The open replication package is another concrete strength. However, the headline comparison against AutoFL and AgentFL currently rests on numbers taken from other papers, one of which (AgentFL) is not reproduced, and no significance testing is reported anywhere; these issues directly affect the strength of the relative claims, even though they do not undermine the core framework or the non-LLM comparisons.

major comments (3)
  1. [Section 5.1.1, Table 4] The claim that FlexFL-Llama3-8B localizes 42 and 63 more Top-5 bugs than AutoFL and AgentFL is based on numbers taken verbatim from the AutoFL and AgentFL papers. AgentFL is not implemented or reproduced, and AutoFL's original results use the deprecated GPT-3.5-turbo-0613. The paper's own AutoFL re-run with GPT-3.5-turbo-1106 yields Top-5=221, reducing the margin against FlexFL-Llama3-8B from 42 to 15 (and to 19 for FlexFL-GPT-3.5-1106). No paired significance test (e.g., McNemar) or confidence interval is reported, so a 15-bug difference on 353 bugs could be within sampling variation. Please add significance tests for all paired comparisons and either reproduce AgentFL under the same protocol or restrict the headline claims to the reproduced AutoFL baseline.
  2. [Section 3.1.3, Table 7] The space-reduction stage imposes a hard ceiling: Agent4LR can only localize a bug if the buggy method appears in the candidate list produced by one of the stage-1 techniques. The paper acknowledges this limitation, but it never reports the candidate-inclusion rate, i.e., for how many of the 835 Defects4J v2.0.0 bugs the buggy method appears in the top-5 lists of SBIR, Ochiai, BoostN, or Agent4SR. Without this number, the gains reported in Table 7 (e.g., Ochiai 167 -> Ochiai+Agent4LR 303, SBIR 222 -> SBIR+Agent4LR 319) conflate the effect of LLM-based reranking with the ceiling imposed by the candidate set. Reporting the inclusion rate would make the scope of the claim precise and would help readers judge how much of the improvement is attributable to Agent4LR's reasoning rather than to the candidate-generation step.
  3. [Section 5.4, Table 11] The GHRB experiment is a useful contamination control, but it uses only 28 reproducible bugs out of 38, and the paper does not report a statistical test or confidence intervals for any of the GHRB differences. The conclusion that FlexFL 'generalizes to the wild' is therefore supported by a small, non-random subset. Please temper the RQ4 conclusion or add a paired significance test (e.g., McNemar on the 28 bugs) and report the 95% confidence interval for the Top-1 proportion.
minor comments (6)
  1. [Abstract and Section 5.1.2] The abstract says FlexFL 'can localize 93 bugs that cannot be localized by non-LLM-based FL techniques at the top 1'; Figure 3(a) shows 93 bugs not localized by the three non-LLM techniques used in the space-reduction stage (BoostN, Ochiai, SBIR). Please rephrase as 'not localized by the studied non-LLM-based FL techniques' to avoid overgeneralization.
  2. [Table 4 caption] The caption contains a duplicated 'vs vs': 'FlexFL vs vs other FL techniques on Defects4J (v1.0)'.
  3. [Section 3.2.1 and Figure 2] There are minor typos: 'function callsto help' should be 'function calls to help', and the Figure 2 prompt contains 'tigger test' instead of 'trigger test'.
  4. [Section 4.5] The choice of m=20 is justified only by 'the limited context length of LLMs'; reporting a sensitivity analysis over m (e.g., 10, 15, 20, 25) would strengthen the design rationale, though it is not required for the main conclusion.
  5. [Section 5.1.1] The statement that FlexFL's Top-3 and Top-5 performance is 'close to AutoFL-GPT-4' is somewhat generous: Table 4 shows gaps of 22 and 15 bugs, respectively. Please qualify this as 'closer than the Top-1 gap' or simply report the numbers without the 'close' characterization.
  6. [Section 5.3, Table 9] The differences among FlexFL-Llama3-8B, FlexFL-Qwen2-7B, and FlexFL-Mistral-Nemo-12B are not tested for significance. A paired test or at least a note that the differences are descriptive would be clearer, given the paper's emphasis on generalizability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FlexFL is evaluated against external benchmarks and baselines, and its ablations isolate the LLM components' contributions rather than reducing the result to its inputs.

full rationale

The paper's central claim is that FlexFL, using an open-source LLM, localizes more Defects4J bugs than non-LLM baselines and than the GPT-3.5-based AutoFL and AgentFL. Nothing in the derivation chain equates a prediction to a fitted input or to a self-citation. The space-reduction stage includes BoostN, Ochiai, SBIR, and Agent4SR to form a candidate list of m=20 methods, and Agent4LR re-ranks those candidates. This is an acknowledged structural upper bound (Section 3.1.3: 'Agent4LR has a chance to localize the buggy method as long as the buggy method is included in the candidate list'), but it is not circular: the paper's ablations (Tables 7 and 8) show that adding Agent4LR improves Top-1 from 222 (SBIR) to 319 (SBIR+Agent4LR), and removing Agent4SR drops FlexFL from 350 to 338, so the LLM refinement adds measurable value beyond simply relabeling the candidate list. The comparison to AutoFL and AgentFL uses numbers reported by those external papers and a re-run of AutoFL with GPT-3.5-turbo-1106; taking baseline numbers from other groups' papers may raise threats to validity or comparability, but it is not circularity because the baseline values are not derived from FlexFL's own outputs. The GHRB experiment and the 'w/o buggy program' ablation (Top-5=75) further support that the LLM is not merely memorizing training data. There is no self-citation chain carrying the load, no parameter fitted to the evaluation data, and no known result renamed as a new framework. The skeptical concerns about unverified AgentFL numbers and lack of significance testing are external-validity and reproducibility issues, not circularity.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

FlexFL is an empirical systems paper, so the ledger contains no mathematical axioms. The central assumptions are about input sufficiency, candidate-list coverage, and the ability of open-source LLMs to follow prompt formats. Hyperparameters such as m=20, k=5, MAX=10, and the Levenshtein threshold are hand-chosen, not fitted to the target metric.

free parameters (7)
  • m (candidate list size) = 20
    Set to 20 due to limited context length of LLMs; affects search space and evaluation results.
  • k (top-k suspicious methods) = 5
    Following prior work AutoFL; affects final ranking and repetition scoring.
  • MAX (max function calls) = 10
    Initial MAX set to 10 following prior work; decreased if context length exceeded.
  • temperature = 0.0 (default), 0.6 for repetition
    Deterministic default; stochastic for the FlexFL+Repetition variant.
  • Levenshtein distance threshold = <5
    Chosen based on empirical findings for name matching in Algorithm 1; affects function call resolution accuracy.
  • SBIR random seed = 1
    Chosen from 10 provided seeds to ensure reproducibility; used both as baseline and as component, so comparison is consistent but single-seed.
  • R (repetition runs) = 5
    For the FlexFL+Repetition variant, following AutoFL's practice of multiple runs.
assumptions (5)
  • domain assumption Bug-related information (bug report and/or trigger test) is sufficient to localize the bug
    FlexFL's agents reason from bug reports and/or trigger tests; if neither contains or implies the buggy method, localization fails (Section 6.1 reports low-quality information as a failure cause).
  • domain assumption The buggy method is included in the candidate list produced by at least one space-reduction technique
    Section 3.1.3 states Agent4LR can only localize methods present in the candidate list; if no technique ranks the buggy method in its top-5 list, FlexFL cannot find it.
  • domain assumption Open-source LLMs follow the Reason-Act prompt and structured output schema
    The pipeline requires the LLM to emit 'FunctionName(Argument)' and 'Top_i: ...' lines (Section 3.2.1); the method assumes instruction-following behavior that may vary across models.
  • domain assumption GHRB subset bugs fixed after March 2023 are unseen by Llama3-8B
    Section 5.4 uses the training cutoff to argue against data leakage; assumes the model did not memorize these specific fixes.
  • domain assumption Defects4J and GHRB buggy-method labels are correct ground truth
    All metrics compare against the benchmark's mapped buggy methods; label errors would affect all compared methods equally but could bias reported accuracy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FlexFL: Flexible and Effective Fault Localization with Open-Source Large Language Models." pith.science (2026). https://pith.science/paper/LH76ZATA

@misc{pith2026241110714,
  author       = {Pith},
  title        = {Pith review of: FlexFL: Flexible and Effective Fault Localization with Open-Source Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LH76ZATA}},
  note         = {Machine review of arXiv:2411.10714}
}
read the original abstract

Due to the impressive code comprehension ability of Large Language Models (LLMs), a few studies have proposed to leverage LLMs to locate bugs, i.e., LLM-based FL, and demonstrated promising performance. However, first, these methods are limited in flexibility. They rely on bug-triggering test cases to perform FL and cannot make use of other available bug-related information, e.g., bug reports. Second, they are built upon proprietary LLMs, which are, although powerful, confronted with risks in data privacy. To address these limitations, we propose a novel LLM-based FL framework named FlexFL, which can flexibly leverage different types of bug-related information and effectively work with open-source LLMs. FlexFL is composed of two stages. In the first stage, FlexFL reduces the search space of buggy code using state-of-the-art FL techniques of different families and provides a candidate list of bug-related methods. In the second stage, FlexFL leverages LLMs to delve deeper to double-check the code snippets of methods suggested by the first stage and refine fault localization results. In each stage, FlexFL constructs agents based on open-source LLMs, which share the same pipeline that does not postulate any type of bug-related information and can interact with function calls without the out-of-the-box capability. Extensive experimental results on Defects4J demonstrate that FlexFL outperforms the baselines and can work with different open-source LLMs. Specifically, FlexFL with a lightweight open-source LLM Llama3-8B can locate 42 and 63 more bugs than two state-of-the-art LLM-based FL approaches AutoFL and AgentFL that both use GPT-3.5.

Figures

Figures reproduced from arXiv: 2411.10714 by the authors.

Figure 1
Figure 1. The Overall Framework of FlexFL times with diversified information to handle manually de￾signed tasks in each step of its process. Unlike AutoFL and AgentFL, FlexFL is a novel and effective two-stage framework that leverages LLMs to refine localization results obtained by one or more FL approaches. As discussed in the introduction, FlexFL does not limit its input to a specific type of bug-related information and is … view at source ↗
Figure 2
Figure 2. The pipeline of agents. Bold text in <> indicates placeholders for input contents or description of function calls designed in Section 3.2.2 [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Overlap Analysis of FlexFL and (a) non-LLM-based FL tech [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Comparison of FlexFL with LBFL approaches on Defects4J (v1.0) [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 56 canonical work pages

  1. [1]

    A survey on software fault localization,

    W. E. Wong, R. Gao, Y. Li, R. Abreu, and F. Wotawa, “A survey on software fault localization,” IEEE Transactions on Software Engi- neering, vol. 42, no. 8, pp. 707–740, 2016

    work page 2016

  2. [2]

    An empir- ical study of fault localization families and their combinations,

    D. Zou, J. Liang, Y. Xiong, M. D. Ernst, and L. Zhang, “An empir- ical study of fault localization families and their combinations,” IEEE Transactions on Software Engineering , vol. 47, no. 2, pp. 332– 347, 2021

    work page 2021

  3. [3]

    Bug localization based on code change histories and bug reports,

    K. C. Youm, J. Ahn, J. Kim, and E. Lee, “Bug localization based on code change histories and bug reports,” in 2015 Asia-Pacific Software Engineering Conference (APSEC), 2015, pp. 190–197

    work page 2015

  4. [4]

    Large-scale evaluation of method-level bug localization with finerbench4bl,

    S. Tsumita, S. Hayashi, and S. Amasaki, “Large-scale evaluation of method-level bug localization with finerbench4bl,” in 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), 2023, pp. 815–824

    work page 2023

  5. [5]

    Finelocator: A novel ap- proach to method-level fine-grained bug localization by query expansion,

    W. Zhang, Z. Li, Q. Wang, and J. Li, “Finelocator: A novel ap- proach to method-level fine-grained bug localization by query expansion,” INFORMATION AND SOFTWARE TECHNOLOGY , vol. 110, pp. 121–135, JUN 2019

    work page 2019

  6. [6]

    BoostNSift: A query boosting and code sifting technique for method level bug localization,

    A. Razzaq, J. Buckley, J. V . Patten, M. Chochlov, and A. R. Sai, “BoostNSift: A query boosting and code sifting technique for method level bug localization,” in 2021 IEEE 21st Interna- tional Working Conference on Source Code Analysis and Manipulation (SCAM). IEEE, 2021, pp. 81–91

    work page 2021

  7. [7]

    On the accuracy of spectrum-based fault localization,

    R. Abreu, P . Zoeteweij, and A. J. van Gemund, “On the accuracy of spectrum-based fault localization,” in Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICP ART-MUTATION 2007), 2007, pp. 89–98

    work page 2007

  8. [8]

    The dstar method for effective software fault localization,

    W. E. Wong, V . Debroy, R. Gao, and Y. Li, “The dstar method for effective software fault localization,” IEEE Trans. Reliab. , vol. 63, no. 1, pp. 290–308, 2014

    work page 2014

Show all 68 references
  1. [9]

    An empirical study of boosting spectrum-based fault localization via pagerank,

    M. Zhang, Y. Li, X. Li et al. , “An empirical study of boosting spectrum-based fault localization via pagerank,” IEEE Transactions on Software Engineering, vol. 47, no. 6, pp. 1089–1113, 2021

    2021

  2. [10]

    Can automated program repair refine fault localization? a unified debugging approach,

    X. L. Yiling Lou, Ali Ghanbari et al. , “Can automated program repair refine fault localization? a unified debugging approach,” in ISSTA ’20: 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, USA, July 18-22, 2020 , S. Khurshid and C. S....

    2020

  3. [11]

    Better automatic program repair by using bug reports and tests together,

    M. Motwani and Y. Brun, “Better automatic program repair by using bug reports and tests together,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) , 2023, pp. 1225–1237

    2023

  4. [12]

    Information retrieval and spectrum based bug localization: better together,

    T. B. Le, R. J. Oentaryo, and D. Lo, “Information retrieval and spectrum based bug localization: better together,” in Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, Bergamo, Italy, August 30 - September 4, 2015 , E. D. Nitto, ...

    2015

  5. [13]

    Evaluating large language models trained on code,

    H. J. Mark Chen, Jerry Tworek et al., “Evaluating large language models trained on code,” CoRR, vol. abs/2107.03374, 2021

    2021 arXiv

  6. [14]

    GPT-4 technical report,

    OpenAI, “GPT-4 technical report,” CoRR, vol. abs/2303.08774, 2023

    2023 arXiv

  7. [15]

    Code llama: Open foundation models for code,

    F. G. Baptiste Rozière, Jonas Gehring et al. , “Code llama: Open foundation models for code,” CoRR, vol. abs/2308.12950, 2023

    2023 arXiv

  8. [16]

    Blog of Meta Llama 3

    “Blog of Meta Llama 3.” https://ai.meta.com/blog/ meta-llama-3/, 2024

    2024

  9. [17]

    Blog of Qwen2

    “Blog of Qwen2.” https://qwenlm.github.io/blog/qwen2/, 2024

    2024

  10. [18]

    Deepseek-coder: When the large language model meets programming - the rise of code intelligence,

    D. Y. Daya Guo, Qihao Zhu et al. , “Deepseek-coder: When the large language model meets programming - the rise of code intelligence,” CoRR, vol. abs/2401.14196, 2024. 16

    2024 arXiv

  11. [19]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu et al. , “React: Synergizing reasoning and acting in language models,” in The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5,

    2023

  12. [20]

    An empirical comparison of pre-trained models of source code,

    V . N. Changan Niu, Chuanyi Li et al., “An empirical comparison of pre-trained models of source code,” in 45th IEEE/ACM Inter- national Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023. IEEE, 2023, pp. 2136–2148

    2023

  13. [21]

    An extensive study on pre- trained models for program understanding and generation,

    Z. Zeng, H. Tan, H. Zhang et al. , “An extensive study on pre- trained models for program understanding and generation,” in ISSTA ’22: 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, South Korea, July 18 - 22, 2022 , S. Ryu and Y. Smar...

    2022

  14. [22]

    Conversational automated program repair,

    C. S. Xia and L. Zhang, “Conversational automated program repair,” CoRR, vol. abs/2301.13246, 2023

    2023 arXiv

  15. [23]

    Large language models are few-shot testers: Exploring llm-based general bug reproduction,

    S. Kang, J. Yoon, and S. Yoo, “Large language models are few-shot testers: Exploring llm-based general bug reproduction,” in 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023. IEEE, 2023, pp. 2312– 2323

    2023

  16. [24]

    Large language models in fault localisation,

    Y. Wu, Z. Li, J. M. Zhang et al., “Large language models in fault localisation,” CoRR, vol. abs/2308.15276, 2023

    2023 arXiv

  17. [25]

    A quantitative and qualitative evaluation of llm-based explainable fault localization,

    S. Kang, G. An, and S. Yoo, “A quantitative and qualitative evaluation of llm-based explainable fault localization,” Proc. ACM Softw. Eng., vol. 1, no. FSE, jul 2024

    2024

  18. [26]

    Agentfl: Scaling llm-based fault localization to project-level context,

    Y. Qin, S. Wang, Y. Lou et al. , “Agentfl: Scaling llm-based fault localization to project-level context,” CoRR, vol. abs/2403.16362, 2024

    2024 arXiv

  19. [27]

    Demystifying faulty code: Step-by-step reasoning for explainable fault localization,

    R. Widyasari, J. W. Ang, T. G. Nguyen, N. Sharma, and D. Lo, “Demystifying faulty code: Step-by-step reasoning for explainable fault localization,” in IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2024, Rovaniemi, Fin- land, March 12-15...

    2024

  20. [28]

    Lost in the middle: How language models use long contexts,

    N. F. Liu, K. Lin, J. Hewitt et al. , “Lost in the middle: How language models use long contexts,” Transactions of the Association for Computational Linguistics, vol. 12, pp. 157–173, 2024

    2024

  21. [29]

    Defects4j: a database of existing faults to enable controlled testing studies for java programs,

    R. Just, D. Jalali, and M. D. Ernst, “Defects4j: a database of existing faults to enable controlled testing studies for java programs,” in Proceedings of the 2014 International Symposium on Software Testing and Analysis, ser. ISSTA 2014. New York, NY, USA: Association for Comp...

    2014

  22. [30]

    Blog of Mistral-Nemo

    “Blog of Mistral-Nemo.” https://mistral.ai/news/ mistral-nemo/, 2024

    2024

  23. [31]

    The github recent bugs dataset for evaluating llm-based debugging applications,

    J. Y. Lee, S. Kang, J. Yoon, and S. Yoo, “The github recent bugs dataset for evaluating llm-based debugging applications,” CoRR, vol. abs/2310.13229, 2023

    2023 arXiv

  24. [32]

    Our replication package

    “Our replication package.” https://doi.org/10.5281/zenodo. 11524997, 2024

    2024 doi

  25. [33]

    Where should the bugs be fixed? more accurate information retrieval-based bug localization based on bug reports,

    J. Zhou, H. Zhang, and D. Lo, “Where should the bugs be fixed? more accurate information retrieval-based bug localization based on bug reports,” in 34th International Conference on Software Engi- neering, ICSE 2012, June 2-9, 2012, Zurich, Switzerland , M. Glinz, G. C. Murphy,...

    2012

  26. [34]

    Improved bug localization based on code change histories and bug reports,

    K. C. Youm, J. Ahn, and E. Lee, “Improved bug localization based on code change histories and bug reports,” Inf. Softw. Technol. , vol. 82, pp. 177–192, 2017

    2017

  27. [35]

    Empirical evaluation of the tarantula automatic fault-localization technique,

    J. A. Jones and M. J. Harrold, “Empirical evaluation of the tarantula automatic fault-localization technique,” in Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’05. New York, NY, USA: Association for Computing Machinery, ...

    2005

  28. [36]

    An analysis and survey of the develop- ment of mutation testing,

    Y. Jia and M. Harman, “An analysis and survey of the develop- ment of mutation testing,” IEEE Trans. Software Eng., vol. 37, no. 5, pp. 649–678, 2011

    2011

  29. [37]

    Hints on test data selec- tion: Help for the practicing programmer,

    R. DeMillo, R. Lipton, and F. Sayward, “Hints on test data selec- tion: Help for the practicing programmer,”Computer, vol. 11, no. 4, pp. 34–41, 1978

    1978

  30. [38]

    Ask the mutants: Mutating faulty programs for fault localization,

    S. Moon, Y. Kim, M. Kim, and S. Yoo, “Ask the mutants: Mutating faulty programs for fault localization,” in 2014 IEEE Seventh In- ternational Conference on Software Testing, Verification and Validation, 2014, pp. 153–162

    2014

  31. [39]

    Metallaxis-fl: mutation-based fault localization,

    M. Papadakis and Y. L. Traon, “Metallaxis-fl: mutation-based fault localization,” Softw. Test. Verification Reliab. , vol. 25, no. 5-7, pp. 605–628, 2015

    2015

  32. [40]

    Contribution- based test case reduction strategy for mutation-based fault local- ization (s)

    H. Wang, K. Yang, X. Zhao, Y. Cui, and W. Wang, “Contribution- based test case reduction strategy for mutation-based fault local- ization (s).” in SEKE, 2023, pp. 142–145

    2023

  33. [41]

    Deepfl: integrating multiple fault diagnosis dimensions for deep fault localization,

    X. Li, W. Li, Y. Zhang, and L. Zhang, “Deepfl: integrating multiple fault diagnosis dimensions for deep fault localization,” in Proceed- ings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2019, Beijing, China, July 15-19, 2019 , D. Zha...

    2019

  34. [42]

    FLUCCS: using code and change metrics to improve fault localization,

    J. Sohn and S. Yoo, “FLUCCS: using code and change metrics to improve fault localization,” in Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis, Santa Barbara, CA, USA, July 10 - 14, 2017 , T. Bultan and K. Sen, Eds. ACM, 2017, pp. 273–283

    2017

  35. [43]

    Learning to rank using gradient descent,

    E. R. Christopher J. C. Burges, Tal Shaked et al., “Learning to rank using gradient descent,” in Machine Learning, Proceedings of the Twenty-Second International Conference (ICML 2005), Bonn, Germany, August 7-11, 2005 , ser. ACM International Conference Proceeding Series, L. ...

    2005

  36. [44]

    Large-scale kernel ranksvm,

    T. Kuo, C. Lee, and C. Lin, “Large-scale kernel ranksvm,” in Proceedings of the 2014 SIAM International Conference on Data Min- ing, Philadelphia, Pennsylvania, USA, April 24-26, 2014 , M. J. Zaki, Z. Obradovic, P . Tan, A. Banerjee, C. Kamath, and S. Parthasarathy, Eds. SIAM,...

    2014

  37. [45]

    The cross-entropy method: A unified approach to combinatorial optimization, monte-carlo simulation, and machine learning,

    L. Deng, “The cross-entropy method: A unified approach to combinatorial optimization, monte-carlo simulation, and machine learning,” Technometrics, vol. 48, no. 1, pp. 147–148, 2006

    2006

  38. [46]

    Expectation vs. experience: Evaluating the usability of code generation tools pow- ered by large language models,

    P . Vaithilingam, T. Zhang, and E. L. Glassman, “Expectation vs. experience: Evaluating the usability of code generation tools pow- ered by large language models,” in Chi conference on human factors in computing systems extended abstracts, 2022, pp. 1–7

    2022

  39. [47]

    Software testing with large lan- guage models: Survey, landscape, and vision,

    J. Wang, Y. Huang, C. Chen et al., “Software testing with large lan- guage models: Survey, landscape, and vision,” IEEE Transactions on Software Engineering, 2024

    2024

  40. [48]

    Repairing bugs in python assignments using large language models,

    J. Zhang, J. Cambronero, S. Gulwani, V . Le, R. Piskac, G. Soares, and G. Verbruggen, “Repairing bugs in python assignments using large language models,” arXiv preprint arXiv:2209.14876, 2022

    2022 arXiv

  41. [49]

    Bug report of Time-25(Defects4J)

    “Bug report of Time-25(Defects4J).” https://sourceforge.net/p/ joda-time/bugs/90/, 2010

    2010

  42. [50]

    Siren’s song in the ai ocean: a survey on hallucination in large language models,

    Y. Zhang, Y. Li, L. Cui et al. , “Siren’s song in the ai ocean: a survey on hallucination in large language models,” arXiv preprint arXiv:2309.01219, 2023

    2023 arXiv

  43. [51]

    Exploring and evaluating hallucinations in llm-powered code generation,

    F. Liu, Y. Liu, L. Shi, H. Huang, R. Wang, Z. Yang, and L. Zhang, “Exploring and evaluating hallucinations in llm-powered code generation,” arXiv preprint arXiv:2404.00971, 2024

    2024

  44. [52]

    Ifixr: Bug report driven program repair,

    A. Koyuncu, K. Liu, T. F. Bissyandé et al., “Ifixr: Bug report driven program repair,” in Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , ser. ESEC/FSE 2019. New York, NY, USA...

    2019

  45. [53]

    Exploring data cleanness in defects4j and its influence on fault localization efficiency,

    M. N. Rafi, A. R. Chen, T.-H. Chen, and S. Wang, “Exploring data cleanness in defects4j and its influence on fault localization efficiency,” in Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, 2024, pp. 386–387

    2024

  46. [54]

    Practitioners’ expectations on automated fault localization,

    P . S. Kochhar, X. Xia, D. Lo, and S. Li, “Practitioners’ expectations on automated fault localization,” in Proceedings of the 25th Interna- tional Symposium on Software Testing and Analysis , ser. ISSTA 2016. New York, NY, USA: Association for Computing Machinery, 2016, p. 165–176

    2016

  47. [55]

    Are automated debugging techniques actually helping programmers?

    C. Parnin and A. Orso, “Are automated debugging techniques actually helping programmers?” in Proceedings of the 2011 Interna- tional Symposium on Software Testing and Analysis , ser. ISSTA ’11. New York, NY, USA: Association for Computing Machinery, 2011, p. 199–209

    2011

  48. [56]

    C. D. Manning, P . Raghavan, and H. Schütze, Introduction to information retrieval. Cambridge University Press, 2008

    2008

  49. [57]

    The TREC-8 question answering track report,

    E. M. Voorhees, “The TREC-8 question answering track report,” in Proceedings of The Eighth Text REtrieval Conference, TREC 1999, Gaithersburg, Maryland, USA, November 17-19, 1999 , ser. NIST Spe- cial Publication, E. M. Voorhees and D. K. Harman, Eds., vol. 500-

    1999

  50. [58]

    Boosting spectrum- based fault localization using pagerank,

    M. Zhang, X. Li, L. Zhang, and S. Khurshid, “Boosting spectrum- based fault localization using pagerank,” in Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis, Santa Barbara, CA, USA, July 10 - 14, 2017 , T. Bultan and K. Sen, Eds. AC...

    2017

  51. [59]

    Gzoltar: an eclipse plug-in for testing and debugging,

    J. Campos, A. Riboira, A. Perez, and R. Abreu, “Gzoltar: an eclipse plug-in for testing and debugging,” in 2012 Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering, 2012, pp. 378–381

    2012

  52. [60]

    Model Card of Llama3-8B-Instruct

    “Model Card of Llama3-8B-Instruct.” https://huggingface.co/ meta-llama/Meta-Llama-3-8B-Instruct, 2024

    2024

  53. [61]

    Open LLM Leaderboard of HuggingFace

    “Open LLM Leaderboard of HuggingFace.” https://huggingface. co/spaces/open-llm-leaderboard/open_llm_leaderboard, 2024

    2024

  54. [62]

    Cutoff date of training dataset of Llama3

    “Cutoff date of training dataset of Llama3.” https://github.com/ meta-llama/llama3/blob/main/MODEL_CARD.md, 2024

    2024

  55. [63]

    Boosting coverage-based fault local- ization via graph-based representation learning,

    Y. Lou, Q. Zhu, J. Dong et al., “Boosting coverage-based fault local- ization via graph-based representation learning,” in ESEC/FSE ’21: 29th ACM Joint European Software Engineering Conference and Sym- posium on the Foundations of Software Engineering, Athens, Greece, August 2...

    2021

  56. [64]

    Recurrent neural network based language model,

    T. Mikolov, M. Karafiát, L. Burget et al., “Recurrent neural network based language model,” in 11th Annual Conference of the Inter- national Speech Communication Association, INTERSPEECH 2010, Makuhari, Chiba, Japan, September 26-30, 2010 , T. Kobayashi, K. Hi- rose, and S. Na...

    2010

  57. [65]

    Multilayer perceptron and neural networks,

    M. C. Popescu, V . E. Balas, L. Perescu-Popescu, and N. Mastorakis, “Multilayer perceptron and neural networks,” Wseas Transactions on Circuits & Systems, vol. 8, no. 7, pp. 579–588, 2009

    2009

  58. [66]

    Gated graph sequence neural networks,

    Y. Li, D. Tarlow, M. Brockschmidt, and R. S. Zemel, “Gated graph sequence neural networks,” in 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings , Y. Bengio and Y. LeCun, Eds., 2016

    2016

  59. [246]

    National Institute of Standards and Technology (NIST), 1999

    1999

  60. [2023]

    OpenReview.net, 2023

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.