{"id":"e6565147-c9d1-4967-8eed-b849a3558753","arxiv_id":"2411.10714","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"FlexFL uses open-source LLMs plus traditional fault localization rankings to locate buggy methods, outperforming GPT-3.5-based fault localization agents on Defects4J.","lead":"The paper introduces FlexFL, a two-stage fault localization framework that combines non-LLM debugging tools with open-source large language models to rank the most likely buggy methods in a Java project. It shows that a lightweight, self-hostable 8B-parameter model can outperform commercial GPT-3.5-based debugging agents on a standard Java bug benchmark, which matters for privacy-sensitive and cost-conscious software teams.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed margins over AutoFL/AgentFL rest on unverified baseline numbers and no significance testing; a paired like-for-like re-run is needed to confirm FlexFL's superiority.","rationale":"The paper's central empirical claim is the comparison in Table 4. In good faith, I looked for the condition that would have to hold for that claim to be true. Two conditions are necessary: (1) the candidate list from stage 1 must contain the buggy method (the reader's point), and (2) the baseline numbers must be measured under the same protocol. I judge (2) to be the less secure. AgentFL has no public implementation, so the 63-bug margin is unverified; AutoFL's original numbers come from GPT-3.5-turbo-0613, and the authors' own re-run with -1106 shrinks the margin to 15. Not only is the headline number fragile, but no significance test is provided. The candidate-list ceiling is real but the paper states it explicitly and its ablations show the LLM refinement stage adds substantial value over the raw candidates. Thus the most damaging outcome would be if the baseline comparison is not apples-to-apples, because then the claimed superiority over prior SOTA is not established. The test I propose—a paired, repeated-measures comparison with confidence intervals on the identical bug set—would settle exactly this. If the margin persists, the central claim stands; if not, the paper's quantitative headline is unsupported.","tokens_in":26453,"tokens_out":18371,"duration_ms":173610,"concrete_test":"Reproduce AutoFL-GPT-3.5-turbo-1106 and, if per-bug results can be obtained for AgentFL, run FlexFL-Llama3-8B and both baselines on the same 353 Defects4J v1.0 bugs with the same evaluation harness, R=5 stochastic repetitions each; compute a paired McNemar test and bootstrap 95% CI for the Top-5 difference between FlexFL and each baseline. If the CI excludes zero and the point estimate remains in FlexFL's favor, the outperformance claim lands; otherwise the 42/63 margins are unsubstantiated.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest claim (Table 4) states that FlexFL-Llama3-8B outperforms AgentFL and AutoFL by 63 and 42 bugs at Top-5 on Defects4J v1.0. This comparison is the empirical core of the paper's headline result, but it is not securely established. First, AgentFL is not implemented or reproduced; its numbers are taken verbatim from the AgentFL paper, and the evaluation protocols for 'active bugs' and Top-N counting may differ. Second, AutoFL's reported Top-5=194 uses the deprecated GPT-3.5-turbo-0613; the paper's own re-run with GPT-3.5-turbo-1106 yields Top-5=221, reducing the margin from 42 to 15. No paired significance test (e.g., McNemar) or confidence interval is reported, so on the 353-bug set a 15-bug advantage could be within sampling variation. The reader's candidate-list ceiling (Section 3.1.3) is a real structural upper bound, but it is acknowledged by the authors and does not by itself threaten the relative comparison; the unverified baseline figures are what directly support the claimed margin. If the baseline numbers are not comparable, the central claim of outperformance is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FlexFL, a two-stage fault localization framework for Java projects. In the space-reduction stage, it combines an LLM-based agent (Agent4SR) with three non-LLM techniques (BoostN, Ochiai, SBIR) to produce a candidate list of 20 suspicious methods. In the localization-refinement stage, a second agent (Agent4LR) double-checks the code snippets of the candidate methods and reranks them. FlexFL is designed to work with open-source LLMs via a prompt-based reason-act loop and a postprocessing step for matching imperfect LLM output to real code entities. On Defects4J v2.0.0, FlexFL with Llama3-8B achieves Top-1 350, Top-3 478, and Top-5 529, outperforming the non-LLM baselines. On the Defects4J v1.0 subset, the paper claims that FlexFL outperforms the GPT-3.5-based AutoFL and AgentFL by 42 and 63 bugs at Top-5, respectively, while also providing a reproduced GPT-3.5-turbo-1106 comparison. The paper includes ablations, a repetition strategy, a GHRB experiment for data-leakage control, and a replication package.","tokens_in":26737,"tokens_out":6221,"duration_ms":56852,"significance":"If the results hold, FlexFL is a significant contribution to LLM-based fault localization: it demonstrates that lightweight open-source LLMs can perform project-level localization, it handles both bug reports and trigger tests in a unified pipeline, and it includes a valuable control (the 'w/o buggy program' variant in Table 6) suggesting that memorization of Defects4J is not the main driver of performance. The Defects4J v2.0.0 results are large and internally consistent, and the GHRB subset provides a best-effort contamination check. The open replication package is another concrete strength. However, the headline comparison against AutoFL and AgentFL currently rests on numbers taken from other papers, one of which (AgentFL) is not reproduced, and no significance testing is reported anywhere; these issues directly affect the strength of the relative claims, even though they do not undermine the core framework or the non-LLM comparisons.","major_comments":[{"comment":"The claim that FlexFL-Llama3-8B localizes 42 and 63 more Top-5 bugs than AutoFL and AgentFL is based on numbers taken verbatim from the AutoFL and AgentFL papers. AgentFL is not implemented or reproduced, and AutoFL's original results use the deprecated GPT-3.5-turbo-0613. The paper's own AutoFL re-run with GPT-3.5-turbo-1106 yields Top-5=221, reducing the margin against FlexFL-Llama3-8B from 42 to 15 (and to 19 for FlexFL-GPT-3.5-1106). No paired significance test (e.g., McNemar) or confidence interval is reported, so a 15-bug difference on 353 bugs could be within sampling variation. Please add significance tests for all paired comparisons and either reproduce AgentFL under the same protocol or restrict the headline claims to the reproduced AutoFL baseline.","section":"Section 5.1.1, Table 4"},{"comment":"The space-reduction stage imposes a hard ceiling: Agent4LR can only localize a bug if the buggy method appears in the candidate list produced by one of the stage-1 techniques. The paper acknowledges this limitation, but it never reports the candidate-inclusion rate, i.e., for how many of the 835 Defects4J v2.0.0 bugs the buggy method appears in the top-5 lists of SBIR, Ochiai, BoostN, or Agent4SR. Without this number, the gains reported in Table 7 (e.g., Ochiai 167 -> Ochiai+Agent4LR 303, SBIR 222 -> SBIR+Agent4LR 319) conflate the effect of LLM-based reranking with the ceiling imposed by the candidate set. Reporting the inclusion rate would make the scope of the claim precise and would help readers judge how much of the improvement is attributable to Agent4LR's reasoning rather than to the candidate-generation step.","section":"Section 3.1.3, Table 7"},{"comment":"The GHRB experiment is a useful contamination control, but it uses only 28 reproducible bugs out of 38, and the paper does not report a statistical test or confidence intervals for any of the GHRB differences. The conclusion that FlexFL 'generalizes to the wild' is therefore supported by a small, non-random subset. Please temper the RQ4 conclusion or add a paired significance test (e.g., McNemar on the 28 bugs) and report the 95% confidence interval for the Top-1 proportion.","section":"Section 5.4, Table 11"}],"minor_comments":[{"comment":"The abstract says FlexFL 'can localize 93 bugs that cannot be localized by non-LLM-based FL techniques at the top 1'; Figure 3(a) shows 93 bugs not localized by the three non-LLM techniques used in the space-reduction stage (BoostN, Ochiai, SBIR). Please rephrase as 'not localized by the studied non-LLM-based FL techniques' to avoid overgeneralization.","section":"Abstract and Section 5.1.2"},{"comment":"The caption contains a duplicated 'vs vs': 'FlexFL vs vs other FL techniques on Defects4J (v1.0)'.","section":"Table 4 caption"},{"comment":"There are minor typos: 'function callsto help' should be 'function calls to help', and the Figure 2 prompt contains 'tigger test' instead of 'trigger test'.","section":"Section 3.2.1 and Figure 2"},{"comment":"The choice of m=20 is justified only by 'the limited context length of LLMs'; reporting a sensitivity analysis over m (e.g., 10, 15, 20, 25) would strengthen the design rationale, though it is not required for the main conclusion.","section":"Section 4.5"},{"comment":"The statement that FlexFL's Top-3 and Top-5 performance is 'close to AutoFL-GPT-4' is somewhat generous: Table 4 shows gaps of 22 and 15 bugs, respectively. Please qualify this as 'closer than the Top-1 gap' or simply report the numbers without the 'close' characterization.","section":"Section 5.1.1"},{"comment":"The differences among FlexFL-Llama3-8B, FlexFL-Qwen2-7B, and FlexFL-Mistral-Nemo-12B are not tested for significance. A paired test or at least a note that the differences are descriptive would be clearer, given the paper's emphasis on generalizability.","section":"Section 5.3, Table 9"}],"recommendation":"major_revision","confidential_remarks":"The strongest and most original part of the paper is the Defects4J v2.0.0 comparison against non-LLM baselines and the associated ablation showing that removing buggy-program access drastically reduces performance. The weakest link is the comparison to AutoFL and AgentFL, which is not on a like-for-like basis and is not significance-tested. This is fixable within the paper's scope: the authors can reproduce AgentFL or drop that claim, add McNemar tests, and report the candidate-inclusion ceiling. I do not see a reason to reject, but the relative claims need to be made rigorous before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read on FlexFL (arXiv:2411.10714). The core result is real: a two-stage method-level FL framework that runs on open-source LLMs, with the first open-source-based FL agents and a prompt-level function-calling pipeline. The Defects4J numbers are large and internally consistent; the w/o-buggy-program control is a good contamination check, and the ablation study does real work. The GHRB experiment is a nice extra, though small. I'd give the authors credit for shipping a replication package and for being upfront about the candidate-list ceiling in Section 3.1.3 — that is an honest statement of a structural constraint.\n\nWhere I push back on the stress-test note: it says the margin over AutoFL/AgentFL is unverified. True, but not fatal. AgentFL has no public implementation, so citing its reported numbers is the only option. The authors do re-run AutoFL with GPT-3.5-turbo-1106 and report 221, which cuts the margin from 42 to 15, and they also report FlexFL-GPT-3.5-1106 at 240, so the claimed pattern holds even with the re-run. What is missing is a paired significance test (McNemar) or confidence intervals; on 353 bugs, 15–19 extra hits could be sampling noise. That should be a required revision, not a desk-reject reason.\n\nA separate soft spot is the GHRB generalization experiment: only 28 of 38 bugs are reproducible, and the selection criteria are not fully discussed. That makes the \"in the wild\" claim suggestive rather than conclusive. Minor, but worth tightening. Also, the m=20 ceiling means FlexFL cannot recover what no stage-1 technique finds; the authors acknowledge it, and the relative comparison still stands, but the absolute ceiling should be reported as a theoretical bound.\n\nWho is this for: anyone working on LLM-based debugging, automated repair, or practical fault localization. The paper deserves a serious referee. I'd send it out with a request for paired statistics and a clearer protocol on baseline numbers.","headline":"Open-source LLM fault localization that mostly delivers; main caveat is unverified baseline margins and missing significance testing.","tokens_in":27312,"tokens_out":1504,"would_cite":true,"duration_ms":15601,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FlexFL claims that a two-stage reduce-then-refine framework lets the open-source Llama3-8B model locate buggy methods more accurately than prior GPT-3.5-based systems, using bug reports, trigger tests, or both.","keywords":["Fault Localization","Large Language Models","LLM-based Agent","Open-Source LLM","Defects4J","Spectrum-based Fault Localization","IR-based Fault Localization","Function Calling"],"falsifier":"Take the set of Defects4J bugs that FlexFL currently localizes at Top-1 and rerun FlexFL with the true buggy method artificially removed from the 20-method candidate list before Agent4LR runs; if a substantial fraction are still Top-1 hits, the framework does not actually depend on the candidate list and the stated two-stage design is not the cause of the result, whereas if performance collapses, first-stage recall is confirmed as the ceiling and every missed bug can be traced to the candidate list.","tokens_in":26245,"feed_emoji":"🐛","tokens_out":15893,"duration_ms":119100,"temperature":0.7,"pith_summary":"The paper is trying to establish that automated fault localization does not require proprietary, closed-source LLMs or a fixed type of bug information. The proposed framework, FlexFL, splits the task into two stages: a space-reduction stage that merges an LLM search agent (Agent4SR) with classical techniques (SBIR, Ochiai, BoostN) into a candidate list of 20 suspicious methods, and a localization-refinement stage in which a second agent (Agent4LR) reads the code of those candidates and ranks the most likely buggy methods. On Defects4J v2.0.0, FlexFL with the open-source Llama3-8B-Instruct model puts the true buggy method first for 350 of 835 bugs (529 within the top five), and on Defects4J v1.0 it finds 42 and 63 more bugs at Top-5 than AutoFL and AgentFL, both of which run on GPT-3.5. The framework is deliberately agnostic to input type, accepting bug reports, trigger tests, or both, and it grants function-calling capability to open-source chat models through prompt engineering and fuzzy name matching. If correct, the result means debugging agents can be self-hosted on mid-size models without sacrificing localization accuracy.","feed_headline":"Open-source 8B model beats GPT-3.5 at finding buggy code","feed_subtitle":"FlexFL's two-stage pipeline lands the right method at Top-1 in 350 of 835 Defects4J bugs on a self-hosted model","key_machinery":"The machinery that carries the argument is the agent pipeline shared by Agent4SR and Agent4LR, together with a normalization layer that adapts open-source chat models to tool use. Each agent runs a three-step loop: task assignment through a dynamic prompt that inserts whatever bug-related information exists; interaction with a fixed set of seven function calls (get_paths, get_classes_of_path, get_methods_of_class, get_code_snippet_of_method, find_class, find_method, exit) that allow repository exploration; and a structured summarization step producing the output schema Top_i: PathName.ClassName.MethodName(ArgTypeList). Two components make this usable with lightweight open-source models that lack native function-calling support: the prompt asks the model to emit exactly one line 'FunctionName(Argument)', which is parsed with regular expressions, and a postprocessing matcher resolves the model's often inaccurate names to real fully qualified method names by segment containment first and Levenshtein distance (threshold below 5) second. Agent4LR is deliberately restricted to get_code_snippet_of_method and exit, called by candidate-list index, so that its limited context is spent scrutinizing code rather than exploring the repository. The postprocessing step also cleans the final ranked list, turning a hallucinated method name into the closest existing method, which the case study of the bug Time-25 illustrates can be what puts the true buggy method at Top-1.","core_discovery":"The central claim, stated on the paper's own terms, is that a two-stage reduce-then-refine architecture is what makes LLM-based fault localization work with modest open-source models. The first stage builds a 20-method candidate list by taking the top-5 suggestions from each of four complementary sources: Agent4SR, an LLM agent that searches the repository guided by bug reports and/or trigger tests; SBIR, a hybrid of spectrum and information retrieval; Ochiai, a spectrum-based technique; and BoostN, an information-retrieval technique. The second stage hands that list to Agent4LR, which calls a single function to fetch the code snippet of each candidate by index and returns a ranked Top-k of the most suspicious methods. Because the pipeline presumes no particular input type, FlexFL can localize from a bug report alone, from trigger tests alone, or from both, and the paper reports that using both together improves Top-1 by at least 31.6% over either alone. The empirical claim that carries the paper is the Defects4J comparison: 350 Top-1 localizations on v2.0.0, 93 Top-1 successes on bugs missed by all non-LLM baselines, and a 42/63-bug Top-5 advantage over the GPT-3.5-based AutoFL and AgentFL on v1.0, plus 19 of 28 Top-1 localizations on a post-training-cutoff GHRB subset used to address data-contamination concerns.","pith_inferences":["The space-reduction stage's recall is the binding constraint on FlexFL's ceiling: whenever the buggy method appears nowhere in the 20-method candidate list, Agent4LR cannot recover it, so the highest-leverage improvement for this line of work is raising first-stage recall (for example, by adding more FL families, call-graph information, or search engines), a direction the paper gestures at in its ","The single-line 'FunctionName(Argument)' protocol plus fuzzy name matching is a general recipe for granting tool use to any chat model, so the same agent pipeline could plausibly be lifted to other repository-scale software tasks such as automated program repair, test generation, or code review with little modification.","Because the function calls and the parser are the only language-dependent parts, a direct replication on a non-Java benchmark (for example, Python or TypeScript bugs) would test whether the reported gains come from the framework itself or from the artifact quality of the Java-based Defects4J benchmark.","An oracle-style experiment would separate the two stages' contributions: forcibly inserting the true buggy method into the candidate list for the bugs FlexFL currently misses would quantify how much of the remaining error is Agent4LR's reasoning versus first-stage recall."],"forward_implications":["FlexFL with Llama3-8B-Instruct localizes the buggy method at Top-1 for 350 of 835 Defects4J v2.0.0 bugs and at Top-5 for 529, exceeding all evaluated non-LLM baselines (BoostN, Ochiai, SBIR) on every metric.","On Defects4J v1.0, FlexFL beats the GPT-3.5-based AutoFL and AgentFL by 42 and 63 more bugs at Top-5, and the FlexFL-GPT-3.5-1106 variant beats a reproduced AutoFL-GPT-3.5-1106 by 19 bugs at Top-5 at less than half the API cost.","The framework accepts bug reports, trigger tests, or both, and combining the two inputs raises Top-1 performance by at least 31.6% over using either input alone, so one pipeline covers bugs found by fuzzing, user reports, and developer tests.","The same two-stage pipeline transfers across open-source models (Llama3-8B, Qwen2-7B, Mistral-Nemo-12B) with comparable results, and repeating runs with stochastic sampling (FlexFL+Repetition) raises Top-5 localization to 558 bugs on Defects4J v2.0.0.","FlexFL is complementary to existing LLM-based localizers: plugging AutoFL-GPT-4's results into the first stage yields a variant that localizes 10 more bugs at Top-5 than AutoFL-GPT-4 alone, and FlexFL alone finds 93 Top-1 bugs that the non-LLM techniques miss."],"supporting_citations":[{"why":"Provides the AutoFL baseline (GPT-3.5/GPT-4 with native function calling) that FlexFL must beat, and supplies the repetition-strategy comparison.","marker":"[25]"},{"why":"Provides the AgentFL baseline, a ChatGPT-based method-level localizer that FlexFL outperforms by 63 bugs at Top-5 on Defects4J v1.0.","marker":"[26]"},{"why":"Supplies the reason-then-act prompting pattern that FlexFL adapts so open-source chat models can plan and issue function calls without native tool-use support.","marker":"[19]"},{"why":"Supplies the Defects4J benchmark (835 version-2.0.0 bugs) on which the paper's main quantitative claims are measured.","marker":"[29]"},{"why":"Supplies the SBIR hybrid technique whose top-5 results feed the space-reduction candidate list and which serves as a HybridFL baseline.","marker":"[11]"},{"why":"Supplies BoostNSift, modified into BoostN, the IRFL technique whose top-5 results feed the candidate list.","marker":"[6]"},{"why":"Supplies Ochiai, the spectrum-based technique whose top-5 results feed the candidate list and which serves as an SBFL baseline.","marker":"[7]"},{"why":"Supplies the GHRB dataset of post-training-cutoff bugs used to argue that results are not an artifact of data contamination.","marker":"[31]"},{"why":"Identifies the base model Llama3-8B-Instruct that FlexFL is built on and whose training cutoff defines the contamination check.","marker":"[16]"}],"fun_headline_variants":["Open-source 8B model beats GPT-3.5 at bug localization","FlexFL: small open-source LLM outperforms GPT-3.5 on Defects4J","Two-stage FlexFL lets 8B open model trounce GPT-3.5 in FL","Open LLM 8B outdoes proprietary GPT-3.5 in fault finding","FlexFL pipeline: 8B open-source model outshines GPT-3.5"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline is only as good as its first stage: the buggy method must make it into the 20-method candidate list, meaning at least one of the four space-reduction techniques must rank it among its top suggestions, and if none does, the second stage cannot find it no matter how well the language model reasons.","fun_headline_variants_meta":{"raw":{"variants":["Open-source 8B model beats GPT-3.5 at bug localization","FlexFL: small open-source LLM outperforms GPT-3.5 on Defects4J","Two-stage FlexFL lets 8B open model trounce GPT-3.5 in FL","Open LLM 8B outdoes proprietary GPT-3.5 in fault finding","FlexFL pipeline: 8B open-source model outshines GPT-3.5"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000306,"raw_usage":{"total_tokens":1872,"prompt_tokens":1183,"completion_tokens":689,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":799,"completion_tokens_details":{"reasoning_tokens":574}},"tokens_in":799,"tokens_out":689,"duration_ms":10620,"temperature":1.0,"reasoning_tokens":574,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:24:12.458523+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the set of Defects4J bugs that FlexFL currently localizes at Top-1 and rerun FlexFL with the true buggy method artificially removed from the 20-method candidate list before Agent4LR runs; if a substantial fraction are still Top-1 hits, the framework does not actually depend on the candidate list and the stated two-stage design is not the cause of the result, whereas if performance collapses, first-stage recall is confirmed as the ceiling and every missed bug can be traced to the candidate list.","supporting_citations":[{"cited_title":"A quantitative and qualitative evaluation of llm-based explainable fault localization,","cited_arxiv_id":null,"evidence_quote":"Provides the AutoFL baseline (GPT-3.5/GPT-4 with native function calling) that FlexFL must beat, and supplies the repetition-strategy comparison."},{"cited_title":"React: Synergizing reasoning and acting in language models,","cited_arxiv_id":null,"evidence_quote":"Supplies the reason-then-act prompting pattern that FlexFL adapts so open-source chat models can plan and issue function calls without native tool-use support."},{"cited_title":"Defects4j: a database of existing faults to enable controlled testing studies for java programs,","cited_arxiv_id":null,"evidence_quote":"Supplies the Defects4J benchmark (835 version-2.0.0 bugs) on which the paper's main quantitative claims are measured."},{"cited_title":"Better automatic program repair by using bug reports and tests together,","cited_arxiv_id":null,"evidence_quote":"Supplies the SBIR hybrid technique whose top-5 results feed the space-reduction candidate list and which serves as a HybridFL baseline."},{"cited_title":"BoostNSift: A query boosting and code sifting technique for method level bug localization,","cited_arxiv_id":null,"evidence_quote":"Supplies BoostNSift, modified into BoostN, the IRFL technique whose top-5 results feed the candidate list."},{"cited_title":"On the accuracy of spectrum-based fault localization,","cited_arxiv_id":null,"evidence_quote":"Supplies Ochiai, the spectrum-based technique whose top-5 results feed the candidate list and which serves as an SBFL baseline."},{"cited_title":"The GitHub Recent Bugs Dataset for Evaluating LLM-based Debugging Applications","cited_arxiv_id":"2310.13229","evidence_quote":"Supplies the GHRB dataset of post-training-cutoff bugs used to argue that results are not an artifact of data contamination."},{"cited_title":"Blog of Meta Llama 3","cited_arxiv_id":null,"evidence_quote":"Identifies the base model Llama3-8B-Instruct that FlexFL is built on and whose training cutoff defines the contamination check."}],"review_version":1}