{"id":"e35b10eb-869c-4641-b4c5-d3ff026fb735","arxiv_id":"2411.17230","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CosFL converts fault localization into semantic code search: an LLM writes a query describing the faulty functionality and embedding-based retrieval ranks methods, reaching 38.8% Top-1 on 835 Defects4J and GrowingBugs bugs.","lead":"This paper proposes CosFL, a bug-finding tool that first asks a large language model to describe what the buggy code should do, then uses code search to match that description against the project. On 835 real Java bugs it ranked the true buggy method first in 324 cases, outperforming two recent LLM-based fault localization systems.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The query generator can read the same module summaries that retrieval later searches, so the reported gains may largely reflect a feedback loop rather than independent semantic code search.","rationale":"I read the paper in good faith and take its positive results at face value: CosFL localizes 324/835 bugs at Top-1 and beats AgentFL and AutoFL on the reported benchmark. The paper is also careful to acknowledge potential data leakage involving Defects4J V1.2.0 and includes GrowingBugs plus ablation and sensitivity analyses. The most load-bearing weakness is not the benchmark composition or the absence of statistical tests; it is the structural coupling between Query Generation and Fault Retrieval through the shared module index. The query generator is explicitly allowed to retrieve and read module summaries, and those same summaries are later used as retrieval targets. This creates a plausible self-confirmation loop: the LLM can produce queries that quote or closely paraphrase the module summary it was shown, so high cosine similarity between query and summary does not constitute new evidence of faultiness. The paper's own ablation cannot resolve this because 'w/o Module Context' removes the summaries from both sides simultaneously. My proposed concrete test is a controlled decoupling of the two channels; it would settle whether the retrieval component is doing independent work. If the loop is confirmed, the paper's interpretation and novelty claim would need substantial revision, but the empirical result may still stand as a system-level contribution. For these reasons I do not move the reader's conditional verdict; I reinforce it with a sharper, testable requirement. I also credit the paper for transparently describing the query-generation mechanism and for making the ablation that exposes the ambiguity, since without that description the problem would be far harder to identify.","tokens_in":22078,"tokens_out":3357,"duration_ms":34507,"concrete_test":"Run a controlled experiment on the same 835 bugs with two variants: (A) block the LLM's ability to request module details during Query Generation (no module summaries in the prompt), while leaving all module summaries in the retrieval indexes for Fault Retrieval; (B) keep Query Generation as-is but remove the module summaries from the retrieval indexes (retrieve only over method and chunk summaries). If variant A's Top-1 is close to the default 324, the feedback loop is not load-bearing. If A drops toward the 244 of the 'w/o Module Context' ablation, the gain is driven by the query generator reading the summaries, not by retrieval. Additionally, for the bugs where variant A fails but the default succeeds, compute n-gram/embedding overlap between the generated query and previously supplied module summaries to confirm the leakage mechanism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"CosFL's central claim is that decomposing FL into LLM-based query generation plus semantic code search retrieval produces the reported 324/835 Top-1 results. That claim requires the retrieval step to supply independent evidence of faultiness. The design does not guarantee this: in Query Generation (Section 3.2), the LLM actively requests module-level knowledge, and the paper states this knowledge is found by 'semantic search' over the module index. The same module summaries are then indexed again in Fault Retrieval (Section 3.3, Formulas 2-4) and matched against the generated queries. Consequently, the generated module-level (and possibly method/chunk-level) query can be a near-paraphrase of the very summary that retrieval will later match, inflating cosine similarity without adding causal evidence about where the bug is. The ablation in Section 5.2 ('w/o Module Context', Top-1 drops from 324 to 244) shows module knowledge matters a lot, but that variant removes module summaries from both query generation and retrieval, so it cannot separate the two channels. If the improvement comes mainly from feeding the summaries to the query generator, the 'semantic code search' framing overstates what the retrieval component contributes. This does not invalidate CosFL's practical performance, but it does undermine the paper's interpretation of why it works and its claimed novelty as a search-driven FL paradigm.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CosFL, a method-level fault localization approach that reframes FL as a semantic code search problem. CosFL first builds a software knowledge base by instrumenting the buggy program, constructing a dynamic method call graph, decomposing it into modules via the Leiden algorithm, and using an LLM to generate module-, method-, and chunk-level natural-language summaries. In the query generation phase, an LLM analyzes the failing-test information and can iteratively request module summaries through semantic search before emitting module-, method-, and chunk-level queries. In the fault retrieval phase, those queries are matched by cosine similarity against the same summaries, and a Borda-like voting scheme aggregates retrieval results into a ranked list of methods. The evaluation on 835 real bugs from Defects4J and GrowingBugs reports 324 Top-1 localizations, outperforming AgentFL and AutoFL on Top-1 by 26.6% and 57.3%, respectively, with additional ablation, sensitivity analysis, and case studies.","tokens_in":2040,"tokens_out":2605,"duration_ms":91198,"significance":"If the claims hold, CosFL is a valuable new angle for LLM-based fault localization: it converts fault information into semantic queries and retrieves program elements across multiple granularities. The paper's strengths include the large benchmark of 835 bugs across 23 projects, detailed per-project results, an ablation study isolating three components, a sensitivity analysis across LLM and embedding backends, and an explicit acknowledgment of the Defects4J V1.2.0 leakage threat. However, the central interpretation that semantic code search retrieval drives the gains is not yet established, because the query generator can consume the same module summaries that retrieval later matches, and the provided ablation cannot separate these channels. The comparative claims also lack statistical testing, the headline totals include a leakage-prone subset, and the metric definitions are internally inconsistent. If these gaps are closed, the work would be a solid contribution; as presented, the evidence supports the practical ranking result but not all of the paper's explanatory claims.","major_comments":[{"comment":"The design creates a potential feedback loop between query generation and fault retrieval. In Query Generation, when the LLM requests additional information, CosFL performs semantic search over the Module Index and inserts the retrieved module summary into Prompt 3 as Module Details; the resulting module-level query is therefore generated after the LLM has already seen the text of that summary. In Fault Retrieval, Eq. (3) computes cosine similarity between the same query and embeddings of the same module summaries, so high similarity can reflect paraphrase agreement with text the LLM was shown rather than independent evidence of faultiness. The 'w/o Module Context' ablation in Table 3 removes module summaries from both query generation and retrieval, so the Top-1 drop from 324 to 244 cannot be attributed to either channel. I request that the two channels be separated: (a) keep module summaries in retrieval but exclude them from query generation, and (b) keep them in query generation but exclude module retrieval. This separation is necessary to support the paper's central claim that semantic code search retrieval, rather than knowledge feeding into query generation, drives the reported gains.","section":"3.2, 3.3, Table 3"},{"comment":"The paper acknowledges in section 6.3 that Defects4J V1.2.0 may have been seen by the LLM during training, but all headline totals in Table 2, including the 324 Top-1 count, are computed over the union of Defects4J V1.2.0 (370 bugs), Defects4J V2.0.0 (321 bugs), and GrowingBugs (144 bugs). The V1.2.0 subset is exactly the most widely used FL benchmark, so the aggregate comparison with AgentFL and AutoFL can be inflated even if the baselines are equally exposed. Please report Top-1/Top-5/Top-10/MFR/MAR separately for each benchmark subset for all three methods, and preferably rerun the experiments on a model whose training data are known not to include these benchmarks, or on newly constructed bug versions. Without this, the paper's cross-benchmark generalization claim is not supported by the reported aggregate numbers.","section":"5.1, 6.3, Table 2"},{"comment":"The paper uses temperature 1.0 for the LLM backend, yet every reported number appears to come from a single run and no significance tests are provided. The RQ1 answer states that CosFL 'significantly outperforms' the baselines; with stochastic query generation, this requires either repeated runs with variance reported or a paired statistical test (e.g., Wilcoxon signed-rank or bootstrap) across the 835 bugs. A related issue is that the definition of MFR in section 4.4 is inconsistent with the reported values: if a bug with no hit in the recall set of size N is assigned rank N+1, then for N=10 the MFR values in Table 2 (e.g., 35.78, 26.80, and project-level values up to 51.00) are impossible. Please clarify the exact ranking protocol, including what rank is assigned to methods outside the retrieved list, and recompute or correct the table accordingly.","section":"4.4, 5.1, Table 2"},{"comment":"The MAR definition in section 4.4 appears incorrect as written: MAR = (1/(|B||M|)) * sum_{i=1..|B|} sum_{j=1..|M|} rank_ij averages over every method in the project rather than over the buggy methods for each bug. The standard MAR in fault localization is the average rank of the actual buggy methods, and the stated formula would be dominated by the total number of methods in each project. Because MFR and MAR are reported as part of the main comparison in Table 2 and cited in the RQ1 answer, the metric definitions must be corrected and the values verified.","section":"4.4, Table 2"},{"comment":"The comparison with FLUCCS, DeepFL, and GRACE is not controlled. Those learning-based methods are trained on Defects4J V1.2.0 and evaluated on V2.0.0 in a cross-project setting, whereas CosFL does not train on V1.2.0 but uses each bug's failing-test and execution information at inference time. The 20 to 48 point Top-1 advantages in Table 4 may therefore reflect CosFL's access to bug-specific runtime and failure information rather than the superiority of the code-search paradigm. Please report the same comparison for AutoFL and AgentFL on the V2.0.0 subset, describe exactly what information each learning-based baseline receives for a given bug, and discuss whether the comparison can be made information-equivalent.","section":"6.2, Table 4"}],"minor_comments":[{"comment":"The text says AutoFL achieves 'Top1, Top3, and Top5 scores of 206, 253, and 254', but Table 2 reports T1, T5, and T10; this should be corrected to avoid confusion.","section":"5.1"},{"comment":"The horizontal axis labels in Figure 5 appear garbled ('T1 T5 50-MFR 50-MAR T10'); the figure should clarify that 50 refers to the default retrieval amount and label the two vertical axes explicitly.","section":"Figure 5"},{"comment":"The table header '# Bugs Techniques' is malformed; the number of bugs (280) and the technique names should be presented as separate columns or structured rows.","section":"Table 4"},{"comment":"The maximum and minimum module sizes (15 and 5) are set as hyper-parameters, but the sensitivity analysis in RQ3 does not vary these bounds, so the robustness claim is only partial.","section":"3.1"},{"comment":"The manuscript does not include an artifact availability statement or a link to the implementation and benchmark scripts; providing these would substantially improve reproducibility, especially given the many LLM and embedding components involved.","section":"General"},{"comment":"The ACM Reference Format block still contains placeholder venue information and year '2018'; this should be updated to the target venue and submission year before publication.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The most important risk is the circularity between query generation and retrieval through shared module summaries. If the authors can run the two-channel separation I describe, the paper's central interpretation can be tested directly. I also recommend that the editors require the benchmark-subset split and the corrected MFR/MAR definitions before considering the paper, since the current aggregate and metric reporting make the comparative claims difficult to assess."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"CosFL is a serious, well-written empirical paper that gives FL a genuinely new framing: generate a natural-language description of the buggy functionality, then retrieve methods by semantic similarity. That is a real departure from the navigation-based LLM agents (AutoFL, AgentFL) and from keyword IRFL. The evaluation on 835 bugs is substantial, the gains over the two LLM baselines are large and consistent across metrics, and the ablation and sensitivity analysis are honest. The authors also acknowledge the Defects4J V1.2.0 training-data risk and show results on V2.0.0 and GrowingBugs, which is the right instinct.\n\nThe soft spots are real, though not fatal. The biggest one is the feedback loop: in query generation (Section 3.2), the LLM can request module summaries that are retrieved from the very module index that the fault-retrieval stage (Section 3.3) later searches with the generated query. The query can become a near-paraphrase of the summary it already saw, so high cosine similarity does not demonstrate independent evidence of faultiness. The ablation 'w/o Module Context' removes module context from both query generation and retrieval, so it cannot separate the two channels. I would want an ablation that removes module context only from query generation, or only from retrieval, to see where the Top-1 gain actually comes from. If the gain is mostly from feeding summaries to the query generator, then the 'semantic code search' framing overstates what retrieval contributes.\n\nAlso missing: significance tests. The Top-1 differences look large, but with no error bars or paired tests, it is hard to know how stable they are. And there is no artifact release, so the numbers cannot be checked. That is a weaker reason, but it matters for a paper whose contribution is empirical.\n\nBottom line: the paper deserves a serious referee, and I would engage with it, but it needs revision before acceptance. The authors should separate the two channels, add significance testing, and release code and data. If the retrieval channel is indeed doing the work, this is a meaningful advance; if not, it is still a useful data point on LLM-based FL, just with a less exciting story.","headline":"A serious, well-written FL paper with a genuinely new framing, but a possible feedback loop between query generation and retrieval means the 'search' story may be partly an artifact of feeding the same summaries to the LLM.","tokens_in":22881,"tokens_out":2687,"would_cite":false,"duration_ms":24936,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fault localization can be recast as semantic code search, and an LLM-query retrieval pipeline ranks the true buggy method first for 324 of 835 real bugs.","keywords":["fault localization","code search","large language models","semantic retrieval","method-level localization","dynamic call graph","Borda voting","Defects4J"],"falsifier":"Blind the query generator: remove the module-summary retrieval from the query-generation loop (or feed it module summaries from a different project) while keeping the retrieval index unchanged, and check whether Top-1 recall falls by roughly the 80 bugs that the ablation attributes to module context. If the drop is much smaller, the reported gain is an artifact of the knowledge-base feedback loop, not of semantic code search. A second check: embed a hallucinated query that describes the same functionality but refers to a method that does not exist, and see whether the true buggy method still appears at the top because its summary text overlaps the module context already shown to the query generator.","tokens_in":21843,"feed_emoji":"🐛","tokens_out":10649,"duration_ms":82962,"temperature":0.7,"pith_summary":"The paper proposes CosFL, a method-level fault localizer that treats debugging as a semantic code-search problem. Given failing tests and stack traces, an LLM first writes a natural-language description of what the buggy code should be doing, at the granularity of a module, a method, and a chunk, and may ask for module-level knowledge before committing to a query. A retrieval step then embeds these queries together with LLM-written summaries of all program elements and ranks methods by cosine similarity, aggregated across granularities with a Borda-style vote. On 835 real Java bugs, CosFL places the buggy method at the top rank in 324 cases (38.8%), exceeding AgentFL and AutoFL by 26.6% and 57.3% on Top-1. The paper's claim is that supplying explicit semantic descriptions of the faulty functionality is what fault localizers have been missing, and that code-search machinery can then do the rest.","feed_headline":"Semantic code search ranks 324 of 835 real bugs at the top spot","feed_subtitle":"Turning a failing test into a plain-language query lets retrieval beat navigation-based LLM debuggers.","key_machinery":"The load-bearing mechanism is a multi-granularity semantic index paired with an LLM-generated query. Modules are discovered by applying Leiden community detection to a weighted dynamic method call graph built by instrumentation; the LLM summarizes each module as a title, summary, and findings, and then summarizes each method and each chunk conditioned on the enclosing module summary. Retrieval (Eq. 2–5) embeds these summaries and the multi-granularity queries with a text embedding model $\\sigma$, returns neighbors by cosine similarity, and uses the methods retrieved at method level to narrow the module and chunk indexes. A Borda-inspired voting rule (Eq. 6–9) turns the similarities into a suspiciousness score: a method gains credit when the module it belongs to matches the module query, when its own summary matches the method query, and when any of its chunks matches the chunk query. The design lets a method rank high even when its own summary is not the closest match, as long as the surrounding module and chunk evidence agrees.","core_discovery":"CosFL's central claim is that fault localization is a search problem in disguise: the missing ingredient is not better navigation but a natural-language description of the faulty functionality. The pipeline first builds a software knowledge base by instrumenting the program to obtain a dynamic call graph, partitioning it into functional modules with the Leiden community-detection algorithm, and having an LLM write summaries for each module, method, and chunk. During query generation, the LLM examines failing tests, stack traces, and test output, asks for module summaries when uncertain, and finally emits module-, method-, and chunk-level queries describing the suspected fault. Fault retrieval embeds queries and summaries in the same vector space, scores them by cosine similarity, and a Borda-inspired vote accumulates module, method, and chunk similarities from all failing tests into per-method suspiciousness scores. The paper reports Top-1, Top-5, and Top-10 counts of 324, 532, and 566 out of 835, with ablation showing that removing module context costs 80 Top-1 bugs while removing chunk retrieval costs 19.","pith_inferences":["Editorial inference: because the query generator may read the same module summaries that the retrieval index contains, part of the Top-1 gain may come from the LLM writing queries that echo text it has already seen, rather than from independent causal diagnosis; a blinded query-generation variant would quantify this.","Editorial inference: the paper itself limits the evaluation to Java projects with JUnit-style tests (Section 6.3), so the same query-then-retrieve pipeline remains untested in other languages and test frameworks.","Editorial inference: the semantic retrieval step ignores execution coverage, so a hybrid that weights retrieval scores by failure-triggering coverage or stack-trace proximity could plausibly raise Top-1 further.","Editorial inference: if the query-description step transfers, the same knowledge base and indexes could serve code repair, test-suite minimization, or API misuse detection, but those applications are not explored in the paper."],"forward_implications":["Method-level fault localization for real Java bugs can reach a 38.8% Top-1 rate without training on historical bugs, using only LLM-generated text and embedding retrieval.","The effectiveness gap between code search and fault localization is in part an input-representation gap: giving a localizer a natural-language description of the buggy functionality is enough to outperform navigation-based LLM agents.","The pipeline is compositionally robust: swapping the LLM backend or the embedding model changes Top-1 counts within about 2%, so the approach does not depend on one specific model.","Chunk-level and module-level retrieval are complementary: the ablation assigns the largest single loss (80 Top-1 bugs) to removing module context during query generation, while chunk retrieval contributes the larger retrieval-side gain.","Cross-project, the retrieval-based approach beats learning-based FL (105 vs 85 Top-1 for GRACE) on Defects4J V2.0.0 without retraining."],"supporting_citations":[{"why":"AutoFL, the LLM function-calling navigation baseline that CosFL must outperform; defines the covered-class/method tool-calling protocol and the Top-1 baseline of 206.","marker":"[Kang et al. 2024]"},{"why":"AgentFL, the strongest previous project-level LLM fault localizer; provides the document-guided navigation baseline and the 256 Top-1 count CosFL improves.","marker":"[Qin et al. 2024]"},{"why":"Supplies Defects4J, the benchmark for 691 of the 835 bugs.","marker":"[Just et al. 2014]"},{"why":"Supplies GrowingBugs, the additional 144-bug benchmark used to broaden the evaluation.","marker":"[Jiang et al. 2022]"},{"why":"Provides the Leiden community-detection algorithm used to partition the dynamic call graph into functional modules.","marker":"[Traag et al. 2019]"},{"why":"Establishes the deep semantic code-search approach that the retrieval step is modeled on.","marker":"[Gu et al. 2018]"},{"why":"Borda count, the social-choice method that inspires the multi-granularity suspicious-method voting.","marker":"[Emerson 2013]"},{"why":"GRACE, the graph-representation-learning FL baseline that CosFL compares against and beats cross-project on Defects4J V2.0.0.","marker":"[Lou et al. 2021]"}],"fun_headline_variants":["Treating bugs as search queries finds 324 top-ranked faults","CosFL turns failing tests into queries to rank bugs","LLM-generated bug queries lift fault localization to 324 top-1","Code search beats navigation: 324 bugs in top-1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the assumption that cosine similarity between the LLM's description of the bug and the LLM's own code summaries marks the buggy method; because the description is written after the same LLM has seen some of those summaries, high similarity can result from the system agreeing with itself rather than from evidence that the code is at fault.","fun_headline_variants_meta":{"raw":{"variants":["Treating bugs as search queries finds 324 top-ranked faults","CosFL turns failing tests into queries to rank bugs","LLM-generated bug queries lift fault localization to 324 top-1","Code search beats navigation: 324 bugs in top-1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00025,"raw_usage":{"total_tokens":1616,"prompt_tokens":1071,"completion_tokens":545,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":687,"completion_tokens_details":{"reasoning_tokens":474}},"tokens_in":687,"tokens_out":545,"duration_ms":5676,"temperature":1.0,"reasoning_tokens":474,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:24:50.074566+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Blind the query generator: remove the module-summary retrieval from the query-generation loop (or feed it module summaries from a different project) while keeping the retrieval index unchanged, and check whether Top-1 recall falls by roughly the 80 bugs that the ablation attributes to module context. If the drop is much smaller, the reported gain is an artifact of the knowledge-base feedback loop, not of semantic code search. A second check: embed a hallucinated query that describes the same functionality but refers to a method that does not exist, and see whether the true buggy method still appears at the top because its summary text overlaps the module context already shown to the query generator.","supporting_citations":[],"review_version":1}