Pith. sign in

REVIEW 5 major objections 6 minor 3 references

Fault Localization from the Semantic Code Search Perspective

T0 review · 5 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fault localization can be recast as semantic code search, and an LLM-query retrieval pipeline ranks the true buggy method first for 324 of 835 real bugs.

desk verdict A serious, well-written FL paper with a genuinely new framing, but a possible feedback loop between query generation and retrieval means the 'search' story may be partly an artifact of feeding the same summaries to the LLM. read the letter →

arxiv 2411.17230 v1 pith:SZ4PFQRX submitted 2024-11-26 cs.SE

Yihao Qin , Shangwen Wang , Yan Lei , Zhuo Zhang , Bo Lin , Xin Peng , Liqian Chen , Xiaoguang Mao This is my paper · ORCID
classification cs.SE
keywords faultlocalizationcodesearchlargelanguagemodelssemanticretrievalmethod-leveldynamiccallgraphBordavotingDefects4J
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes CosFL, a method-level fault localizer that treats debugging as a semantic code-search problem. Given failing tests and stack traces, an LLM first writes a natural-language description of what the buggy code should be doing, at the granularity of a module, a method, and a chunk, and may ask for module-level knowledge before committing to a query. A retrieval step then embeds these queries together with LLM-written summaries of all program elements and ranks methods by cosine similarity, aggregated across granularities with a Borda-style vote. On 835 real Java bugs, CosFL places the buggy method at the top rank in 324 cases (38.8%), exceeding AgentFL and AutoFL by 26.6% and 57.3% on Top-1. The paper's claim is that supplying explicit semantic descriptions of the faulty functionality is what fault localizers have been missing, and that code-search machinery can then do the rest.

What carries the argument

The load-bearing mechanism is a multi-granularity semantic index paired with an LLM-generated query. Modules are discovered by applying Leiden community detection to a weighted dynamic method call graph built by instrumentation; the LLM summarizes each module as a title, summary, and findings, and then summarizes each method and each chunk conditioned on the enclosing module summary. Retrieval (Eq. 2–5) embeds these summaries and the multi-granularity queries with a text embedding model $\sigma$, returns neighbors by cosine similarity, and uses the methods retrieved at method level to narrow the module and chunk indexes. A Borda-inspired voting rule (Eq. 6–9) turns the similarities into a suspiciousness score: a method gains credit when the module it belongs to matches the module query, when its own summary matches the method query, and when any of its chunks matches the chunk query. The design lets a method rank high even when its own summary is not the closest match, as long as the surrounding module and chunk evidence agrees.

What would settle it

Blind the query generator: remove the module-summary retrieval from the query-generation loop (or feed it module summaries from a different project) while keeping the retrieval index unchanged, and check whether Top-1 recall falls by roughly the 80 bugs that the ablation attributes to module context. If the drop is much smaller, the reported gain is an artifact of the knowledge-base feedback loop, not of semantic code search. A second check: embed a hallucinated query that describes the same functionality but refers to a method that does not exist, and see whether the true buggy method still appears at the top because its summary text overlaps the module context already shown to the query generator.

Watch

Extended reading notes

Core claim

CosFL's central claim is that fault localization is a search problem in disguise: the missing ingredient is not better navigation but a natural-language description of the faulty functionality. The pipeline first builds a software knowledge base by instrumenting the program to obtain a dynamic call graph, partitioning it into functional modules with the Leiden community-detection algorithm, and having an LLM write summaries for each module, method, and chunk. During query generation, the LLM examines failing tests, stack traces, and test output, asks for module summaries when uncertain, and finally emits module-, method-, and chunk-level queries describing the suspected fault. Fault retrieval embeds queries and summaries in the same vector space, scores them by cosine similarity, and a Borda-inspired vote accumulates module, method, and chunk similarities from all failing tests into per-method suspiciousness scores. The paper reports Top-1, Top-5, and Top-10 counts of 324, 532, and 566 out of 835, with ablation showing that removing module context costs 80 Top-1 bugs while removing chunk retrieval costs 19.

Load-bearing premise

Everything rests on the assumption that cosine similarity between the LLM's description of the bug and the LLM's own code summaries marks the buggy method; because the description is written after the same LLM has seen some of those summaries, high similarity can result from the system agreeing with itself rather than from evidence that the code is at fault.

Editorial extensions

If this is right

  • Method-level fault localization for real Java bugs can reach a 38.8% Top-1 rate without training on historical bugs, using only LLM-generated text and embedding retrieval.
  • The effectiveness gap between code search and fault localization is in part an input-representation gap: giving a localizer a natural-language description of the buggy functionality is enough to outperform navigation-based LLM agents.
  • The pipeline is compositionally robust: swapping the LLM backend or the embedding model changes Top-1 counts within about 2%, so the approach does not depend on one specific model.
  • Chunk-level and module-level retrieval are complementary: the ablation assigns the largest single loss (80 Top-1 bugs) to removing module context during query generation, while chunk retrieval contributes the larger retrieval-side gain.
  • Cross-project, the retrieval-based approach beats learning-based FL (105 vs 85 Top-1 for GRACE) on Defects4J V2.0.0 without retraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the query generator may read the same module summaries that the retrieval index contains, part of the Top-1 gain may come from the LLM writing queries that echo text it has already seen, rather than from independent causal diagnosis; a blinded query-generation variant would quantify this.
  • Editorial inference: the paper itself limits the evaluation to Java projects with JUnit-style tests (Section 6.3), so the same query-then-retrieve pipeline remains untested in other languages and test frameworks.
  • Editorial inference: the semantic retrieval step ignores execution coverage, so a hybrid that weights retrieval scores by failure-triggering coverage or stack-trace proximity could plausibly raise Top-1 further.
  • Editorial inference: if the query-description step transfers, the same knowledge base and indexes could serve code repair, test-suite minimization, or API misuse detection, but those applications are not explored in the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes CosFL, a method-level fault localization approach that reframes FL as a semantic code search problem. CosFL first builds a software knowledge base by instrumenting the buggy program, constructing a dynamic method call graph, decomposing it into modules via the Leiden algorithm, and using an LLM to generate module-, method-, and chunk-level natural-language summaries. In the query generation phase, an LLM analyzes the failing-test information and can iteratively request module summaries through semantic search before emitting module-, method-, and chunk-level queries. In the fault retrieval phase, those queries are matched by cosine similarity against the same summaries, and a Borda-like voting scheme aggregates retrieval results into a ranked list of methods. The evaluation on 835 real bugs from Defects4J and GrowingBugs reports 324 Top-1 localizations, outperforming AgentFL and AutoFL on Top-1 by 26.6% and 57.3%, respectively, with additional ablation, sensitivity analysis, and case studies.

Significance. If the claims hold, CosFL is a valuable new angle for LLM-based fault localization: it converts fault information into semantic queries and retrieves program elements across multiple granularities. The paper's strengths include the large benchmark of 835 bugs across 23 projects, detailed per-project results, an ablation study isolating three components, a sensitivity analysis across LLM and embedding backends, and an explicit acknowledgment of the Defects4J V1.2.0 leakage threat. However, the central interpretation that semantic code search retrieval drives the gains is not yet established, because the query generator can consume the same module summaries that retrieval later matches, and the provided ablation cannot separate these channels. The comparative claims also lack statistical testing, the headline totals include a leakage-prone subset, and the metric definitions are internally inconsistent. If these gaps are closed, the work would be a solid contribution; as presented, the evidence supports the practical ranking result but not all of the paper's explanatory claims.

major comments (5)
  1. [3.2, 3.3, Table 3] The design creates a potential feedback loop between query generation and fault retrieval. In Query Generation, when the LLM requests additional information, CosFL performs semantic search over the Module Index and inserts the retrieved module summary into Prompt 3 as Module Details; the resulting module-level query is therefore generated after the LLM has already seen the text of that summary. In Fault Retrieval, Eq. (3) computes cosine similarity between the same query and embeddings of the same module summaries, so high similarity can reflect paraphrase agreement with text the LLM was shown rather than independent evidence of faultiness. The 'w/o Module Context' ablation in Table 3 removes module summaries from both query generation and retrieval, so the Top-1 drop from 324 to 244 cannot be attributed to either channel. I request that the two channels be separated: (a) keep module summaries in retrieval but exclude them from query generation, and (b) keep them in query generation but exclude module retrieval. This separation is necessary to support the paper's central claim that semantic code search retrieval, rather than knowledge feeding into query generation, drives the reported gains.
  2. [5.1, 6.3, Table 2] The paper acknowledges in section 6.3 that Defects4J V1.2.0 may have been seen by the LLM during training, but all headline totals in Table 2, including the 324 Top-1 count, are computed over the union of Defects4J V1.2.0 (370 bugs), Defects4J V2.0.0 (321 bugs), and GrowingBugs (144 bugs). The V1.2.0 subset is exactly the most widely used FL benchmark, so the aggregate comparison with AgentFL and AutoFL can be inflated even if the baselines are equally exposed. Please report Top-1/Top-5/Top-10/MFR/MAR separately for each benchmark subset for all three methods, and preferably rerun the experiments on a model whose training data are known not to include these benchmarks, or on newly constructed bug versions. Without this, the paper's cross-benchmark generalization claim is not supported by the reported aggregate numbers.
  3. [4.4, 5.1, Table 2] The paper uses temperature 1.0 for the LLM backend, yet every reported number appears to come from a single run and no significance tests are provided. The RQ1 answer states that CosFL 'significantly outperforms' the baselines; with stochastic query generation, this requires either repeated runs with variance reported or a paired statistical test (e.g., Wilcoxon signed-rank or bootstrap) across the 835 bugs. A related issue is that the definition of MFR in section 4.4 is inconsistent with the reported values: if a bug with no hit in the recall set of size N is assigned rank N+1, then for N=10 the MFR values in Table 2 (e.g., 35.78, 26.80, and project-level values up to 51.00) are impossible. Please clarify the exact ranking protocol, including what rank is assigned to methods outside the retrieved list, and recompute or correct the table accordingly.
  4. [4.4, Table 2] The MAR definition in section 4.4 appears incorrect as written: MAR = (1/(|B||M|)) * sum_{i=1..|B|} sum_{j=1..|M|} rank_ij averages over every method in the project rather than over the buggy methods for each bug. The standard MAR in fault localization is the average rank of the actual buggy methods, and the stated formula would be dominated by the total number of methods in each project. Because MFR and MAR are reported as part of the main comparison in Table 2 and cited in the RQ1 answer, the metric definitions must be corrected and the values verified.
  5. [6.2, Table 4] The comparison with FLUCCS, DeepFL, and GRACE is not controlled. Those learning-based methods are trained on Defects4J V1.2.0 and evaluated on V2.0.0 in a cross-project setting, whereas CosFL does not train on V1.2.0 but uses each bug's failing-test and execution information at inference time. The 20 to 48 point Top-1 advantages in Table 4 may therefore reflect CosFL's access to bug-specific runtime and failure information rather than the superiority of the code-search paradigm. Please report the same comparison for AutoFL and AgentFL on the V2.0.0 subset, describe exactly what information each learning-based baseline receives for a given bug, and discuss whether the comparison can be made information-equivalent.
minor comments (6)
  1. [5.1] The text says AutoFL achieves 'Top1, Top3, and Top5 scores of 206, 253, and 254', but Table 2 reports T1, T5, and T10; this should be corrected to avoid confusion.
  2. [Figure 5] The horizontal axis labels in Figure 5 appear garbled ('T1 T5 50-MFR 50-MAR T10'); the figure should clarify that 50 refers to the default retrieval amount and label the two vertical axes explicitly.
  3. [Table 4] The table header '# Bugs Techniques' is malformed; the number of bugs (280) and the technique names should be presented as separate columns or structured rows.
  4. [3.1] The maximum and minimum module sizes (15 and 5) are set as hyper-parameters, but the sensitivity analysis in RQ3 does not vary these bounds, so the robustness claim is only partial.
  5. [General] The manuscript does not include an artifact availability statement or a link to the implementation and benchmark scripts; providing these would substantially improve reproducibility, especially given the many LLM and embedding components involved.
  6. [General] The ACM Reference Format block still contains placeholder venue information and year '2018'; this should be updated to the target venue and submission year before publication.

Circularity Check

1 steps flagged · score 4.0 of 10

Module-level query generation reuses the same module summaries that module retrieval later searches, making the module vote partially self-confirming; method- and chunk-level retrieval retain independent code-derived content.

  1. self definitional [Section 3.2 (Query Generation) and Section 3.3, Formula 3; Prompt 3]
    "In this condition, we use semantic search to find the most relevant module summary from the module index and add it to the Module Details section within # Fault Information. ... ˆGi ={ ˆg1,..., ˆg| ˆGi|} = Retrieve(IG p,ci,λ)"

    The module summary shown to the LLM during query generation is an element of the same module index IG that Formula 3 searches. The generated module-level query ci is therefore produced after the LLM has already seen a candidate module's summary; when ci paraphrases that summary, cosine retrieval against the same index will rank that module highly, and Formula 7 transfers this module similarity into the suspiciousness score of every method in that module. This is a feedback loop: the module-level 'evidence' of faultiness is the system recovering text it already supplied to the query generator, rather than an independent signal.

full rationale

CosFL's central claim is the 324/835 Top-1 method-level result, and that claim is not derived from fitted parameters or from a self-citation chain; the evaluation is against external real-world benchmarks (Defects4J and GrowingBugs) and the method/chunk summaries are generated from the actual code, comments, and runtime call graphs, giving the retrieval independent empirical content. However, the module-level component does contain a by-construction feedback loop: the query generator can request and receive module summaries from the module index, and the same index is then searched by the module-level query to produce a module vote. For bugs where the LLM relies on the retrieved summary, the module retrieval measures how well the query paraphrases a summary it was already shown, not new evidence about faultiness. This partially inflates the contribution attributed to 'semantic code search', especially since the w/o Module Context ablation removes module knowledge from both query generation and retrieval and cannot separate the two channels. The AgentFL baseline is prior work by the same group, but it functions as a comparative baseline rather than a load-bearing premise of the derivation, so it does not raise the score further. Overall the circularity is real but partial, hence a score of 4 rather than 6-8.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on several unproven premises: that buggy methods are always covered and indexed, that LLM summaries are faithful, and that embedding similarity between generated queries and summaries is a valid proxy for fault location. The pipeline also has a feedback loop in which query generation can consume module summaries from the same retrieval index, weakening the independence of the search signal. These are not free parameters in a derivation; they are empirical assumptions. The paper's main free parameters are retrieval count, module size bounds, and LLM temperature.

free parameters (3)
  • lambda (retrieved method count) = 50 (default; 25, 50, 75 tested)
    Controls how many methods from method retrieval are considered before module and chunk retrieval are narrowed. Chosen by sensitivity analysis, not derived.
  • Module size bounds = max=15, min=5
    Heuristic limits on Leiden community size to keep LLM summaries manageable; no theoretical basis.
  • LLM temperature = 1.0
    Set to 1.0 for query generation and summarization; affects determinism and reproducibility, not derived from data.
assumptions (5)
  • domain assumption Buggy methods are covered by the failing test execution and therefore appear in the dynamic call graph used to build the index.
    Section 3.1 excludes methods not executed at runtime. If a buggy method is not executed, it cannot be retrieved. This premise is plausible for Defects4J but not guaranteed for all bugs.
  • domain assumption LLM-generated module, method, and chunk summaries accurately represent code semantics and contain no hallucinated functionality.
    Section 3.1 uses LLM outputs as ground-truth knowledge for both query generation and retrieval. Hallucinated summaries would corrupt the index.
  • ad hoc to paper Cosine similarity between query text and summary text is a valid proxy for the location of the fault.
    Section 3.3 Formula 2 treats semantic similarity as suspiciousness. The paper provides no evidence that this mapping holds independent of the pipeline's feedback loop.
  • domain assumption The LLM can produce a useful description of faulty functionality from fault information (test, stack trace, output) alone, possibly after retrieving module details.
    Section 3.2 Query Generation. Query quality is not independently measured; the paper only measures final localization accuracy.
  • standard math Leiden algorithm with modularity optimization identifies functional modules relevant to the bug.
    Section 3.1 Module Detection. Standard community detection invoked without domain-specific validation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fault Localization from the Semantic Code Search Perspective." pith.science (2026). https://pith.science/paper/SZ4PFQRX

@misc{pith2026241117230,
  author       = {Pith},
  title        = {Pith review of: Fault Localization from the Semantic Code Search Perspective},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SZ4PFQRX}},
  note         = {Machine review of arXiv:2411.17230}
}
read the original abstract

The software development process is characterized by an iterative cycle of continuous functionality implementation and debugging, essential for the enhancement of software quality and adaptability to changing requirements. This process incorporates two isolatedly studied tasks: Code Search (CS), which retrieves reference code from a code corpus to aid in code implementation, and Fault Localization (FL), which identifies code entities responsible for bugs within the software project to boost software debugging. These two tasks exhibit similarities since they both address search problems. Notably, CS techniques have demonstrated greater effectiveness than FL ones, possibly because of the precise semantic details of the required code offered by natural language queries, which are not readily accessible to FL methods. Drawing inspiration from this, we hypothesize that a fault localizer could achieve greater proficiency if semantic information about the buggy methods were made available. Based on this idea, we propose CosFL, an FL approach that decomposes the FL task into two steps: query generation, which describes the functionality of the problematic code in natural language, and fault retrieval, which uses CS to find program elements semantically related to the query. Specifically, to depict the buggy functionalities and generate high-quality queries, CosFL extensively harnesses the code analysis, semantic comprehension, and decision-making capabilities of LLMs. Moreover, to enhance the accuracy of CS, CosFL captures varying levels of context information and employs a multi-granularity code search strategy, which facilitates a more precise identification of buggy methods from a holistic view. The evaluation on 835 real bugs from 23 Java projects shows that CosFL successfully localizes 324 bugs within Top-1, which significantly outperforms the state-of-the-art approaches by 26.6%-57.3%.

Figures

Figures reproduced from arXiv: 2411.17230 by the authors.

Figure 1
Figure 1. Illustration of CosFL and other FL pipelines. The rationale is generated by LLMs. 1 INTRODUCTION The software development process is an iterative cycle of continuous functionality implementation and debugging [Katz and Anderson 1987]. During the implementation phase, programmers write code that meets specific requirements, but may simultaneously introduce bugs into the software system. In the debugging phase, develo… view at source ↗
Figure 2
Figure 2. Concept diagram of localizing Closure-1 bug from the perspective of semantic code search. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Overview of CosFL. 3 APPROACH In this work, we propose CosFL, a new paradigm for Fault Localization from the perspective of semantic COde Search. CosFL primarily focuses on method-level FL, since this granularity level is most preferred by developers [Kochhar et al. 2016] and also serves as the foundation for many automated software development techniques [Yang et al. 2024]. To achieve better compatibility between C… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Overlap of Top5 results. In [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: The impact of different hyper-parameters on [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Comparison between AgentFL and CosFL in localizing bug Closure-41. The keywords that AgentFL and CosFL focus on are marked in gold and red , respectively. Failed Test Case Fault Information Exception Trace AutoFL Function AutoFL Buggy Method . . . Module Knowledge CosF…
Figure 7
Figure 7. Figure 7: Comparison between AutoFL and CosFL in localizing bug Jsoup-2. The keywords that AutoFL and CosFL focus on are marked in gold and red , respectively. inferFromOverriddenFunction “serves as the primary mechanism for inferring function types from overridden functions”. T…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

3 extracted references · 1 linked inside Pith

  1. [3]

    ACM Trans

    A Survey of Source Code Search: A 3-Dimensional Perspective. ACM Trans. Softw. Eng. Methodol. 33, 6, Article 166 (June 2024), 51 pages. https://doi.org/10.1145/3656341 Tan Thongtan and Tanasanee Phienthrakul. 2019. Sentiment Classification Using Document Embeddings Trained with Cosine Similarity. In Proceedings of the 57th Annual Meeting of the Associatio...

    arXiv 2019

  2. [2022]

    IEEE Transactions on Software Engineering 49, 4 (2022), 1443–1463

    Bugbuilder: An automated approach to building bug repository. IEEE Transactions on Software Engineering 49, 4 (2022), 1443–1463. https://doi.org/10.1109/TSE.2022.3177713 Haolin Jin, Linghan Huang, Haipeng Cai, Jun Yan, Bo Li, and Huaming Chen. 2024. From llms to llm-based agents for software engineering: A survey of current, challenges and future. arXiv:2...

    arXiv 2022

  3. [2024]

    arXiv:2404.16130 [cs.CL] (2024)

    From local to global: A graph rag approach to query-focused summarization. arXiv:2404.16130 [cs.CL] (2024). Peter Emerson. 2013. The original Borda count and partial voting. Social Choice and Welfare 40, 2 (2013), 353–358. https://doi.org/10.1007/s00355-011-0603-9 Xiaodong Gu, Hongyu Zhang, and Sunghun Kim. 2018. Deep code search. In 2018 IEEE/ACM 40th In...

    arXiv 2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.