{"id":"890fce8f-662d-4b20-bd86-b4520b8a6388","arxiv_id":"2501.10542","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"BRaIn uses LLM relevance judgments to expand queries and re-rank search results, improving IR-based bug localization on the Bench4BL dataset.","lead":"This paper introduces BRaIn, a bug localization tool that uses a large language model to judge whether code files match a bug report, then rewrites the search query with terms from the files the model flags and re-ranks results. On 4,683 bug reports from the Bench4BL benchmark, it reports higher accuracy than several existing methods, though the largest gains are against a weak baseline and some headline numbers are based on a filtered subset.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's '52% of low-quality bugs' claim is computed on a filtered subset (520 of 1,101); on the full low-quality set the rate is ~24%, materially weakening the headline contribution.","rationale":"The reader's verdict of CONDITIONAL is appropriate. The reader's weakest_assumption focused on the hard filter in Eq. 2, which is a genuine robustness risk: if the LLM incorrectly marks the true buggy document as irrelevant, that document is permanently excluded. However, the empirical evaluation could still support the method's effectiveness even if that risk exists. My stress-test identifies a more immediate and quantifiable problem with the reported evidence: the '52% of low-quality bugs' claim uses an invalid denominator. The paper defines 1,101 low-quality reports, then excludes 581 because ES cannot retrieve the ground truth in the top-50. Since BRaIn only reranks ES's top-50 candidates, those 581 are also impossible for BRaIn. Thus, the true success rate is 24.3%, not 51.5%. This directly affects the abstract and the RQ1 summary, which are the most visible parts of the paper. The baseline comparison also overstates margins by highlighting improvements over the weakest baseline (DNNLOC), while the improvement over the best baseline (Blizzard) is modest (MAP 0.537 vs 0.506, ~6%). The paper does provide a significance test against Blizzard, which is good, but the abstract's '87.6%' margin is misleading. The prompt-optimization using 20 bug reports from the dataset without specifying train/test separation is another validity threat, but the authors acknowledge it and claim a limited re-test showed similar performance; without details, it remains a secondary concern. The central method—using LLM relevance feedback for query expansion and rescoring—is plausible and the replication package is a positive. However, the reported quantitative claims need correction before the paper can be accepted as-is. The recommended verdict remains CONDITIONAL, requiring the authors to recompute the low-quality success rate on the full set, report per-baseline comparisons with significance tests, and disclose the prompt-optimization split.","tokens_in":19352,"tokens_out":4504,"duration_ms":45689,"concrete_test":"Recompute the low-quality success rate as the number of low-quality bug reports (out of all 1,101) for which BRaIn (Mistral) has at least one ground-truth document in its top-10 results, divided by 1,101. If this rate is approximately 24% rather than 52%, the abstract and RQ1 summary must be corrected, and the claim that BRaIn localizes 'bugs that cannot be localized by baseline techniques' should be qualified to the subset where ES retrieves the buggy file within its top-50 results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline contribution includes the claim that BRaIn can localize approximately 52% of bugs that cannot be localized by baseline techniques due to poor bug report quality (Abstract, RQ1 Summary). In Section IV-D, the authors define 1,101 low-quality bug reports as those where Elasticsearch (ES) does not retrieve any ground-truth document within its top-10 results. They then exclude 581 reports where ES does not retrieve the ground truth even in the top-50 results, leaving 520 low-quality reports. BRaIn starts from ES's top-50 candidates, so for the 581 excluded reports, the true buggy document is not in BRaIn's candidate pool and cannot possibly be retrieved by BRaIn. Therefore, the correct denominator for the claim is the full set of 1,101 low-quality reports, and the success rate is 268/1,101 ≈ 24.3%, not 51.5% (268/520). Reporting 52% on the filtered subset overstates BRaIn's ability to handle low-quality reports by more than a factor of two. This is not a subtle methodological choice; it directly undermines the abstract's 'approximately 52%' claim and the RQ1 summary. The concern is concrete and quantitative: the denominator is wrong for the stated claim. Other issues, such as the hard filter in Eq. 2 and possible prompt-optimization leakage, are real but secondary; this evaluation misreport is the most load-bearing because it affects a headline result that is likely to be cited.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BRaIn, a bug localization technique that uses LLM-based binary relevance feedback to filter, expand, and rerank Elasticsearch results. The approach first retrieves a top-50 candidate pool with BM25, asks an LLM (Mistral, LLaMA, or Qwen) for a yes/no relevance judgment on each candidate, uses the 'yes' documents to extract expansion terms via PageRank, and finally rescores the candidates by combining softmax-normalized BM25 scores with the binary relevance feedback. The evaluation on 4,683 bug reports from Bench4BL reports MAP 0.537, MRR 0.571, and HIT@10 0.781 for BRaIn (Mistral), and compares against six baselines. The paper also claims that BRaIn can localize about 52% of low-quality bug reports that stymie the baseline.","tokens_in":19700,"tokens_out":6858,"duration_ms":62760,"significance":"The central idea of using LLM relevance judgments as a substitute for pseudo-relevance feedback is timely and practically motivated, and the paper ships a replication package, which strengthens reproducibility. The gains over the strongest baselines (e.g., Blizzard, RLocator) are real: 3.7–13.8% in the reported metrics. However, the headline claim about low-quality bug reports is computed on a filtered subset and overstates the success rate by more than a factor of two, and the hard binary filter in the scoring equation raises robustness concerns that are not analyzed. These issues materially affect the paper's stated significance and need to be fixed before publication.","major_comments":[{"comment":"The abstract and RQ1 summary state that BRaIn can localize approximately 52% of bugs that cannot be localized by the baseline due to poor bug report quality. However, this number is computed on a subset of 520 low-quality reports after excluding 581 of the 1,101 reports because their ground truth was not in the top-50 Elasticsearch results. Since BRaIn's candidate pool is the ES top-50, the excluded reports cannot possibly be retrieved by BRaIn. The correct success rate over the full low-quality set is 268/1,101 ≈ 24.3%, not 268/520 ≈ 51.5%. This is a load-bearing overstatement of a headline contribution, and the abstract, RQ1 summary, and Section IV-D must be corrected.","section":"IV-D and RQ1 Summary / Abstract"},{"comment":"The rescoring step multiplies the softmax-normalized BM25 score by a binary relevance factor r_i, where r_i=0 for any document the LLM marks 'no'. A single LLM false negative on the true buggy document therefore gives that document a final score of zero and removes it from the top-K ranking. The paper does not report the LLM's false-negative rate or the sensitivity of the results to this hard filtering. An ablation using a soft weighting (e.g., r_i in [0,1] or a small non-zero penalty for 'no') and an analysis of failure cases is needed to establish that the method is robust to LLM misjudgments.","section":"III-D, Eq. (2)"},{"comment":"The prompt template was optimized with SAMMO using 20 bug reports from the same Bench4BL dataset used for evaluation. The paper acknowledges this in the threats to validity but does not state whether these 20 reports come from the training or test split, nor does it describe the 'limited experiment' that allegedly showed similar performance. This makes it impossible to assess the degree of leakage. The authors should confirm that the 20 optimization reports are outside the test set, or re-run the evaluation with a prompt optimized on a separate set, and report the details of the supplementary experiment.","section":"VI and III-B"},{"comment":"The abstract's claim that BRaIn outperforms baselines by 87.6% in MAP, 89.5% in MRR, and 48.8% in HIT@10 is based on the comparison with DNNLOC, which in Table VII-b is the weakest baseline. The improvements over the strongest baselines are much smaller: 6.1% MAP over Blizzard, 8.8% over RLocator, and 1.0–5.5% over Rocchio/SCP depending on the metric. The abstract or the RQ3 summary should either report the range of improvements across all baselines or explicitly identify the baseline for each margin, so that readers are not misled into thinking the near-doubling is typical.","section":"Abstract and Section VII"}],"minor_comments":[{"comment":"Table IV reports performance on the full dataset, while Table VII reports comparison on the test set only (as stated in Section IV-D for RQ3). The table captions should explicitly state the dataset split used to avoid confusion.","section":"Tables IV and VII"},{"comment":"The text reporting improvements of 7.0–10.6% in MAP, 7.9–10.6% in MRR, and 6.0–6.9% in HIT@10 should specify that these percentages are relative to the Elasticsearch baseline, since the reader might otherwise interpret them as absolute differences.","section":"Section IV-D"},{"comment":"The paper uses both 'Sysman-SCP' and 'Sisman-SCP' for the same technique; please choose one consistent spelling (the reference is to Sisman et al., so 'Sisman-SCP' is preferable).","section":"Throughout"},{"comment":"The row label '4≥' appears to be a typographical inversion; it should read '≥4'.","section":"Table V"},{"comment":"The sentence 'For 122 bug reports, the large difference of 432 indicates that BRaIn identified the first buggy documents more often than Blizzard' is unclear; please clarify what the 432 represents (e.g., total rank-point difference or sum of rank improvements).","section":"Figure 4"},{"comment":"Reference [37] is incomplete: it lacks author names and a full title. Please complete the bibliographic entry.","section":"Reference [37]"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid empirical foundation and the core method is plausible and reproducible. The main load-bearing problem is the overstatement of the low-quality-report success rate; with the corrected denominator the contribution is still positive but considerably less dramatic. The hard-filter robustness and prompt-optimization transparency issues should also be addressed before the paper is publishable. I would not reject the paper, but I think the requested changes are substantial enough to require a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"BRaIn is worth a look if you care about IR-based bug localization. The new bit is using an LLM's binary relevance judgment as feedback to expand the query and rescore documents, rather than pseudo-relevance feedback or human feedback. The method is clearly described, the ablation in RQ2 shows both components matter, and they ship a replication package. That's real work.\n\nThe paper's central claim—LLM relevance feedback improves bug localization over six baselines—probably holds up. BRaIn (Mistral) beats the strongest traditional baseline (Blizzard) by about 6% MAP and the ML baselines by larger margins. The statistical test against Blizzard is a nice touch.\n\nThe soft spot is the headline \"≈52% of low-quality bugs\" claim. It's computed on 520 of 1,101 low-quality reports, excluding 581 where Elasticsearch didn't retrieve ground truth in the top 50. Since BRaIn starts from ES's top-50 candidates, those 581 can never be retrieved. The correct rate on the full low-quality set is roughly 268/1,101 ≈ 24%. That's still better than baseline (0%), but it's a factor-of-two overstatement and it's in the abstract. The authors should either report the per-set rate or clearly state the conditioning on retrievability.\n\nOther issues are secondary. The hard filter in Eq. 2 zeroes out any document the LLM marks \"no,\" so a false negative is unrecoverable; that's worth sensitivity analysis. Prompt optimization used 20 bug reports from the same benchmark—the paper discloses this in threats to validity and claims a limited experiment showed similar performance, but the potential leakage is real and should be addressed by optimizing on a separate validation set. Also, the 87.6% margin in the abstract is against DNNLOC, the weakest baseline; against stronger baselines the gains are 3.7–13.8%, which is respectable but not dramatic.\n\nVerdict: conditional accept territory. The method is sound and the evaluation is mostly careful, but the low-quality claim needs correction and the prompt-leakage question needs a cleaner answer. It deserves a serious referee.\n\nI'd bring it to reading group and would cite it if I were working on bug localization.\n\nRecommendation: send to peer review with a request for a revised analysis of the low-quality subset and explicit per-baseline significance tests.","headline":"A solid LLM-feedback bug localization method whose headline low-quality bug claim overstates success by a factor of two due to a filtered denominator.","tokens_in":20182,"tokens_out":1814,"would_cite":true,"duration_ms":16754,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Using a large language model's binary relevance judgments to expand queries and gate document scores improves IR-based bug localization, with the top variant reaching a mean average precision of 0.537 on a 4,683-report benchmark.","keywords":["bug localization","intelligent relevance feedback","large language models","query expansion","information retrieval","BM25","software maintenance","relevance feedback"],"falsifier":"Take the 520 low-quality bug reports where the baseline fails to retrieve the buggy file in the top 50, run the LLM relevance judge on the candidate lists, and compare its labels to the ground-truth buggy files: if the LLM marks the true buggy file 'no' in a substantial fraction of cases, the hard filter in Eq. 2 is the reason those bugs are missed. Alternatively, rerun BRaIn with LLM-negative documents downweighted instead of zeroed; if MAP does not drop, the hard filter is not necessary for the reported gain.","tokens_in":19145,"feed_emoji":"🐞","tokens_out":7721,"duration_ms":66730,"temperature":0.7,"pith_summary":"The paper argues that a large language model's 'yes/no' judgment about whether a code segment caused a reported bug—called Intelligent Relevance Feedback—can close the contextual gap that ordinary text matching leaves open. BRaIn takes the top BM25 hits for a bug report, has an LLM judge each one, expands the original query with terms from the documents the LLM marked relevant, and rescales BM25 scores so that documents marked irrelevant receive zero. On 4,683 bug reports from 42 Java systems, the best variant (Mistral) achieves a mean average precision (MAP) of 0.537, a mean reciprocal rank (MRR) of 0.571, and a HIT@10 of 0.781, ahead of existing baselines. If the claim holds, the practical payoff is that a developer-facing bug-localization tool can be built by prompting an off-the-shelf LLM rather than training a bespoke model or hand-crafting query reformulation rules.","feed_headline":"LLM yes/no feedback lifts bug localization past older methods","feed_subtitle":"Using an LLM to judge relevance and expand queries puts the buggy file in the top 10 for 78% of bug reports.","key_machinery":"The load-bearing mechanism is the Intelligent Relevance Feedback (IRF) signal, produced by prompting a zero-shot LLM to answer 'yes' or 'no' to whether a code segment is responsible for the reported bug, with the prompt tuned by a compile-time prompt optimizer on twenty ground-truth reports. Documents marked 'yes' are parsed into class, method, and field signatures; camel-case identifiers are split into phrases; and a term co-occurrence graph is ranked by PageRank to select roughly ten expansion terms. The expanded query is scored by BM25, the scores are softmax-normalized, and each score is multiplied by the binary relevance flag $r_i$ (Eq. 2), so LLM-negative documents get zero. The paper's component ablation shows that query expansion and the relevance-gated rescoring each improve over the BM25 baseline separately, but the best performance requires both.","core_discovery":"The central claim is that binary relevance feedback from an LLM, used both to expand the search query and to rescore retrieval results, improves IR-based bug localization beyond what textual or semantic matching achieves. In the scoring step the feedback acts as a hard gate: the softmax-normalized BM25 score of each document is multiplied by the LLM's relevance flag $r_i$ in Eq. 2, so any document judged 'no' is excluded from the final ranking. On Bench4BL, BRaIN with Mistral reaches MAP 0.537, MRR 0.571, and HIT@10 0.781, beating plain BM25 retrieval (0.484/0.513/0.732), Blizzard (0.506/0.536/0.758), Rocchio (0.489/0.558/0.765), and the ML-based RLocator (0.488/0.561/0.735) and DNNLOC (0.283/0.296/0.518). The paper attributes the gain to the LLM's contextual understanding: it can connect a report symptom like 'BASIC HTTP authentication' to code that handles a different but related authentication type, which surface-level term overlap misses.","pith_inferences":["A cheaper and more direct evaluation of the method would measure LLM yes/no agreement with ground-truth buggy files on the candidate set, rather than only the end-to-end rank; this would separate the quality of the feedback from the quality of the retrieval pipeline.","Because the pipeline only sees the top-50 BM25 hits, any bug whose true file is absent from that candidate pool is invisible regardless of how well the LLM judges relevance; using an LLM as a first-stage retriever over a larger candidate set is a natural untested extension.","The same binary-feedback-plus-query-expansion recipe could transfer to other software-engineering retrieval tasks with a vocabulary gap between natural-language queries and code, such as issue-to-commit linking or API recommendation.","The prompt was optimized on 20 reports drawn from the same dataset used for evaluation; the paper reports a limited repeat experiment with similar performance, but how much of the gain depends on the exact prompt wording across projects remains open."],"forward_implications":["IR-based bug localization can be improved without training a task-specific model or hand-tuning query-reformulation rules; a prompted off-the-shelf LLM supplies the feedback.","Low-quality bug reports that fail to retrieve the buggy file in the top 50 textually can still be localized: BRaIn with Mistral placed 268 of 520 such reports (51.5%) in the top 10.","Query expansion and rescoring are synergistic: removing either component lowers MAP by 3.0–7.8% relative to the full pipeline, though each alone still beats the plain BM25 baseline.","Because documents marked 'no' receive a final score of zero, the method's top-K output is an upper bound on the LLM's relevance-judgment accuracy; a mistaken 'no' on the true buggy file cannot be recovered by ranking."],"supporting_citations":[{"why":"Supplies the Bench4BL dataset of 4,683 bug reports and the reproducibility baseline used in the evaluation.","marker":"[21]"},{"why":"Provides the signature-extraction and PageRank-based term selection method adapted for query expansion.","marker":"[64]"},{"why":"Supplies the Blizzard baseline and the context-aware query-reformulation approach this work builds on and compares against.","marker":"[10]"},{"why":"Defines Okapi BM25, the retrieval and reranking algorithm at the core of the pipeline.","marker":"[40]"},{"why":"Identifies the Mistral model that delivers the best relevance feedback in the experiments.","marker":"[34]"},{"why":"Provides the LLaMA-3 model used both in prompt optimization and as one of the three feedback LLMs.","marker":"[57]"},{"why":"Supplies the compile-time prompt optimizer used to derive the optimized relevance-judgment prompt.","marker":"[56]"},{"why":"Defines the PageRank algorithm used to rank terms in the co-occurrence graph for query expansion.","marker":"[66]"}],"fun_headline_variants":["LLM relevance feedback finds bugs that text matching misses","Binary LLM feedback re-ranks code to locate bugs faster","BRaIn uses LLM feedback to hit top 10 for 78% of bugs","Yes/no LLM feedback improves bug pinpointing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the LLM's yes/no judgment on each candidate file is reliable enough to be used as a hard filter: any file the LLM marks 'no' receives a final score of zero in Eq. 2 and is dropped from the top-K list, so one wrong 'no' on the true buggy file makes that bug unlocalizable.","fun_headline_variants_meta":{"raw":{"variants":["LLM relevance feedback finds bugs that text matching misses","Binary LLM feedback re-ranks code to locate bugs faster","BRaIn uses LLM feedback to hit top 10 for 78% of bugs","Yes/no LLM feedback improves bug pinpointing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000305,"raw_usage":{"total_tokens":1804,"prompt_tokens":1055,"completion_tokens":749,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":671,"completion_tokens_details":{"reasoning_tokens":676}},"tokens_in":671,"tokens_out":749,"duration_ms":8230,"temperature":1.0,"reasoning_tokens":676,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:08:24.066838+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 520 low-quality bug reports where the baseline fails to retrieve the buggy file in the top 50, run the LLM relevance judge on the candidate lists, and compare its labels to the ground-truth buggy files: if the LLM marks the true buggy file 'no' in a substantial fraction of cases, the hard filter in Eq. 2 is the reason those bugs are missed. Alternatively, rerun BRaIn with LLM-negative documents downweighted instead of zeroed; if MAP does not drop, the hard filter is not necessary for the reported gain.","supporting_citations":[{"cited_title":"Bench4bl: reproducibility study on the performance of ir-based bug localization,","cited_arxiv_id":null,"evidence_quote":"Supplies the Bench4BL dataset of 4,683 bug reports and the reproducibility baseline used in the evaluation."},{"cited_title":"Improved query reformulation for concept location using coderank and document structures,","cited_arxiv_id":null,"evidence_quote":"Provides the signature-extraction and PageRank-based term selection method adapted for query expansion."},{"cited_title":"Mistral: A new approach to language models,","cited_arxiv_id":null,"evidence_quote":"Identifies the Mistral model that delivers the best relevance feedback in the experiments."}],"review_version":1}