Pith. sign in

REVIEW 4 major objections 6 minor 91 references

Improved IR-based Bug Localization with Intelligent Relevance Feedback

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Using a large language model's binary relevance judgments to expand queries and gate document scores improves IR-based bug localization, with the top variant reaching a mean average precision of 0.537 on a 4,683-report benchmark.

desk verdict A solid LLM-feedback bug localization method whose headline low-quality bug claim overstates success by a factor of two due to a filtered denominator. read the letter →

arxiv 2501.10542 v2 pith:HQ4566J5 submitted 2025-01-17 cs.SE cs.AIcs.CL

Asif Mohammed Samir , Mohammad Masudur Rahman This is my paper · ORCID
classification cs.SEcs.AIcs.CL
keywords buglocalizationintelligentrelevancefeedbacklargelanguagemodelsqueryexpansioninformationretrievalBM25softwaremaintenance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a large language model's 'yes/no' judgment about whether a code segment caused a reported bug—called Intelligent Relevance Feedback—can close the contextual gap that ordinary text matching leaves open. BRaIn takes the top BM25 hits for a bug report, has an LLM judge each one, expands the original query with terms from the documents the LLM marked relevant, and rescales BM25 scores so that documents marked irrelevant receive zero. On 4,683 bug reports from 42 Java systems, the best variant (Mistral) achieves a mean average precision (MAP) of 0.537, a mean reciprocal rank (MRR) of 0.571, and a HIT@10 of 0.781, ahead of existing baselines. If the claim holds, the practical payoff is that a developer-facing bug-localization tool can be built by prompting an off-the-shelf LLM rather than training a bespoke model or hand-crafting query reformulation rules.

What carries the argument

The load-bearing mechanism is the Intelligent Relevance Feedback (IRF) signal, produced by prompting a zero-shot LLM to answer 'yes' or 'no' to whether a code segment is responsible for the reported bug, with the prompt tuned by a compile-time prompt optimizer on twenty ground-truth reports. Documents marked 'yes' are parsed into class, method, and field signatures; camel-case identifiers are split into phrases; and a term co-occurrence graph is ranked by PageRank to select roughly ten expansion terms. The expanded query is scored by BM25, the scores are softmax-normalized, and each score is multiplied by the binary relevance flag $r_i$ (Eq. 2), so LLM-negative documents get zero. The paper's component ablation shows that query expansion and the relevance-gated rescoring each improve over the BM25 baseline separately, but the best performance requires both.

What would settle it

Take the 520 low-quality bug reports where the baseline fails to retrieve the buggy file in the top 50, run the LLM relevance judge on the candidate lists, and compare its labels to the ground-truth buggy files: if the LLM marks the true buggy file 'no' in a substantial fraction of cases, the hard filter in Eq. 2 is the reason those bugs are missed. Alternatively, rerun BRaIn with LLM-negative documents downweighted instead of zeroed; if MAP does not drop, the hard filter is not necessary for the reported gain.

Watch

Extended reading notes

Core claim

The central claim is that binary relevance feedback from an LLM, used both to expand the search query and to rescore retrieval results, improves IR-based bug localization beyond what textual or semantic matching achieves. In the scoring step the feedback acts as a hard gate: the softmax-normalized BM25 score of each document is multiplied by the LLM's relevance flag $r_i$ in Eq. 2, so any document judged 'no' is excluded from the final ranking. On Bench4BL, BRaIN with Mistral reaches MAP 0.537, MRR 0.571, and HIT@10 0.781, beating plain BM25 retrieval (0.484/0.513/0.732), Blizzard (0.506/0.536/0.758), Rocchio (0.489/0.558/0.765), and the ML-based RLocator (0.488/0.561/0.735) and DNNLOC (0.283/0.296/0.518). The paper attributes the gain to the LLM's contextual understanding: it can connect a report symptom like 'BASIC HTTP authentication' to code that handles a different but related authentication type, which surface-level term overlap misses.

Load-bearing premise

The method assumes the LLM's yes/no judgment on each candidate file is reliable enough to be used as a hard filter: any file the LLM marks 'no' receives a final score of zero in Eq. 2 and is dropped from the top-K list, so one wrong 'no' on the true buggy file makes that bug unlocalizable.

Editorial extensions

If this is right

  • IR-based bug localization can be improved without training a task-specific model or hand-tuning query-reformulation rules; a prompted off-the-shelf LLM supplies the feedback.
  • Low-quality bug reports that fail to retrieve the buggy file in the top 50 textually can still be localized: BRaIn with Mistral placed 268 of 520 such reports (51.5%) in the top 10.
  • Query expansion and rescoring are synergistic: removing either component lowers MAP by 3.0–7.8% relative to the full pipeline, though each alone still beats the plain BM25 baseline.
  • Because documents marked 'no' receive a final score of zero, the method's top-K output is an upper bound on the LLM's relevance-judgment accuracy; a mistaken 'no' on the true buggy file cannot be recovered by ranking.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A cheaper and more direct evaluation of the method would measure LLM yes/no agreement with ground-truth buggy files on the candidate set, rather than only the end-to-end rank; this would separate the quality of the feedback from the quality of the retrieval pipeline.
  • Because the pipeline only sees the top-50 BM25 hits, any bug whose true file is absent from that candidate pool is invisible regardless of how well the LLM judges relevance; using an LLM as a first-stage retriever over a larger candidate set is a natural untested extension.
  • The same binary-feedback-plus-query-expansion recipe could transfer to other software-engineering retrieval tasks with a vocabulary gap between natural-language queries and code, such as issue-to-commit linking or API recommendation.
  • The prompt was optimized on 20 reports drawn from the same dataset used for evaluation; the paper reports a limited repeat experiment with similar performance, but how much of the gain depends on the exact prompt wording across projects remains open.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes BRaIn, a bug localization technique that uses LLM-based binary relevance feedback to filter, expand, and rerank Elasticsearch results. The approach first retrieves a top-50 candidate pool with BM25, asks an LLM (Mistral, LLaMA, or Qwen) for a yes/no relevance judgment on each candidate, uses the 'yes' documents to extract expansion terms via PageRank, and finally rescores the candidates by combining softmax-normalized BM25 scores with the binary relevance feedback. The evaluation on 4,683 bug reports from Bench4BL reports MAP 0.537, MRR 0.571, and HIT@10 0.781 for BRaIn (Mistral), and compares against six baselines. The paper also claims that BRaIn can localize about 52% of low-quality bug reports that stymie the baseline.

Significance. The central idea of using LLM relevance judgments as a substitute for pseudo-relevance feedback is timely and practically motivated, and the paper ships a replication package, which strengthens reproducibility. The gains over the strongest baselines (e.g., Blizzard, RLocator) are real: 3.7–13.8% in the reported metrics. However, the headline claim about low-quality bug reports is computed on a filtered subset and overstates the success rate by more than a factor of two, and the hard binary filter in the scoring equation raises robustness concerns that are not analyzed. These issues materially affect the paper's stated significance and need to be fixed before publication.

major comments (4)
  1. [IV-D and RQ1 Summary / Abstract] The abstract and RQ1 summary state that BRaIn can localize approximately 52% of bugs that cannot be localized by the baseline due to poor bug report quality. However, this number is computed on a subset of 520 low-quality reports after excluding 581 of the 1,101 reports because their ground truth was not in the top-50 Elasticsearch results. Since BRaIn's candidate pool is the ES top-50, the excluded reports cannot possibly be retrieved by BRaIn. The correct success rate over the full low-quality set is 268/1,101 ≈ 24.3%, not 268/520 ≈ 51.5%. This is a load-bearing overstatement of a headline contribution, and the abstract, RQ1 summary, and Section IV-D must be corrected.
  2. [III-D, Eq. (2)] The rescoring step multiplies the softmax-normalized BM25 score by a binary relevance factor r_i, where r_i=0 for any document the LLM marks 'no'. A single LLM false negative on the true buggy document therefore gives that document a final score of zero and removes it from the top-K ranking. The paper does not report the LLM's false-negative rate or the sensitivity of the results to this hard filtering. An ablation using a soft weighting (e.g., r_i in [0,1] or a small non-zero penalty for 'no') and an analysis of failure cases is needed to establish that the method is robust to LLM misjudgments.
  3. [VI and III-B] The prompt template was optimized with SAMMO using 20 bug reports from the same Bench4BL dataset used for evaluation. The paper acknowledges this in the threats to validity but does not state whether these 20 reports come from the training or test split, nor does it describe the 'limited experiment' that allegedly showed similar performance. This makes it impossible to assess the degree of leakage. The authors should confirm that the 20 optimization reports are outside the test set, or re-run the evaluation with a prompt optimized on a separate set, and report the details of the supplementary experiment.
  4. [Abstract and Section VII] The abstract's claim that BRaIn outperforms baselines by 87.6% in MAP, 89.5% in MRR, and 48.8% in HIT@10 is based on the comparison with DNNLOC, which in Table VII-b is the weakest baseline. The improvements over the strongest baselines are much smaller: 6.1% MAP over Blizzard, 8.8% over RLocator, and 1.0–5.5% over Rocchio/SCP depending on the metric. The abstract or the RQ3 summary should either report the range of improvements across all baselines or explicitly identify the baseline for each margin, so that readers are not misled into thinking the near-doubling is typical.
minor comments (6)
  1. [Tables IV and VII] Table IV reports performance on the full dataset, while Table VII reports comparison on the test set only (as stated in Section IV-D for RQ3). The table captions should explicitly state the dataset split used to avoid confusion.
  2. [Section IV-D] The text reporting improvements of 7.0–10.6% in MAP, 7.9–10.6% in MRR, and 6.0–6.9% in HIT@10 should specify that these percentages are relative to the Elasticsearch baseline, since the reader might otherwise interpret them as absolute differences.
  3. [Throughout] The paper uses both 'Sysman-SCP' and 'Sisman-SCP' for the same technique; please choose one consistent spelling (the reference is to Sisman et al., so 'Sisman-SCP' is preferable).
  4. [Table V] The row label '4≥' appears to be a typographical inversion; it should read '≥4'.
  5. [Figure 4] The sentence 'For 122 bug reports, the large difference of 432 indicates that BRaIn identified the first buggy documents more often than Blizzard' is unclear; please clarify what the 432 represents (e.g., total rank-point difference or sum of rank improvements).
  6. [Reference [37]] Reference [37] is incomplete: it lacks author names and a full title. Please complete the bibliographic entry.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: BRaIn's results are measured on an external benchmark; the prompt-tuning and low-quality-denominator issues are validity concerns, not definitional reductions.

full rationale

BRaIn's central claim — that LLM relevance feedback improves IR-based bug localization, with MAP 0.537, MRR 0.571, and HIT@10 0.781 on Bench4BL — is an empirical result rather than a derivation from its own inputs. The pipeline is ES/BM25 retrieval (Section III-A), LLM yes/no relevance judgments (Section III-B), PageRank-based term expansion (Section III-C), and Eq. 2 rescoring (softmax BM25 score multiplied by the binary relevance flag). None of these equations define the evaluation metric in terms of itself; the LLM judgments and BM25 scores are inputs, and the reported metrics are measured across 4,683 bug reports from an external benchmark with six baseline comparisons. The paper's self-citations ([64] for term-graph/PageRank query expansion and [11] for the low-quality-report definition) describe published, independently checkable techniques, not uniqueness theorems, and the method does not depend on accepting an unverified self-citation as its core premise. Section VI discloses: 'we used 20 bug reports from our dataset to optimize prompts with LLaMA. Since they are part of our experimental dataset, it could introduce bias.' I flag this as a real evaluation-independence threat — the paper does not demonstrate that the 20 tuning reports are disjoint from the test split — but it does not make the MAP value a forced consequence of the fitted prompt by construction. Section IV-D's low-quality analysis is a separate reporting problem: of 1,101 low-quality reports, 581 'do not contain any ground truth within their top 50 results' and are excluded before computing 268/520 ≈ 51.5%, so the abstract's '≈52%' overstates the full low-quality success rate (≈24.3%). That is a denominator/reporting error, not a circular reduction. Overall, no load-bearing step reduces a prediction to its own input or to a self-citation chain.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard IR components (BM25, PageRank) and a domain assumption that LLM relevance feedback is reliable. The main free parameters are retrieval/expansion sizes and a prompt tuned on the benchmark itself.

free parameters (4)
  • top-K candidate documents = 50
    Number of documents retrieved from Elasticsearch per bug report; chosen ad hoc, affects recall of the LLM filtering stage.
  • top-N expansion terms = 10
    Number of PageRank-selected terms added to the query, following Rahman et al. [64]; affects query expansion.
  • PageRank damping factor = 0.85
    Standard PageRank parameter, used in Eq. 1.
  • Prompt template = optimized via SAMMO on 20 bug reports
    The prompt used for all LLM relevance judgments was selected by optimizing on a small sample from the same benchmark dataset, a form of tuning on the evaluation data.
assumptions (4)
  • domain assumption BM25 retrieval with Elasticsearch default analyzer ranks candidate documents reasonably
    Used as initial retrieval stage; assumes top-50 contains the buggy document often enough.
  • domain assumption LLM zero-shot relevance judgments correlate with actual bugginess
    Central assumption of IRF; if LLM judgments are noisy, query expansion and rescoring may be harmed.
  • domain assumption Buggy documents are exactly those changed in the fix commits (Bench4BL ground truth)
    Evaluation relies on the benchmark's ground truth linking bug reports to changed files.
  • ad hoc to paper Exclusion of bug reports without version info, stack-trace-only reports, and low-quality reports with no candidate in top 50 does not bias the evaluation
    Data curation decisions in Section IV-A affect the dataset composition and the low-quality analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improved IR-based Bug Localization with Intelligent Relevance Feedback." pith.science (2026). https://pith.science/paper/HQ4566J5

@misc{pith2026250110542,
  author       = {Pith},
  title        = {Pith review of: Improved IR-based Bug Localization with Intelligent Relevance Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HQ4566J5}},
  note         = {Machine review of arXiv:2501.10542}
}
read the original abstract

Software bugs pose a significant challenge during development and maintenance, and practitioners spend nearly 50% of their time dealing with bugs. Many existing techniques adopt Information Retrieval (IR) to localize a reported bug using textual and semantic relevance between bug reports and source code. However, they often struggle to bridge a critical gap between bug reports and code that requires in-depth contextual understanding, which goes beyond textual or semantic relevance. In this paper, we present a novel technique for bug localization - BRaIn - that addresses the contextual gaps by assessing the relevance between bug reports and code with Large Language Models (LLM). It then leverages the LLM's feedback (a.k.a., Intelligent Relevance Feedback) to reformulate queries and re-rank source documents, improving bug localization. We evaluate BRaIn using a benchmark dataset, Bench4BL, and three performance metrics and compare it against six baseline techniques from the literature. Our experimental results show that BRaIn outperforms baselines by 87.6%, 89.5%, and 48.8% margins in MAP, MRR, and HIT@K, respectively. Additionally, it can localize approximately 52% of bugs that cannot be localized by the baseline techniques due to the poor quality of corresponding bug reports. By addressing the contextual gaps and introducing Intelligent Relevance Feedback, BRaIn advances not only theory but also improves IR-based bug localization.

Figures

Figures reproduced from arXiv: 2501.10542 by the authors.

Figure 1
Figure 1. Buggy Code with Diff The above evidence suggests that an in-depth analysis involving contextual relevance is essential. A seasoned devel￾oper would recognize the missing clause of BASIC HTTP authentication in the code, although it is not explicitly stated in the bug report. By probing deeper, they would infer that the missing BASIC authentication is likely the root cause of the reported issue. Large language models … view at source ↗
Figure 2
Figure 2. Schematic Diagram of BRaIn: (A) Document Indexing & Retrieval, (B) Intelligent Relevance Feedback, (C) Query Expansion, and (D) Bug Localization segments (ground truth) to guide our optimization process. SAMMO iteratively generated prompt variants by applying various modification operations to the candidate prompt with LLaMA’s assistance. In each iteration, we used LLaMA, bug reports and ground truth code to determi… view at source ↗
Figure 3
Figure 3. Performance of BRaIn with Low Quality Bug Reports [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Rank Improvement: BRaIn vs Blizzard TABLE VIII: Statistical Test: BRaIn vs. Blizzard Evaluation Point p-value Effect Size (Cliff’s δ) Top-1 0.0023 ** Medium (0.41) Top-5 0.0015 ** Large (0.66) Top-10 0.0008 *** Large (0.82) *=statistical significance Finding the first …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

91 extracted references · 57 canonical work pages

  1. [1]

    Cpsq 2020 report,

    C. for IT Software Quality, “Cpsq 2020 report,” IT-CISQ, Tech. Rep., 2020. [Online]. Available: https://www.it-cisq.org/cisq-files/pdf/ CPSQ-2020-report.pdf

    2020

  2. [2]

    F. T. Council. (2023) Costly code: The price of software errors. [Online]. Available: https://shorturl.at/2j7wu

    2023

  3. [3]

    Helping our customers through the crowdstrike outage - the official microsoft blog,

    D. Weston, “Helping our customers through the crowdstrike outage - the official microsoft blog,” Jul. 2024. [Online]. Available: https://shorturl.at/3OCxV

    2024

  4. [4]

    Here comes the wave of insurance claims for the crowdstrike outage,

    L. K. Wee, “Here comes the wave of insurance claims for the crowdstrike outage,” Jul. 2024. [Online]. Available: https://shorturl.at/5Y1jQ

    2024

  5. [5]

    How practitioners perceive automated bug report management techniques,

    W. Zou, D. Lo, Z. Chen, X. Xia, Y . Feng, and B. Xu, “How practitioners perceive automated bug report management techniques,” IEEE Transac- tions on Software Engineering , vol. 46, no. 8, pp. 836–862, 2018

    2018

  6. [6]

    The debugging mindset: Understanding the psychology of learning strategies leads to effective problem-solving skills

    D. H. O’Dell, “The debugging mindset: Understanding the psychology of learning strategies leads to effective problem-solving skills.” Queue, vol. 15, no. 1, pp. 71–90, 2017

    2017

  7. [7]

    (2024) Survey: Fixing bugs stealing time from development

    DevOps. (2024) Survey: Fixing bugs stealing time from development. [Online]. Available: https://shorturl.at/Fj8sB

    2024

  8. [8]

    Reversible debugging software,

    T. Britton, L. Jeng, G. Carver, P. Cheak, and T. Katzenellenbogen, “Reversible debugging software,” Judge Bus. School, Univ. Cambridge, Cambridge, UK, Tech. Rep , vol. 229, 2013

    2013

Show all 91 references
  1. [9]

    The vocabulary problem in human-system communication,

    G. W. Furnas, T. K. Landauer, L. M. Gomez, and S. T. Dumais, “The vocabulary problem in human-system communication,” Communications of the ACM , vol. 30, no. 11, pp. 964–971, 1987

    1987

  2. [10]

    Improving ir-based bug localization with context-aware query reformulation,

    M. M. Rahman and C. K. Roy, “Improving ir-based bug localization with context-aware query reformulation,” in Proceedings of the 2018 26th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering , 2018, pp. 621–632

    2018

  3. [11]

    The forgotten role of search queries in ir-based bug localization: an empirical study,

    M. M. Rahman, F. Khomh, S. Yeasmin, and C. K. Roy, “The forgotten role of search queries in ir-based bug localization: an empirical study,” Empirical Software Engineering , vol. 26, no. 6, p. 116, 2021

    2021

  4. [12]

    On the use of stack traces to improve text retrieval-based bug localization,

    L. Moreno, J. J. Treadway, A. Marcus, and W. Shen, “On the use of stack traces to improve text retrieval-based bug localization,” in 2014 IEEE International Conference on Software Maintenance and Evolution, 2014, pp. 151–160

    2014

  5. [13]

    Evaluating the usefulness of ir-based fault localization techniques,

    Q. Wang, C. Parnin, and A. Orso, “Evaluating the usefulness of ir-based fault localization techniques,” in Proceedings of the 2015 International Symposium on Software Testing and Analysis , ser. ISSTA 2015. New York, NY , USA: Association for Computing Machinery, 2015, p. 1–11...

    2015

  6. [14]

    Evaluating the usefulness of ir-based fault localization tech- niques,

    ——, “Evaluating the usefulness of ir-based fault localization tech- niques,” in Proceedings of the 2015 International Symposium on Soft- ware Testing and Analysis , 2015, pp. 1–11

    2015

  7. [15]

    A topic-based approach for narrowing the search space of buggy files from a bug report,

    A. T. Nguyen, T. T. Nguyen, J. Al-Kofahi, H. V . Nguyen, and T. N. Nguyen, “A topic-based approach for narrowing the search space of buggy files from a bug report,” in 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011) . IEEE, 2011, pp. 263–272

    2011

  8. [16]

    Information retrieval and spectrum based bug localization: Better together,

    T.-D. B. Le, R. J. Oentaryo, and D. Lo, “Information retrieval and spectrum based bug localization: Better together,” in Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, 2015, pp. 579–590

    2015

  9. [17]

    Bug localization with combination of deep learning and information re- trieval,

    A. N. Lam, A. T. Nguyen, H. A. Nguyen, and T. N. Nguyen, “Bug localization with combination of deep learning and information re- trieval,” in 2017 IEEE/ACM 25th International Conference on Program Comprehension (ICPC). IEEE, 2017, pp. 218–229

    2017

  10. [18]

    Deep reinforced query refor- mulation for information retrieval

    X. Wang, C. Macdonald, and I. Ounis, “Deep reinforced query refor- mulation for information retrieval.” arXiv preprint arXiv:2007.07987 , 2020

    2007 arXiv

  11. [19]

    Version history, similar report, and structure: Putting them together for improved bug localization,

    S. Wang and D. Lo, “Version history, similar report, and structure: Putting them together for improved bug localization,” in Proceedings of the 22nd International Conference on Program Comprehension , 2014, pp. 53–63

    2014

  12. [20]

    Bug localization based on code change histories and bug reports,

    K. C. Youm, J. Ahn, J. Kim, and E. Lee, “Bug localization based on code change histories and bug reports,” in 2015 Asia-Pacific Software Engineering Conference (APSEC) , 2015, pp. 190–197

    2015

  13. [21]

    Bench4bl: reproducibility study on the performance of ir-based bug localization,

    J. Lee, D. Kim, T. F. Bissyand ´e, W. Jung, and Y . Le Traon, “Bench4bl: reproducibility study on the performance of ir-based bug localization,” in Proceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis , 2018, pp. 61–72

    2018

  14. [22]

    Using observed behavior to reformulate queries during text retrieval-based bug localization,

    O. Chaparro, J. M. Florez, and A. Marcus, “Using observed behavior to reformulate queries during text retrieval-based bug localization,” in 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2017, pp. 376–387

    2017

  15. [23]

    Incorporating version histories in information retrieval based bug localization,

    B. Sisman and A. C. Kak, “Incorporating version histories in information retrieval based bug localization,” in 2012 9th IEEE working conference on mining software repositories (MSR) . IEEE, 2012, pp. 50–59

    2012

  16. [24]

    Nprf: A neural pseudo relevance feedback framework for ad-hoc information retrieval,

    C. Li, Y . Sun, B. He, L. Wang, K. Hui, A. Yates, L. Sun, and J. Xu, “Nprf: A neural pseudo relevance feedback framework for ad-hoc information retrieval,” Jan. 2018. [Online]. Available: https://arxiv.org/abs/1810.12936

    2018 arXiv

  17. [25]

    Improving pseudo-relevance feedback in web information retrieval using web page segmentation,

    S. Yu, D. Cai, J.-R. Wen, and W.-Y . Ma, “Improving pseudo-relevance feedback in web information retrieval using web page segmentation,” in Proceedings of the 12th international conference on World Wide Web , 2003, pp. 11–18

    2003

  18. [26]

    A pseudo- relevance feedback framework combining relevance matching and se- mantic matching for information retrieval,

    J. Wang, M. Pan, T. He, X. Huang, X. Wang, and X. Tu, “A pseudo- relevance feedback framework combining relevance matching and se- mantic matching for information retrieval,” Information Processing & Management, vol. 57, no. 6, p. 102342, 2020

    2020

  19. [27]

    A novel approach to automatic query reformulation for ir-based bug localization,

    M. Kim and E. Lee, “A novel approach to automatic query reformulation for ir-based bug localization,” in Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing , ser. SAC ’19. New York, NY , USA: Association for Computing Machinery, 2019, p. 1752–1759. [Online]. Ava...

    2019

  20. [28]

    On the use of relevance feedback in ir-based concept location,

    G. Gay, S. Haiduc, A. Marcus, and T. Menzies, “On the use of relevance feedback in ir-based concept location,” in 2009 IEEE International Conference on Software Maintenance , 2009, pp. 351–360

    2009

  21. [29]

    Automatic query reformulations for text retrieval in soft- ware engineering,

    S. Haiduc, G. Bavota, A. Marcus, R. Oliveto, A. De Lucia, and T. Menzies, “Automatic query reformulations for text retrieval in soft- ware engineering,” in 2013 35th International Conference on Software Engineering (ICSE). IEEE, 2013, pp. 842–851

    2013

  22. [30]

    Assisting code search with automatic query reformulation for bug localization,

    B. Sisman and A. C. Kak, “Assisting code search with automatic query reformulation for bug localization,” in 2013 10th Working Conference on Mining Software Repositories (MSR) , 2013, pp. 309–318

    2013

  23. [31]

    A survey of machine learning for big code and naturalness,

    M. Allamanis, E. T. Barr, P. Devanbu, and C. Sutton, “A survey of machine learning for big code and naturalness,” ACM Computing Surveys (CSUR), vol. 51, no. 4, pp. 1–37, 2018

    2018

  24. [32]

    Concise and consistent naming,

    F. Deissenboeck and M. Pizka, “Concise and consistent naming,” Soft- ware Quality Journal , vol. 14, pp. 261–282, 2006

    2006

  25. [33]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    2017

  26. [34]

    Mistral: A new approach to language models,

    M. Team, “Mistral: A new approach to language models,” 2023. [Online]. Available: https://mistral.ai

    2023

  27. [35]

    Improving bug localization using structured information retrieval,

    R. K. Saha, M. Lease, S. Khurshid, and D. E. Perry, “Improving bug localization using structured information retrieval,” in 2013 28th IEEE/ACM International Conference on Automated Software Engineer- ing (ASE). IEEE, 2013, pp. 345–355

    2013

  28. [36]

    Bug localization with combination of deep learning and information re- trieval,

    A. N. Lam, A. T. Nguyen, H. A. Nguyen, and T. N. Nguyen, “Bug localization with combination of deep learning and information re- trieval,” in 2017 IEEE/ACM 25th International Conference on Program Comprehension (ICPC), 2017, pp. 218–229

    2017

  29. [37]

    Combining word embedding with information retrieval to recommend similar bug reports

    “Combining word embedding with information retrieval to recommend similar bug reports.” IEEE Computer Society, 12 2016, pp. 127–137

    2016

  30. [38]

    Rlocator: Reinforcement learning for bug localization,

    P. Chakraborty, M. Alfadel, and M. Nagappan, “Rlocator: Reinforcement learning for bug localization,” IEEE Transactions on Software Engineer- ing, 2024

    2024

  31. [39]

    BRaIn: Replication Package,

    M. M. R. Asif Samir, “BRaIn: Replication Package,” https://github.com/ asifsamir/BRaIn, 2024

    2024

  32. [40]

    Okapi at trec-3,

    S. E. Robertson, S. Walker, S. Jones, M. M. Hancock-Beaulieu, M. Gat- ford et al., “Okapi at trec-3,” Nist Special Publication Sp , vol. 109, p. 109, 1995

    1995

  33. [41]

    Elasticsearch,

    “Elasticsearch,” Elastic. [Online]. Available: https://www.elastic.co/ elasticsearch/

  34. [42]

    Llm-based code generation method for golang compiler testing,

    Q. Gu, “Llm-based code generation method for golang compiler testing,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2023, pp. 2201–2203

    2023

  35. [43]

    Llm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models,

    L. Lian, B. Li, A. Yala, and T. Darrell, “Llm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models,” arXiv preprint arXiv:2305.13655 , 2023

    2023 arXiv

  36. [44]

    Prompting large language models with speech recognition abilities,

    Y . Fathullah, C. Wu, E. Lakomkin, J. Jia, Y . Shangguan, K. Li, J. Guo, W. Xiong, J. Mahadeokar, O. Kalinli et al., “Prompting large language models with speech recognition abilities,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processin...

    2024

  37. [45]

    Do prompt-based models really understand the meaning of their prompts?

    A. Webson and E. Pavlick, “Do prompt-based models really understand the meaning of their prompts?” arXiv preprint arXiv:2109.01247, 2021

    2021 arXiv

  38. [46]

    Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing,

    P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, “Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing,” ACM Computing Surveys, vol. 55, no. 9, pp. 1–35, 2023

    2023

  39. [47]

    A brief history of prompt: Leveraging language models. (through advanced prompting),

    M. Muktadir, Golam, “A brief history of prompt: Leveraging language models. (through advanced prompting),” Sep. 2023. [Online]. Available: https://arxiv.org/abs/2310.04438

    2023 arXiv

  40. [48]

    Prompt engineering as an important emerging skill for medical professionals: tutorial,

    B. Mesk ´o, “Prompt engineering as an important emerging skill for medical professionals: tutorial,” Journal of medical Internet research , vol. 25, p. e50638, 2023

    2023

  41. [49]

    Unleashing the potential of prompt engineering in large language models: a comprehensive review,

    B. Chen, Z. Zhang, N. Langren ´e, and S. Zhu, “Unleashing the potential of prompt engineering in large language models: a comprehensive review,” arXiv preprint arXiv:2310.14735 , 2023

    2023 arXiv

  42. [50]

    Promptify: Text-to-image generation through interactive prompt exploration with large language models,

    S. Brade, B. Wang, M. Sousa, S. Oore, and T. Grossman, “Promptify: Text-to-image generation through interactive prompt exploration with large language models,” in Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, 2023, pp. 1–14

    2023

  43. [51]

    Prompting is all you need: Automated android bug replay with large language models,

    S. Feng and C. Chen, “Prompting is all you need: Automated android bug replay with large language models,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering , 2024, pp. 1–13

    2024

  44. [52]

    Prompt engineering or fine tuning: An empirical assessment of large language models in automated software engineering tasks,

    J. Shin, C. Tang, T. Mohati, M. Nayebi, S. Wang, and H. Hem- mati, “Prompt engineering or fine tuning: An empirical assessment of large language models in automated software engineering tasks,” arXiv preprint arXiv:2310.10508, 2023

    arXiv

  45. [53]

    The prompt report: A systematic survey of prompting techniques,

    S. Schulhoff, M. Ilie, N. Balepur, K. Kahadze, A. Liu, C. Si, Y . Li, A. Gupta, H. Han, S. Schulhoff et al., “The prompt report: A systematic survey of prompting techniques,” arXiv preprint arXiv:2406.06608 , 2024

    2024 arXiv

  46. [54]

    Prompting: how to guides

    “Prompting: how to guides.” [Online]. Available: https://www.llama. com/docs/how-to-guides/prompting/

  47. [55]

    Effective prompts for ai: The essentials,

    M. S. T. . L. Technologies, “Effective prompts for ai: The essentials,” September 16 2024. [Online]. Available: https://mitsloanedtech.mit.edu/ ai/basics/effective-prompts/

    2024

  48. [56]

    Prompts as programs: A structure-aware approach to efficient compile-time prompt optimization,

    T. Schnabel and J. Neville, “Prompts as programs: A structure-aware approach to efficient compile-time prompt optimization,” arXiv preprint arXiv:2404.02319, 2024

    2024 arXiv

  49. [57]

    Llama: Open and efficient foundation language models,

    H. Touvron, A. Bosselut, K. Sinha, and et al., “Llama: Open and efficient foundation language models,” 2023. [Online]. Available: https://arxiv.org/abs/2302.13971

    2023 arXiv

  50. [58]

    Improving neural language models by segmenting, attending, and predicting the future,

    H. Luo, L. Jiang, Y . Belinkov, and J. Glass, “Improving neural language models by segmenting, attending, and predicting the future,” arXiv preprint arXiv:1906.01702, 2019

    1906 arXiv

  51. [59]

    An improved algorithm for slicing machine code,

    V . Srinivasan and T. Reps, “An improved algorithm for slicing machine code,” ACM SIGPLAN Notices , vol. 51, no. 10, pp. 378–393, 2016

    2016

  52. [60]

    Javaparser,

    C. Fischer, “Javaparser,” 2019. [Online]. Available: https://github.com/ javaparser/javaparser

    2019

  53. [61]

    Qwen: A high-performance language model,

    Q. Team, “Qwen: A high-performance language model,” 2023. [Online]. Available: https://huggingface.co/Qwen

    2023

  54. [62]

    Huggingface’s transformers: State-of-the-art natural language processing,

    T. Wolf, L. Debut, V . Sanh, J. Chaumond, and Delangue, “Huggingface’s transformers: State-of-the-art natural language processing,” arXiv preprint arXiv:1910.03771 , 2019. [Online]. Available: https://huggingface.co/docs/transformers/index

    1910 arXiv

  55. [63]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , 2023

    2023

  56. [64]

    Improved query reformulation for concept location using coderank and document structures,

    M. M. Rahman and C. K. Roy, “Improved query reformulation for concept location using coderank and document structures,” in 2017 32nd IEEE/ACM International Conference on Automated Software Engineer- ing (ASE). IEEE, 2017, pp. 428–439

    2017

  57. [65]

    Javalang: Pure python java parser and tools

    “Javalang: Pure python java parser and tools.” [Online]. Available: https://github.com/c2nes/javalang

  58. [66]

    The anatomy of a large-scale hypertextual web search engine,

    S. Brin and L. Page, “The anatomy of a large-scale hypertextual web search engine,” Computer networks and ISDN systems , vol. 30, no. 1-7, pp. 107–117, 1998

    1998

  59. [67]

    The softmax function: Properties, motivation, and interpretation,

    M. Franke and J. Degen, “The softmax function: Properties, motivation, and interpretation,” 2023

    2023

  60. [68]

    Why 70/30 or 80/20 relation between training and testing sets: A pedagogical explanation,

    A. Gholamy, V . Kreinovich, and O. Kosheleva, “Why 70/30 or 80/20 relation between training and testing sets: A pedagogical explanation,” Int. J. Intell. Technol. Appl. Stat , vol. 11, no. 2, pp. 105–111, 2018

    2018

  61. [69]

    Open-llm-leaderboard: From multi-choice to open-style questions for large language models,

    F. Author and S. Author, “Open-llm-leaderboard: From multi-choice to open-style questions for large language models,” arXiv preprint arXiv:2406.07545, 2024

    2024 arXiv

  62. [70]

    Gptq: Accurate post-training quantization for generative pre-trained transformers,

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “Gptq: Accurate post-training quantization for generative pre-trained transformers,” arXiv preprint arXiv:2210.17323, 2022

    2022 arXiv

  63. [71]

    Apache lucene,

    T. A. S. Foundation, “Apache lucene,” 2021. [Online]. Available: https://lucene.apache.org/

    2021

  64. [72]

    Blizzard

    M. Rahman, “Blizzard.” [Online]. Available: https://github.com/ masud-technope/BLIZZARD

  65. [73]

    Lecture 7: Relevance feedback and query expansion,

    R. Cummins, N. Language, and I. P. N. Group, “Lecture 7: Relevance feedback and query expansion,” p. 271, 2017. [Online]. Available: https://www.cl.cam.ac.uk/teaching//1617/InfoRtrv/ lecture7-relevance-feedback.pdf?formCode=MG0A V3

    2017

  66. [74]

    Where should we fix this bug? a two-phase recommendation model,

    D. Kim, Y . Tao, S. Kim, and A. Zeller, “Where should we fix this bug? a two-phase recommendation model,” IEEE transactions on software Engineering, vol. 39, no. 11, pp. 1597–1610, 2013

    2013

  67. [75]

    Distributed representations of words and phrases and their compositionality,

    T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” 2013. [Online]. Available: https://arxiv.org/abs/1310.4546

    2013 arXiv

  68. [76]

    Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,

    Y . Wang, W. Wang, S. Joty, and S. C. Hoi, “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” arXiv preprint arXiv:2109.00859 , 2021

    2021 arXiv

  69. [77]

    Zenodo, Jul. 2024. [Online]. Available: https://doi.org/10.5281/zenodo. 11265302

    2024 doi

  70. [78]

    Barnes, S

    T. Barnes, S. C. Moore, and K. Osatuke, Testing Significance Tests: A Simulation with Cliff’s Delta, t-tests, and Mann-Whitney U . National Center for Organizational Development, Department of Veteran Affairs, 2018

    2018

  71. [79]

    Document ranking and the vector- space model,

    D. Lee, H. Chuang, and K. Seamons, “Document ranking and the vector- space model,” IEEE Software, vol. 14, no. 2, pp. 67–75, 1997

    1997

  72. [80]

    On the effectiveness of information retrieval based bug localization for c programs,

    R. K. Saha, J. Lawall, S. Khurshid, and D. E. Perry, “On the effectiveness of information retrieval based bug localization for c programs,” in 2014 IEEE international conference on software maintenance and evolution . IEEE, 2014, pp. 161–170

    2014

  73. [81]

    Locus: Locating bugs from software changes,

    M. Wen, R. Wu, and S.-C. Cheung, “Locus: Locating bugs from software changes,” in Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering , 2016, pp. 262–273

    2016

  74. [82]

    Indri : A language-model based search engine for complex queries ( extended version ),

    T. Strohman, D. Metzler, H. R. Turtle, and W. B. Croft, “Indri : A language-model based search engine for complex queries ( extended version ),” 2005. [Online]. Available: https://api.semanticscholar.org/ CorpusID:18471028

    2005

  75. [83]

    Amalgam+: Composing rich information sources for accurate bug localization,

    S. Wang and D. Lo, “Amalgam+: Composing rich information sources for accurate bug localization,” Journal of Software: Evolution and Process, vol. 28, no. 10, pp. 921–942, 2016

    2016

  76. [84]

    Feature location using probabilistic ranking of methods based on execution scenarios and information retrieval,

    D. Poshyvanyk, Y .-G. Guneuc, A. Marcus, G. Antoniol, and V . Rajlich, “Feature location using probabilistic ranking of methods based on execution scenarios and information retrieval,” IEEE Transactions on Software Engineering, vol. 33, no. 6, pp. 420–432, 2007

    2007

  77. [85]

    On the relationship between bug reports and queries for text retrieval-based bug localization,

    C. Mills, E. Parra, J. Pantiuchina, G. Bavota, and S. Haiduc, “On the relationship between bug reports and queries for text retrieval-based bug localization,” Empirical Software Engineering , vol. 25, pp. 3086–3127, 2020

    2020

  78. [86]

    Fast changeset-based bug localization with bert,

    A. Ciborowska and K. Damevski, “Fast changeset-based bug localization with bert,” in Proceedings of the 44th International Conference on Software Engineering, 2022, pp. 946–957

    2022

  79. [87]

    Colbert: Efficient and effective passage search via contextualized late interaction over bert,

    O. Khattab and M. Zaharia, “Colbert: Efficient and effective passage search via contextualized late interaction over bert,” in Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval , 2020, pp. 39–48

    2020

  80. [88]

    Enhancing the unified features to locate buggy files by exploiting the sequential nature of source code

    X. Huo and M. Li, “Enhancing the unified features to locate buggy files by exploiting the sequential nature of source code.” in IJCAI, 2017, pp. 1909–1915

    2017

  81. [89]

    Cooba: Cross-project bug localization via adversarial transfer learning,

    Z. Zhu, Y . Li, H. Tong, and Y . Wang, “Cooba: Cross-project bug localization via adversarial transfer learning,” in IJCAI, 2020

    2020

  82. [90]

    Rack: Automatic api recom- mendation using crowdsourced knowledge

    M. M. Rahman, C. K. Roy, and D. Lo, “Rack: Automatic api recom- mendation using crowdsourced knowledge.” Institute of Electrical and Electronics Engineers (IEEE), 5 2016, pp. 349–359

    2016

  83. [91]

    Harman, Information retrieval evaluation

    D. Harman, Information retrieval evaluation . Morgan & Claypool Publishers, 2011

    2011

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.