Pith. sign in

REVIEW 4 major objections 4 minor 39 references

Lighting the Way for BRIGHT: Reproducible Baselines with Anserini, Pyserini, and RankLLM

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper shows that the BRIGHT benchmark's reported BM25 baseline was actually produced by query-side BM25—applying BM25 weighting to query terms—and that on long reasoning queries this detail beats standard bag-of-words BM25, with…

desk verdict Useful reproducibility paper that correctly identifies BRIGHT's BM25 as query-side BM25, but the abstract overclaims a five-benchmark study the body explicitly says was not done. read the letter →

arxiv 2509.02558 v2 pith:LOS4FBTJ submitted 2025-09-02 cs.IR

Sahel Sharifymoghaddam , Yijun Ge , Jimmy Lin This is my paper · ORCID
classification cs.IR
keywords query-sideBM25BRIGHTbenchmarklongqueriesretrieval-augmentedgenerationreciprocalrankfusionLLMrerankingreproduciblebaselines
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that in the era of long, reasoning-intensive queries, the standard way BM25 builds query vectors—counting query terms as a bag of words—should be reconsidered. Reproducing the lexical baseline of the BRIGHT benchmark, the authors find that the originally reported BM25 scores come from an implementation that weights each query term with BM25 itself (query-side BM25), and that on BRIGHT's long multi-sentence queries this variant outperforms bag-of-words: 0.148 vs 0.137 nDCG@10 overall. They turn the finding into reproducible baselines in the standard open-source retrieval and reranking toolkits they extend, and show that fusing BM25 with learned sparse and dense retrievers, then applying LLM listwise reranking, gives further large gains (overall nDCG@10 up to 0.274). A sympathetic reader would care because search toolkits default to bag-of-words query vectors, and this paper shows a small implementation choice that can matter as RAG prompts and LLM-expanded queries grow longer. The paper also audits the BRIGHT corpus, finding duplicate and degenerate documents and missing gold IDs that slightly distort evaluation.

What carries the argument

The load-bearing mechanism is the query-side BM25 query vector, in which each query term is weighted by the BM25 formula applied to the query itself—so terms that are frequent in the query and the corpus receive saturating, rarity-adjusted weights instead of raw counts. This replaces the bag-of-words query vector inside the same inner-product similarity scoring, isolating the query-representation choice from everything else. The mechanism is what allows long, multi-sentence queries with repeated token distributions to be reweighted, and the paper's experiments compare it directly against bag-of-words while controlling for document-length normalization quantization.

What would settle it

A paired significance test over per-query scores on the twelve BRIGHT tasks comparing query-side BM25 with bag-of-words, or an independent long-query benchmark with queries of 16-256 tokens, would settle it; if the advantage fails to exceed noise or to replicate, the central recommendation collapses.

Watch

Extended reading notes

Core claim

The central discovery is that the BRIGHT benchmark's reported BM25 baseline was produced by applying the BM25 scoring function to each query token to build the query vector, rather than by the bag-of-words weighting that standard toolkit defaults use; both approaches score documents by the inner product of the query and document vectors. On BRIGHT's long, reasoning-intensive queries, query-side BM25 is more effective overall (nDCG@10 0.148 vs 0.137), winning 7 of 12 tasks, with per-query analysis showing gains grow in frequency and size as query length increases until BM25's length-normalization term dominates. The authors make query-side BM25 available as a standard option in the toolkits they extend, recommend it as the strongest lexical baseline for reasoning-oriented retrieval, and find that its advantage does not generalize to five other benchmarks, while fusing BM25 with learned retrievers (or with standard BM25) is the most consistently helpful choice.

Load-bearing premise

The load-bearing premise is that the observed advantage of query-side BM25 on mid-length queries is a real effect rather than noise: it rests on a small 0.011-point average gain, mixed per-task results, and a length-bucket trend from a benchmark that has no development/validation split to confirm it.

Editorial extensions

If this is right

  • On BRIGHT, query-side BM25 becomes the lexical baseline to beat: 0.148 vs 0.137 nDCG@10 over bag-of-words, with wins on 7 of 12 tasks.
  • For queries of roughly 16-256 tokens, the paper recommends applying BM25 weighting to the query; below 16 tokens the two query representations are nearly identical.
  • Fusing BM25 with learned sparse or dense retrievers via reciprocal rank fusion beats every individual retriever on BRIGHT (overall 0.172-0.173 vs 0.137-0.156 nDCG@10), and the BM25-plus-dense combination is best.
  • LLM listwise reranking of the top 100 results raises overall nDCG@10 from about 0.16-0.17 at first stage to 0.241-0.274, with the larger open-weight reranker consistently better.
  • Duplicate documents, near-empty chunks, and missing gold IDs in BRIGHT distort evaluation by up to 0.010 nDCG@10 after reranking; the authors release adjusted relevance labels.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 16-256 token recommendation is post hoc and untested for significance; a natural extension is a paired per-query significance test across the twelve tasks, which the authors do not report.
  • Because BRIGHT has no dedicated development/validation split, the length-bucket pattern in Figure 1 is an observed trend; holding out a random query subset and checking whether BM25Q's advantage persists would separate the length effect from task-specific quirks.
  • If query-side weighting helps long queries, it may compound with LLM query expansion, whose outputs are long and token-repetitive; the paper does not test BM25Q on the LLM-expanded queries BRIGHT provides, which would be a direct next experiment.
  • The corpus audit implies a cheap benchmark repair—deduplicating documents and propagating gold labels across duplicate IDs—that the adjusted qrels approximate; the paper predicts stronger retrievers will be even more penalized by missing gold IDs, which is testable with the released qrels.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents reproducible first-stage retrieval and reranking baselines for the BRIGHT benchmark, integrated into Anserini, Pyserini, and RankLLM. It identifies that the BRIGHT repository's BM25 implementation weights query terms with BM25 ('query-side BM25') rather than using the usual bag-of-words query vectors, replicates the BRIGHT BM25 scores using the 'accurate' variant, evaluates dense (BGE-large-en-v1.5), sparse (SPLADE-v3), and RRF/fusion combinations, adds LLM listwise reranking results with Qwen3-8b and gpt-oss-20b, and audits the corpus for duplicate, short, and empty documents plus missing gold IDs. The abstract additionally claims a five-benchmark generalization study of BM25Q, but the body's Section 5 states the method was evaluated only on BRIGHT.

Significance. The toolkit integrations and released runs/indexes are valuable and reproducible; the discovery that BRIGHT's BM25 scores come from query-side weighting is a concrete, useful observation for the community, and the exact replication (Table 1 columns d vs e) is a strength. However, the central generalization claims—'consistently outperforms' and the 16–256 token recommendation—are not supported by significance testing or a validation split, and the five-benchmark study promised in the abstract is absent from the body. The paper is a useful resource but needs revision before its headline claims can be accepted.

major comments (4)
  1. [Abstract / §5] The abstract states that the paper 'study[ies] the generalizability of BM25Q across five additional benchmarks,' but Section 5 (Limitations and Future Work) explicitly says 'we evaluated query-side BM25 only on BRIGHT.' No section of the manuscript reports experiments on additional benchmarks. This is a direct factual inconsistency: either the experiments must be added or the abstract must be revised to state the scope.
  2. [§4.1, Table 1, Figure 1] The claim that BM25Q 'consistently outperforms' bag-of-words is not statistically established. The aggregate nDCG@10 difference is 0.011 (Table 1, 0.148 vs 0.137), but the per-task comparison is 7 wins, 4 losses, 1 tie, which is not significant by a sign test (one-sided p≈0.27). No paired significance test at the query level is reported, and Figure 1's length-stratified deltas are inspected post hoc on the same data for which no development split exists (stated in §4.1). As written, the recommendation to prefer BM25Q for 16–256 token queries is an observed trend, not a validated rule.
  3. [§4.1, 'recommend applying BM25 weighting to queries of roughly 16–256 tokens'] The 16–256 token recommendation is derived from the same BRIGHT data used to produce the comparison, with no held-out validation or significance test for the bucket boundaries. The paper should either provide a proper validation methodology (e.g., cross-task splits, bootstrap confidence intervals) or soften the recommendation to a hypothesis for future work.
  4. [§4.2, Table 3] The text claims that in the RRF section 'all scores in the RRF section exceed their counterparts in the individual section,' but this is not true for every cell: for example, Biology BM25 alone is 0.197 while BM25+BGE fusion (column e) is 0.175. The claim of consistent fusion gains should be qualified to overall averages or to a stated subset of tasks, and the actual per-task behavior should be described accurately.
minor comments (4)
  1. [§3.1] There is a typo in the last paragraph: 'Anserini baslines' should be 'Anserini baselines.'
  2. [Figure 1] The figure label 'BM25 - BoW' is confusing because the paper elsewhere distinguishes 'BM25Q' (query-side BM25) from 'BM25' (which in Table 1 refers to the generic method); the axis label should read 'BM25Q - BoW' or the caption should explicitly define the notation.
  3. [Table 1] The header 'BRIGHT' for column (e) is ambiguous; it is the BRIGHT-reported score, so the caption or table header should say 'BRIGHT reported' to distinguish it from the authors' own 'BM25Q Acc.' column.
  4. [§4.2] The sentence 'Combining BM25 with either S-v3 or BGE proves more effective than fusing S-v3 and BGE' is based on overall averages; readers should be reminded that per-task exceptions exist (e.g., Psychology in Table 3 where S-v3+BGE exceeds BM25+BGE).

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: the BM25Q-versus-bag-of-words comparison is a direct external measurement; the abstract's five-benchmark claim is contradicted by Section 5, but that is a support error, not circularity.

full rationale

The paper's central claim is an empirical comparison between two query-vector constructions on BRIGHT, an externally labeled benchmark. The nDCG@10 scores in Table 1 are direct measurements under fixed k1=0.9 and b=0.4; no parameter is fitted to the comparison outcome, and no equation is defined in terms of the result it is supposed to establish. The 16-256-token recommendation is a post hoc reading of Figure 1, not a fitted parameter renamed as a prediction, and the paper itself notes in Section 4.1 that BRIGHT lacks a dedicated dev/validation split. Self-citations to Anserini, Pyserini, RankLLM, and Kamphuis et al. describe the tooling or corroborate the subordinate quantization point; none is used to justify the BM25Q advantage, so the self-citation is not load-bearing. Explicit flags from the text: Section 5 states, 'we evaluated query-side BM25 only on BRIGHT, where we initially discovered its effectiveness,' directly contradicting the abstract's claim of a five-benchmark generalization study; this is an unsupported or inconsistent assertion, not a circular reduction. The abstract mismatch and the lack of significance testing are correctness and evidence-quality concerns, not derivation-by-construction. Thus the paper is substantially self-contained with respect to its central measurement, and the score reflects only minor consistency and self-citation concerns.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central empirical comparison is largely self-contained: it uses standard BM25 variants, SPLADE-v3, BGE, and RRF on an external benchmark, and it introduces no new theoretical entities. The main assumptions are the choices of fixed BM25 hyperparameters (k1=0.9, b=0.4), the validity of nDCG@10 and the adjusted qrels, and the representativeness of BRIGHT's long queries for RAG. The abstract's claimed five-benchmark generalization is not backed by any listed artifact, which is a missing evidence statement rather than an extra assumption.

free parameters (3)
  • BM25 k1 = 0.9
    BM25 scoring depends on k1; the paper fixes k1=0.9 (from BRIGHT) across all variants. Since the bag-of-words versus query-side BM25 comparison is measured under this fixed value, the ordering could change if k1 were re-tuned, and the authors explicitly avoid tuning because BRIGHT lacks a dev split.
  • BM25 b = 0.4
    b controls document length normalization and is fixed at 0.4. The paper keeps b fixed instead of re-tuning it for exact normalization, citing the lack of a development/validation split in BRIGHT (Section 4.1).
  • RankLLM context length = 16k tokens
    The authors extend RankLLM's default 4k context to 16k to accommodate BRIGHT's long queries and documents (Section 3.2). This is a hand-chosen setting required for the reranking runs, not a constant fitted to maximize scores.
assumptions (4)
  • domain assumption BRIGHT's relevance labels and nDCG@10 are a valid operationalization of retrieval quality for reasoning-intensive RAG queries.
    All effectiveness conclusions are expressed as nDCG@10 over BRIGHT qrels. Section 4.3 shows the authors do not trust the original labels completely, since they add missing gold IDs, but the adjusted qrels are still treated as ground truth.
  • standard math The Lucene and Pyserini implementations of BM25, including approximate length normalization, faithfully instantiate the BM25 variants under study.
    The paper relies on toolkit implementations and argues quantization has negligible impact (Section 3.1 and Table 1 comparison of columns a/b and c/d). If the implementations deviated from the intended scoring functions, the BM25Q versus bag-of-words comparison would not be clean.
  • domain assumption Exact duplicate documents are interchangeable for relevance labeling.
    The adjusted-qrels procedure in Section 4.3 propagates relevance from any member of a duplicate set to all members. This is plausible for chunked passages but is an assumption about how the benchmark's evidence should be evaluated.
  • domain assumption RRF fusion with the default formulation is an appropriate combination method without tuning.
    Fusion conclusions (Tables 3, 8, 9) depend on the chosen RRF implementation and any implicit constants; the paper does not report an RRF k value or a parameter search.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Lighting the Way for BRIGHT: Reproducible Baselines with Anserini, Pyserini, and RankLLM." pith.science (2026). https://pith.science/paper/LOS4FBTJ

@misc{pith2026250902558,
  author       = {Pith},
  title        = {Pith review of: Lighting the Way for BRIGHT: Reproducible Baselines with Anserini, Pyserini, and RankLLM},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LOS4FBTJ}},
  note         = {Machine review of arXiv:2509.02558}
}
read the original abstract

Retrieval benchmarks for large language models (LLMs) should reflect the long, reasoning-intensive queries typical of retrieval-augmented generation (RAG). We present a systematic study of BRIGHT, a reasoning-focused retrieval benchmark, along with strong, reproducible reference methods integrated into Anserini, Pyserini, and RankLLM. We evaluate lexical, sparse, dense, and fusion-based retrievers, as well as LLM rerankers, under long-query settings. In reproducing BRIGHT's lexical baseline, we identify a key under-documented detail: query-side BM25 (BM25Q), which applies BM25 weighting to the query itself. On long, multi-sentence queries, BM25Q consistently outperforms standard BM25, making it the strongest lexical baseline for reasoning-oriented retrieval. We further audit the BRIGHT corpus, uncovering data quality issues that impact evaluation, and offer mitigation. Finally, we study the generalizability of BM25Q across five additional benchmarks, finding its gains largely specific to BRIGHT, while fusion with standard BM25 provides the most consistent improvements across datasets.

Figures

Figures reproduced from arXiv: 2509.02558 by the authors.

Figure 1
Figure 1. Query length (in tokens) vs. ∆nDCG@10 (BM25 - BoW) for all queries in BRIGHT tasks. • How queries are represented—treating them as bag-of-words vs. applying BM25 directly per-term, that is, columns (a)/(b) vs. columns (c)/(d) in [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 24 canonical work pages

  1. [1]

    Apache Lucene 4

    Andrzej Białecki, Robert Muir, Grant Ingersoll, and Lucid Imagination. Apache Lucene 4. In SIGIR 2012 Workshop on Open Source Information Retrieval, page 17. sn, 2012

    2012

  2. [2]

    Overview of the TREC 2020 Deep Learning Track

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. Overview of the TREC 2020 Deep Learning Track. In Proceedings of the Twenty-Ninth Text REtrieval Conference (TREC 2021), 2021

    work page 2020

  3. [3]

    Overview of the TREC 2021 Deep Learning Track

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Jimmy Lin. Overview of the TREC 2021 Deep Learning Track. In Proceedings of the Thirtieth Text REtrieval Conference (TREC 2021), 2021

    work page 2021

  4. [4]

    V oorhees, and Ian Soboroff

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, Jimmy Lin, Ellen M. V oorhees, and Ian Soboroff. Overview of the TREC 2022 Deep Learning Track. In Proceedings of the Thirty-First Text REtrieval Conference (TREC 2022), 2022

    work page 2022

  5. [5]

    V oorhees

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. V oorhees. Overview of the TREC 2019 Deep Learning Track. arXiv:2003.07820, 2020

    arXiv 2019

  6. [6]

    Rahmani, Daniel Campos, Jimmy Lin, Ellen M

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Hossein A. Rahmani, Daniel Campos, Jimmy Lin, Ellen M. V oorhees, and Ian Soboroff. Overview of the TREC 2023 Deep Learning Track.arXiv:2507.08890, 2025

    arXiv 2023

  7. [7]

    The Faiss Library

    Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The Faiss Library. arXiv:2401.08281, 2024

    arXiv 2024

  8. [8]

    FIRST: Faster Improved Listwise Reranking with Single Token Decoding

    Revanth Gangi Reddy, JaeHyeok Doo, Yifei Xu, Md Arafat Sultan, Deevya Swain, Avirup Sil, and Heng Ji. FIRST: Faster Improved Listwise Reranking with Single Token Decoding. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024), pages 8642–8652, 2024

    work page 2024

Show all 39 references
  1. [9]

    de Vries, Leonid Boytsov, and Jimmy Lin

    Chris Kamphuis, Arjen P. de Vries, Leonid Boytsov, and Jimmy Lin. Which BM25 Do You Mean? A Large-Scale Reproducibility Study of Scoring Variants. In Advances in Information Retrieval: 42nd European Conference on IR Research (ECIR 2020), Part II, page 28–34, 2020

    2020

  2. [10]

    Dense Passage Retrieval for Open-Domain Question Answering

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 202...

    2020

  3. [11]

    SPLADE-v3: New Baselines for SPLADE

    Carlos Lassance, Hervé Déjean, Thibault Formal, and Stéphane Clinchant. SPLADE-v3: New Baselines for SPLADE. arXiv:2403.06789, 2024

    2024 arXiv

  4. [12]

    Retrieval-Augmented Generation for Knowledge- Intensive NLP Tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-Augmented Generation for Knowledge- Intensive NLP Tasks. Advances in Neural Information Processing System...

    2020

  5. [13]

    R2MED: A Benchmark for Reasoning-Driven Medical Retrieval

    Lei Li, Xiao Zhou, and Zheng Liu. R2MED: A Benchmark for Reasoning-Driven Medical Retrieval. arXiv:2505.14558, 2025

    2025 arXiv

  6. [14]

    Multi-Task Dense Retrieval via Model Uncertainty Fusion for Open-Domain Question Answering

    Minghan Li, Ming Li, Kun Xiong, and Jimmy Lin. Multi-Task Dense Retrieval via Model Uncertainty Fusion for Open-Domain Question Answering. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Findings of the Association for Computational Lingu...

    2021

  7. [15]

    A Proposed Conceptual Framework for a Representational Approach to Information Retrieval

    Jimmy Lin. A Proposed Conceptual Framework for a Representational Approach to Information Retrieval. arXiv:2110.01529, 2021

    2021 arXiv

  8. [16]

    Gosling Grows Up: Retrieval with Learned Dense and Sparse Representations Using Anserini

    Jimmy Lin, Arthur Haonan Chen, Carlos Lassance, Xueguang Ma, Ronak Pradeep, Tommaso Teofili, Jasper Xian, Jheng-Hong Yang, Brayden Zhong, and Vincent Zhong. Gosling Grows Up: Retrieval with Learned Dense and Sparse Representations Using Anserini. In Proceedings of the 48th Int...

    2025

  9. [17]

    Pyserini: A Python Toolkit for Reproducible Information Retrieval Research with Sparse and Dense Representations

    Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. Pyserini: A Python Toolkit for Reproducible Information Retrieval Research with Sparse and Dense Representations. In Proceedings of the 44th Annual International ACM SIGIR Conference...

    2021

  10. [18]

    PubMed Related Articles: A Probabilistic Topic-Based Model for Content Similarity

    Jimmy Lin and W John Wilbur. PubMed Related Articles: A Probabilistic Topic-Based Model for Content Similarity. BMC Bioinformatics, 8(1):423, 2007

    2007

  11. [19]

    ReasonRank: Empowering Passage Ranking with Strong Reasoning Ability

    Wenhan Liu, Xinyu Ma, Weiwei Sun, Yutao Zhu, Yuchen Li, Dawei Yin, and Zhicheng Dou. ReasonRank: Empowering Passage Ranking with Strong Reasoning Ability. arXiv:2508.07050, 2025

    2025 arXiv

  12. [20]

    DIVER: A Multi-Stage Approach for Reasoning-Intensive Information Retrieval.arXiv:2508.07995, 2025

    Meixiu Long, Duolin Sun, Dan Yang, Junjie Wang, Yue Shen, Jian Wang, Peng Wei, Jinjie Gu, and Jiahai Wang. DIVER: A Multi-Stage Approach for Reasoning-Intensive Information Retrieval.arXiv:2508.07995, 2025

    2025

  13. [21]

    Zero-Shot Listwise Document Reranking with a Large Language Model

    Xueguang Ma, Xinyu Zhang, Ronak Pradeep, and Jimmy Lin. Zero-Shot Listwise Document Reranking with a Large Language Model. arXiv:2305.02156, 2023

    2023 arXiv

  14. [22]

    RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze! arXiv:2312.02724, 2023

    Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze! arXiv:2312.02724, 2023

    2023 arXiv

  15. [23]

    Language Models are Unsupervised Multitask Learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language Models are Unsupervised Multitask Learners. OpenAI Blog, 1(8):9, 2019

    2019

  16. [24]

    In-context Retrieval-Augmented Language Models

    Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. In-context Retrieval-Augmented Language Models. Transactions of the Association for Computational Linguistics, 11:1316–1331, 2023

    2023

  17. [25]

    Software Framework for Topic Modelling with Large Corpora

    Radim ˇReh˚ uˇrek and Petr Sojka. Software Framework for Topic Modelling with Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, pages 45–50, 2010

    2010

  18. [26]

    Walker, M

    Stephen Robertson, S. Walker, M. M. Hancock-Beaulieu, M. Gatford, and A. Payne. Okapi at TREC-4. In Proceedings of the Fourth Text REtrieval Conference (TREC 4), pages 73–96, 1996

    1996

  19. [27]

    Walker, S

    Stephen Robertson, S. Walker, S. Jones, M. M. Hancock-Beaulieu, and M. Gatford. Okapi at TREC-3. In Proceedings of the Third Text REtrieval Conference (TREC 3), pages 109–126, 1995

    1995

  20. [28]

    The Probabilistic Relevance Framework: BM25 and Beyond

    Stephen Robertson and Hugo Zaragoza. The Probabilistic Relevance Framework: BM25 and Beyond. Found. Trends Inf. Retr., 3(4):333–389, 2009

    2009

  21. [29]

    Optimizing Base Rankers Using Clicks

    Anne Schuth, Floor Sietsma, Shimon Whiteson, and Maarten de Rijke. Optimizing Base Rankers Using Clicks. In Maarten de Rijke, Tom Kenter, Arjen P. de Vries, ChengXiang Zhai, Franciska de Jong, Kira Radinsky, and Katja Hofmann, editors, Advances in Information Retrieval, pages ...

    2014

  22. [30]

    ReasonIR: Training Retrievers for Reasoning Tasks

    Rulin Shao, Rui Qiao, Varsha Kishore, Niklas Muennighoff, Xi Victoria Lin, Daniela Rus, Bryan Kian Hsiang Low, Sewon Min, Wen-tau Yih, Pang Wei Koh, et al. ReasonIR: Training Retrievers for Reasoning Tasks. arXiv:2504.20595, 2025

    2025 arXiv

  23. [31]

    RankLLM: A Python Package for Reranking with LLMs

    Sahel Sharifymoghaddam, Ronak Pradeep, Andre Slavescu, Ryan Nguyen, Andrew Xu, Zijian Chen, Yilin Zhang, Yidi Chen, Jasper Xian, and Jimmy Lin. RankLLM: A Python Package for Reranking with LLMs. In Proceedings of the 48th International ACM SIGIR Conference on Research and Deve...

    2025

  24. [32]

    Siegel, Michael Tang, Ruoxi Sun, Jinsung Yoon, Sercan O

    Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han yu Wang, Haisu Liu, Quan Shi, Zachary S. Siegel, Michael Tang, Ruoxi Sun, Jinsung Yoon, Sercan O. Arik, Danqi Chen, and Tao Yu. BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retri...

    2025 arXiv

  25. [33]

    Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents

    Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Proce...

    2023

  26. [34]

    QwQ-32B: Embracing the Power of Reinforcement Learning, 2025

    Qwen Team. QwQ-32B: Embracing the Power of Reinforcement Learning, 2025

    2025

  27. [35]

    BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models

    Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Ro...

    2021

  28. [36]

    C-Pack: Packaged Resources to Advance General Chinese Embedding

    Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-Pack: Packaged Resources to Advance General Chinese Embedding. arXiv:2309.07597, 2023

    2023 arXiv

  29. [37]

    Anserini: Reproducible Ranking Baselines Using Lucene

    Peilin Yang, Hui Fang, and Jimmy Lin. Anserini: Reproducible Ranking Baselines Using Lucene. J. Data and Information Quality, 10(4), 2018

    2018

  30. [38]

    Qwen3 Embedding: Advancing Text Embedding and Reranking through Foundation Models

    Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 Embedding: Advancing Text Embedding and Reranking through Foundation Models. arXiv:2506.05176, 2025

    2025 arXiv

  31. [39]

    Rank-R1: Enhancing Reasoning in LLM-Based Document Rerankers via Reinforcement Learning

    Shengyao Zhuang, Xueguang Ma, Bevan Koopman, Jimmy Lin, and Guido Zuccon. Rank-R1: Enhancing Reasoning in LLM-Based Document Rerankers via Reinforcement Learning. arXiv:2503.06034, 2025. 13 A Dependency Configuration for RankLLM Table 7: Dependency versions used with RankLLM f...

    2025 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.