Pith. sign in

REVIEW 2 major objections 5 minor 57 references

RankLLM: A Python Package for Reranking with LLMs

T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RankLLM is a modular Python package that reproduces published LLM reranking results across pointwise, pairwise, and listwise methods.

desk verdict A solid software resource paper: the package is real and reproducibly demonstrated, though the word 'reproduce' overreaches for the GPT-based rows because a different model was used. read the letter →

arxiv 2505.19284 v1 pith:FCGF5R3H submitted 2025-05-25 cs.IR

Sahel Sharifymoghaddam , Ronak Pradeep , Andre Slavescu , Ryan Nguyen , Andrew Xu , Zijian Chen , Yilin Zhang , Yidi Chen
show 2 more authors
Jasper Xian Jimmy Lin
This is my paper
classification cs.IR
keywords RankLLMrerankinglargelanguagemodelslistwisepointwisepairwisereproducibilityretrieval-augmentedgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RankLLM is an open-source Python package that puts LLM-based reranking behind a single, configurable interface. The paper claims that one package can support pointwise, pairwise, and listwise reranking across proprietary and open-source models, and can reproduce reported effectiveness numbers from methods such as RankGPT, LRL, RankVicuna, RankZephyr, LiT5, and FIRST on the TREC Deep Learning track benchmarks. The motivation is that reranking research has relied on ad hoc, incompatible implementations, which makes comparisons slow and reproduction hard; a shared package is meant to change that. A reader should care because, if the claim holds, researchers and RAG practitioners can swap and evaluate rerankers in a few lines of code instead of reconstructing each method.

What carries the argument

The load-bearing machinery is the model coordinator plus the sliding-window listwise algorithm. A coordinator turns a Request (a query and candidate list) into a Result (a reordered list plus an invocation history), and each coordinator owns its prompt template, model endpoint, and a post-processing step that removes extra tokens, de-duplicates IDs, and appends missing IDs so the output is always a valid permutation. For long candidate lists, a sliding window with configurable window size and stride reorders the list in chunks from the end toward the beginning; a window of size two with stride one reduces to bubble sort. This combination is what lets one package span pointwise, pairwise, and listwise methods while keeping reproduction and analysis unified.

What would settle it

Run the same DL19-DL23 top-100 BM25 requests through the original RankGPT and LRL codebases with their original models and through RankLLM, then compare per-query rankings. A systematic per-query divergence beyond known nondeterminism would show that RankLLM reproduces the numbers but not the methods.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that RankLLM is a modular and extensible platform that covers the full reranking workflow: retrieve candidate lists, rerank them with an LLM, evaluate the results, analyze model invocations, and optionally train a custom reranker. The core architectural idea is the model coordinator, a class that encapsulates a prompt template, an inference framework, and response post-processing so that the same reranker interface works for MonoT5, DuoT5, LiT5, RankVicuna, RankZephyr, FirstMistral, GPT-4o-mini, Gemini, and any vLLM/SGLang/TensorRT-LLM-compatible model. The paper reports nDCG@10 results on DL19-DL23 reranking the top 100 BM25 candidates, reproducing published results from RankGPT, LRL, RankVicuna, RankZephyr, and other recent models, with all GPT runs using GPT-4o-mini with different prompt templates. It also reports that graceful post-processing of malformed LLM responses keeps effectiveness competitive even when 28-75% of GPT-4o-mini responses contain missing candidate ids.

Load-bearing premise

The faithful-reproduction claim assumes that RankLLM's prompt templates, inference calls, and response post-processing match the original methods closely enough for the Table 1 numbers to demonstrate reproduction, even though GPT rows use GPT-4o-mini rather than the original GPT models and no direct comparison against the original codebases is reported.

Editorial extensions

If this is right

  • New reranking methods can be benchmarked against RankGPT, RankZephyr, LiT5, FIRST, and other baselines without reimplementing each baseline's prompts and post-processing.
  • A RAG pipeline can switch between a proprietary model like GPT-4o-mini and an open-source model served by vLLM by changing one coordinator, making cost-effectiveness comparisons straightforward.
  • Reported results become checkable through the included two-click reproduction pages and end-to-end scripts, lowering the cost of verifying or extending published numbers.
  • The training module with learning-to-rank losses lets teams fine-tune custom rerankers using the same data format and evaluation tooling, so reproduction and innovation share one path.
  • The invocation-analysis module turns nondeterministic or malformed model behavior into inspectable statistics, which could make LLM API reliability issues visible during experiments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if RankLLM becomes a common substrate, the field may consolidate around a few standard prompt templates and post-processing rules, making reported gains easier to attribute to the model rather than to implementation details.
  • Editorial inference: the malformed-response tolerance suggests that cheap instruction-tuned models can be used in production reranking without strict output-format enforcement; a stress test on longer candidate lists or smaller models would show how far this tolerance extends.
  • Editorial inference: because the paper reports single-run results and uses GPT-4o-mini for GPT-family rows, the reproduction numbers are best read as engineering demonstrations; a head-to-head run against the original codebases would make the fidelity claim directly testable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces RankLLM, an open-source Python package for reranking with large language models. It supports pointwise, pairwise, and listwise reranking, integrates with Pyserini for retrieval, offers evaluation and response-analysis tools, and includes a training module. The authors demonstrate the package by reporting nDCG@10 on TREC DL19–DL23 for a range of reranking models, including MonoT5, DuoT5, LiT5, RankVicuna, RankZephyr, FirstMistral, and several out-of-the-box LLMs such as GPT-4o-mini and Gemini. The package is available on PyPI and GitHub, and the reported results are generated by an end-to-end script with 2CR reproduction pages.

Significance. If the package works as described, it is a valuable community resource: a modular, configurable, and open-source toolkit that unifies diverse reranking approaches and lowers the barrier to reproducible experimentation. Concrete strengths include a versioned package, a public repository, an end-to-end demo script that produced Table 1 and Table 2, integration with established frameworks (e.g., Pyserini, LangChain, LlamaIndex), and support for multiple inference backends. The paper does not introduce a new ranking method, but as a software resource its significance is real, especially for the SIGIR resource track.

major comments (2)
  1. [§5, Table 1 rows 4d–4f; Abstract] The claim that RankLLM 'reproduces results from RankGPT, LRL, ...' is not supported by the experiments as reported, because all GPT-based runs use GPT-4o-mini rather than the models used in the original papers (e.g., RankGPT used GPT-3.5/GPT-4, LRL used GPT-3.5-turbo). The row labels 'RankGPT*', 'RankGPTAPEER*', and 'LRL*' thus reflect the prompt templates, not the original model outputs. The asterisk in Table 1 and the note in Section 5 are transparent, but the abstract and introduction still assert reproduction without this caveat. To support the reproduction claim, the authors should either (a) provide a side-by-side comparison with the original repositories on identical inputs and models, or (b) explicitly rephrase the claim to state that RankLLM reproduces the experimental pipelines (prompts, post-processing, evaluation) and can produce comparable results with newer models. As written, the reproduction claim is an assertion rather than a demonstrated result.
  2. [§5, Table 1] All reported nDCG@10 values come from a single run of non-deterministic models, with no variance or significance testing. The authors acknowledge this cost-saving choice, but because the paper's central claim is reproducibility, a single run cannot establish that the package reliably reproduces reported numbers, especially for the GPT-based rows where Table 2 shows 28–75% of responses have missing candidate IDs and thus rely heavily on the graceful post-processing of §4.3.3. At minimum, a small multi-run experiment (e.g., 3–5 runs on DL19) with mean and standard deviation for one GPT-based and one open-source model would demonstrate stability and materially strengthen the reproducibility claim.
minor comments (5)
  1. [Footnotes 1, 3, 4, 5] The URLs in these footnotes are malformed, with duplicated text such as 'huggingface.co/docs/transformershuggingface.co/docs/transformers'; each should be a single, clickable URL.
  2. [§4.5, Eq. (1)] The RankNet loss as written appears to have the indicator and denominator arranged unusually: 1_{r_i < r_j}/(i+j) log(1+exp(s_i - s_j)). Please verify this against the original RankNet formulation and the FIRST paper; if this is a weighted variant, clarify the notation and the intended semantics.
  3. [§4.3.3, Sliding Window Algorithm] The formula for the number of model calls in a single pass, ceil((k−M)/N), appears to be off by one; for k=100, M=20, N=10 the described process requires 9 invocations (windows at 81–100, 71–90, ..., 1–20), not 8. Please correct the formula or the description.
  4. [§1, Introduction] The phrase 'two-click reproducibility' is used without a definition; the linked 2CR pages are helpful, but a one-sentence explanation of what the two clicks are would improve clarity for readers.
  5. [§5, Table 2] The table would be easier to interpret if the rows were ordered consistently with Table 1 and if the percentages were accompanied by the total number of invocations, since the denominators differ across models.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the package's claims are validated by running the open-source code on external benchmarks, with self-citation present but not load-bearing.

full rationale

This paper is a software-system description rather than a derivation. Its central claim is that RankLLM supports pointwise, pairwise, and listwise reranking and reproduces published results, supported by Table 1, which reports nDCG@10 over external DL19-DL23 benchmarks with a BM25 first stage and standard trec_eval-based evaluation. No model parameter is fitted to the reported metrics, no prediction is defined in terms of the quantity it is said to predict, and no conclusion is forced by a self-citation chain. The reproduced methods (RankGPT, LRL, RankVicuna, RankZephyr, LiT5, FIRST, etc.) are independently published artifacts; several are co-authored by members of the same group, but the paper does not use those citations as an unverified substitute for evidence, since the package itself is open source and the numbers come from running its own code end to end. The disclosed substitution of GPT-4o-mini for the original GPT models and the absence of a side-by-side comparison against the original codebases weaken the fidelity of the RankGPT/LRL reproduction claim, but this is a validation concern, not a circularity. There is no equation, fitted parameter, or self-referential definition that reduces the central result to its own inputs, so the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim of the paper is about the software working, so it rests on practical and benchmark assumptions rather than mathematical axioms. The listed assumptions are the ones a skeptical reader would need to accept for the reproduction results to be meaningful.

assumptions (3)
  • domain assumption The TREC DL19-DL23 datasets and nDCG@10 are valid benchmarks for assessing reranking effectiveness.
    Used as the evaluation protocol throughout Section 5; the paper does not justify this choice beyond standard practice in prior work.
  • ad hoc to paper The default sliding window parameters (window size 20, stride 10) and prompt-then-parse post-processing yield rankings consistent with the original methods being reproduced.
    Chosen in Section 4.3 and used for all Table 1 experiments; no sensitivity analysis is provided.
  • domain assumption Underlying inference frameworks (vLLM, SGLang, TensorRT-LLM) and Hugging Face Transformers correctly execute the model computations without introducing errors.
    The package relies on these external libraries for all model inference (Sections 4.3 and 4.5).

how reviews work

0 comments
Cite this review

Pith. "Pith review of RankLLM: A Python Package for Reranking with LLMs." pith.science (2026). https://pith.science/paper/FCGF5R3H

@misc{pith2026250519284,
  author       = {Pith},
  title        = {Pith review of: RankLLM: A Python Package for Reranking with LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FCGF5R3H}},
  note         = {Machine review of arXiv:2505.19284}
}
read the original abstract

The adoption of large language models (LLMs) as rerankers in multi-stage retrieval systems has gained significant traction in academia and industry. These models refine a candidate list of retrieved documents, often through carefully designed prompts, and are typically used in applications built on retrieval-augmented generation (RAG). This paper introduces RankLLM, an open-source Python package for reranking that is modular, highly configurable, and supports both proprietary and open-source LLMs in customized reranking workflows. To improve usability, RankLLM features optional integration with Pyserini for retrieval and provides integrated evaluation for multi-stage pipelines. Additionally, RankLLM includes a module for detailed analysis of input prompts and LLM responses, addressing reliability concerns with LLM APIs and non-deterministic behavior in Mixture-of-Experts (MoE) models. This paper presents the architecture of RankLLM, along with a detailed step-by-step guide and sample code. We reproduce results from RankGPT, LRL, RankVicuna, RankZephyr, and other recent models. RankLLM integrates with common inference frameworks and a wide range of LLMs. This compatibility allows for quick reproduction of reported results, helping to speed up both research and real-world applications. The complete repository is available at rankllm.ai, and the package can be installed via PyPI.

Figures

Figures reproduced from arXiv: 2505.19284 by the authors.

Figure 1
Figure 1. Overview of RankLLM with the Reranker component at the center. Other components facilitating optional flows [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Three reranking methods applied to a Query Q and a list of [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Sample requests creation either inline (lines 5–22) [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Sample requests creation via Pyserini retrieval. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 7
Figure 7. Figure 7: Reranking with different model coordinators: (a) [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 9
Figure 9. Figure 9: Sample training run for memory optimization.4 To monitor the training process, we incorporate logging to Weights & Biases (W&B).5 This module sup￾ports multiple training objectives, including traditional language modeling (LM) and various learning-to-rank losses, along…
Figure 8
Figure 8. Figure 8: Results evaluation and analysis 4.4 Evaluation and Analysis As an optional step, the Evaluation sub-package quantifies the effectiveness of reranking by evaluating the list[Result] output. By default, it measures the nDCG@10 metric, but users can specify other metrics,…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

57 extracted references · 32 canonical work pages

  1. [1]

    Abdelrahman Abdallah, Jamshid Mozafari, Bhawna Piryani, Mohammed Ali, and Adam Jatowt. 2025. Rankify: A Comprehensive Python Toolkit for Retrieval, Re-Ranking, and Retrieval-Augmented Generation.arXiv:2502.02464(2025)

    arXiv 2025

  2. [2]

    Bruce Croft, Fernando Diaz, Leah Larkey, Xiaoyan Li, Donald Metzler, Mark D

    Nasreen Abdul-Jaleel, James Allan, W. Bruce Croft, Fernando Diaz, Leah Larkey, Xiaoyan Li, Donald Metzler, Mark D. Smucker, Trevor Strohman, Howard Turtle, and Courtney Wade. 2004. UMass at TREC 2004: Novelty and HARD. InPro- ceedings of the Thirteenth Text REtrieval Conference (TREC 2004). Gaithersburg, Maryland

    work page 2004

  3. [3]

    Nima Asadi and Jimmy Lin. 2013. Effectiveness/Efficiency Tradeoffs for Candidate Generation in Multi-Stage Retrieval Architectures. InProceedings of the 36th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2013). Dublin, Ireland, 997–1000

    work page 2013

  4. [4]

    Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2016. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. arXiv:1611.09268v3(2016)

    arXiv 2016

  5. [5]

    Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, and Rodrigo Nogueira. 2022. InPars: Unsupervised Dataset Generation for Information Retrieval. InProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2022). Madrid, Spain, 2387–2392

    work page 2022

  6. [6]

    Leonid Boytsov, Preksha Patel, Vivek Sourabh, Riddhi Nisar, Sayani Kundu, Ramya Ramanathan, and Eric Nyberg. 2023. InPars-Light: Cost-Effective Unsu- pervised Training of Efficient Rankers.arXiv:2301.02998(2023)

    arXiv 2023

  7. [7]

    Barla Cambazoglu, Hugo Zaragoza, Olivier Chapelle, Jiang Chen, Ciya Liao, Zhaohui Zheng, and Jon Degenhardt

    B. Barla Cambazoglu, Hugo Zaragoza, Olivier Chapelle, Jiang Chen, Ciya Liao, Zhaohui Zheng, and Jon Degenhardt. 2010. Early Exit Optimizations for Additive Machine Learned Ranking Systems. InProceedings of the Third ACM International Conference on Web Search and Data Mining (WSDM 2010). New York, New York, 411–420

    work page 2010

  8. [8]

    2022.LangChain

    Harrison Chase. 2022.LangChain

    work page 2022

Show all 57 references
  1. [9]

    Zijian Chen, Ronak Pradeep, and Jimmy Lin. 2024. An Early FIRST Reproduc- tion and Improvements to Single-Token Decoding for Fast Listwise Reranking. arXiv:2411.05508(2024). SIGIR ’25, July 13–18, 2025, Padua, Italy Sahel Sharifymoghaddam et al

    2024 arXiv

  2. [10]

    Benjamin Clavié. 2024. Rerankers: A Lightweight Python Library to Unify Ranking Methods.arXiv:2408.17344(2024)

    2024 arXiv

  3. [11]

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2021. Overview of the TREC 2020 Deep Learning Track.arXiv:2102.07662(2021)

    2021 arXiv

  4. [12]

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Jimmy Lin

  5. [13]

    Voorhees, and Ian Soboroff

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, Jimmy Lin, Ellen M. Voorhees, and Ian Soboroff. 2023. Overview of the TREC 2022 Deep Learning Track. InProceedings of the Thirty-First Text REtrieval Conference (TREC 2022)

    2023

  6. [14]

    Voorhees

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. 2020. Overview of the TREC 2019 Deep Learning Track. arXiv:2003.07820(2020)

    2020 arXiv

  7. [15]

    Hall, and Ming-Wei Chang

    Zhuyun Dai, Vincent Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B. Hall, and Ming-Wei Chang. 2022. Promptagator: Few-shot Dense Retrieval From 8 Examples.arXiv:2209.11755(2022)

    2022 arXiv

  8. [16]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The LLaMA 3 Herd of Models.arXiv:2407.21783(2024)

    2024 arXiv

  9. [17]

    Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant

  10. [18]

    Sebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. 2021. Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling. InProceedings of the 44th Annual International ACM SIGIR Conference on Research and Development in ...

    2021

  11. [19]

    InProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2022)

    From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective. InProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2022). Madrid, Spain, 2353–2359

    2022

  12. [20]

    Bartoldson, Bhavya Kailkhura, Avi Schwarzschild, Aniruddha Saha, et al

    Neel Jain, Ping-yeh Chiang, Yuxin Wen, John Kirchenbauer, Hong-Min Chu, Gowthami Somepalli, Brian R. Bartoldson, Bhavya Kailkhura, Avi Schwarzschild, Aniruddha Saha, et al. 2023. NEFTune: Noisy Embeddings Improve Instruction Finetuning.arXiv:2310.05914(2023)

    2023 arXiv

  13. [21]

    Mathew Jacob, Erik Lindgren, Matei Zaharia, Michael Carbin, Omar Khattab, and Andrew Drozdov. 2024. Drowning in Documents: Consequences of Scaling Reranker Inference.arXiv:2411.11767(2024)

    2024 arXiv

  14. [22]

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open- Domain Question Answering. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)...

    2020

  15. [23]

    Can Jin, Hongwu Peng, Shiyu Zhao, Zhenting Wang, Wujiang Xu, Ligong Han, Jiahui Zhao, Kai Zhong, Sanguthevar Rajasekaran, and Dimitris N. Metaxas

  16. [24]

    Jimmy Lin. 2022. Building a Culture of Reproducibility in Academic Research. arXiv:2212.13534(2022)

    2022 arXiv

  17. [25]

    Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini: A Python Toolkit for Reproducible Informa- tion Retrieval Research with Sparse and Dense Representations. InProceedings of the 44th Annual International ACM SIGIR Con...

    2021

  18. [26]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with Paged-Attention. InProceedings of the 29th Symposium on Operating Systems Princip...

    2023

  19. [27]

    2022.LlamaIndex

    Jerry Liu. 2022.LlamaIndex

    2022

  20. [28]

    Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. 2024. Fine- tuning LLaMA for Multi-Stage Text Retrieval. InProceedings of the 47th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2024). 2421–2425

    2024

  21. [29]

    2021.Pretrained Transformers for Text Ranking: BERT and Beyond

    Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2021.Pretrained Transformers for Text Ranking: BERT and Beyond. Morgan & Claypool Publishers

    2021

  22. [30]

    Craig Macdonald and Nicola Tonellotto. 2020. Declarative Experimentation inInformation Retrieval using PyTerrier. InProceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval (ICTIR 2020)

    2020

  23. [31]

    Irina Matveeva, Chris Burges, Timo Burkard, Andy Laucius, and Leon Wong

  24. [32]

    Xueguang Ma, Xinyu Zhang, Ronak Pradeep, and Jimmy Lin. 2023. Zero-Shot Listwise Document Reranking with a Large Language Model.arXiv:2305.02156 (2023)

    2023 arXiv

  25. [33]

    Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage Re-ranking with BERT. arXiv:1901.04085(2019)

    2019 arXiv

  26. [34]

    Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Docu- ment Ranking with a Pretrained Sequence-to-Sequence Model. InFindings of the Association for Computational Linguistics: EMNLP 2020. 708–718

    2020

  27. [35]

    Ronak Pradeep, Haonan Chen, Lingwei Gu, Manveer Singh Tamber, and Jimmy Lin. 2023. PyGaggle: A Gaggle of Resources for Open-Domain Question Answer- ing. InAdvances in Information Retrieval: 45th European Conference on Information Retrieval (ECIR 2023), Part III. Dublin, Irelan...

    2023

  28. [36]

    Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, et al. 2022. Text and Code Embeddings by Contrastive Pre-Training.arXiv:2201.10005(2022)

    2022 arXiv

  29. [37]

    Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. 2023. RankVicuna: Zero-shot Listwise Document Reranking with Open-Source Large Language Models.arXiv:2309.15088(2023)

    2023 arXiv

  30. [38]

    Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. 2023. RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!arXiv:2312.02724 (2023)

    2023 arXiv

  31. [39]

    Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Ben- dersky. 2024. Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting. InFindings of the Association...

    2024

  32. [40]

    Ronak Pradeep, Rodrigo Nogueira, and Jimmy Lin. 2021. The Expando-Mono- Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models.arXiv:2101.05667(2021)

    2021 arXiv

  33. [41]

    Robertson and Hugo Zaragoza

    Stephen E. Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Beyond.Foundations and Trends in Information Retrieval 3, 4 (2009), 333–389

    2009

  34. [42]

    Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. 2023. Is ChatGPT Good at Search? In- vestigating Large Language Models as Re-Ranking Agents. InProceedings of the 2023 Conference on Empirical Methods in Natural Languag...

    2023

  35. [43]

    Manveer Singh Tamber, Ronak Pradeep, and Jimmy Lin. 2023. Scaling Down, LiT- ting Up: Efficient Zero-Shot Listwise Reranking with Seq2seq Encoder-Decoder Models.arXiv:2312.16098(2023)

    2023 arXiv

  36. [44]

    Revanth Gangi Reddy, JaeHyeok Doo, Yifei Xu, Md Arafat Sultan, Deevya Swain, Avirup Sil, and Heng Ji. 2024. FIRST: Faster Improved Listwise Reranking with Single Token Decoding. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Miami, Fl...

    2024

  37. [45]

    Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, Vol. 1

    2021

  38. [46]

    Jasper Xian, Saron Samuel, Faraz Khoubsirat, Ronak Pradeep, Md Arafat Sultan, Radu Florian, Salim Roukos, Avirup Sil, Christopher Potts, and Omar Khattab

  39. [47]

    Jasper Xian, Tommaso Teofili, Ronak Pradeep, and Jimmy Lin. 2024. Vector Search with OpenAI Embeddings: Lucene is All You Need. InProceedings of the 17th ACM International Conference on Web Search and Data Mining. New York, NY, USA, 1090–1093

    2024

  40. [48]

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking Multimodal Understanding Across Millions of Tokens of Context.arXiv:2403.05530(2024)

    2024 arXiv

  41. [49]

    Crystina Zhang, Sebastian Hofstätter, Patrick Lewis, Raphael Tang, and Jimmy Lin. 2025. Rank-Without-GPT: Building GPT-Independent Listwise Rerankers on Open-Source Large Language Models. InAdvances in Information Retrieval. Cham, 233–247

    2025

  42. [50]

    Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. 2021. Mr. TyDi: A Multi- lingual Benchmark for Dense Retrieval. InProceedings of the 1st Workshop on Multilingual Representation Learning. Punta Cana, Dominican Republic, 127–137

    2021

  43. [51]

    Prompts as Auto-Optimized Training Hyperparameters: Training Best-in- Class IR Models from Scratch with 10 Gold Labels.arXiv:2406.11706(2024)

    2024 arXiv

  44. [52]

    Honglei Zhuang, Zhen Qin, Rolf Jagerman, Kai Hui, Ji Ma, Jing Lu, Jianmo Ni, Xuanhui Wang, and Michael Bendersky. 2023. RankT5: Fine-tuning T5 for Text Ranking with Ranking Losses. InProceedings of the 46th International ACM SIGIR Conference on Research and Development in Info...

    2023

  45. [53]

    Peilin Yang, Hui Fang, and Jimmy Lin. 2017. Anserini: Enabling the Use of Lucene for Information Retrieval Research. InProceedings of the 40th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2017). Tokyo, Japan, 1253–1256

    2017

  46. [56]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Language Model Programs. InAdvances in Neural Informa...

    2024

  47. [2006]

    InProceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2006)

    High Accuracy Retrieval with Multiple Nested Ranker. InProceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2006). Seattle, Washington, 437–444

    2006

  48. [2022]

    InProceedings of the Thirtieth Text REtrieval Conference (TREC 2021)

    Overview of the TREC 2021 Deep Learning Track. InProceedings of the Thirtieth Text REtrieval Conference (TREC 2021)

    2021

  49. [2024]

    APEER: Automatic Prompt Engineering Enhances Large Language Model Reranking.arXiv:2406.14449(2024)

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.