REVIEW 35 cited by
RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Researchers have successfully applied large language models (LLMs) such as ChatGPT to reranking in an information retrieval context, but to date, such work has mostly been built on proprietary models hidden behind opaque API endpoints. This approach yields experimental results that are not reproducible and non-deterministic, threatening the veracity of outcomes that build on such shaky foundations. To address this significant shortcoming, we present RankVicuna, the first fully open-source LLM capable of performing high-quality listwise reranking in a zero-shot setting. Experimental results on the TREC 2019 and 2020 Deep Learning Tracks show that we can achieve effectiveness comparable to zero-shot reranking with GPT-3.5 with a much smaller 7B parameter model, although our effectiveness remains slightly behind reranking with GPT-4. We hope our work provides the foundation for future research on reranking with modern LLMs. All the code necessary to reproduce our results is available at https://github.com/castorini/rank_llm.
Forward citations
Cited by 35 Pith papers
-
How Good are LLM-based Rerankers? An Empirical Analysis of State-of-the-Art Reranking Models
On a new benchmark of post-April 2025 queries, LLM rerankers show a 5-15% performance drop compared with familiar benchmarks, and lightweight models match them on efficiency and sometimes accuracy.
-
Structure-aware Relative Policy Optimization for Ranking
Dividing pairwise reward differences by top-weighted Kendall-tau distance in group-relative policy optimization improves listwise ranking performance.
-
TabRank: Chain-of-Thought Distillation for Table Re-Rankers
Conditioning a table reranker on teacher chain-of-thought traces during training, without supervising generation of the traces, improves out-of-distribution table retrieval accuracy over base, naive SFT, and standard ...
-
Trie-based Experiment Plans for Efficient IR Pipeline Experiments
Trie-based experiment plans reduce the duration of comparative evaluations of IR pipelines by 26% versus linear plans in a BM25-MonoT5-DuoT5 demonstration on MSMARCO v2.
-
KaLM-Reranker-V1: Fast but Not Late Interaction for Compressed Document Reranking
KaLM-Reranker-V1 uses encoder–decoder FBNL with Matryoshka pooling to match Qwen3-class reranking quality at substantially lower online cost.
-
Entities as Retrieval Signals: A Systematic Study of Coverage, Supervision, and Evaluation in Entity-Oriented Ranking
Entity signals cover only 19.7% of relevant documents on Robust04 and no configuration among 443 systems improves MAP by more than 0.05 in open-world evaluation, despite gains when entities are pre-restricted.
-
From Ranking to Selection: A Simple but Efficient Dynamic Passage Selector for Retrieval Augmented Generation
DPS is a supervised dynamic passage selector that improves RAG answer accuracy over fixed-K rerankers on five benchmarks.
-
Harnessing Pairwise Ranking Prompting Through Sample-Efficient Ranking Distillation
Pairwise Ranking Distillation trains a pointwise ranker from pairwise LLM teacher labels and matches full-pair distillation using just 2% of pairs on TREC-DL.
-
JointRank: Rank Large Set with Single Pass
JointRank partitions candidates into overlapping blocks, ranks each block in parallel with an LLM, and reconstructs a global ranking by aggregating the resulting pairwise comparisons.
-
Precise Zero-Shot Pointwise Ranking with LLMs through Post-Aggregated Global Context Information
A summary-based anchor document enables contrastive pointwise scoring that, when averaged with ordinary pointwise scores, improves zero-shot LLM reranking.
-
SQLens: An End-to-End Framework for Error Detection and Correction in Text-to-SQL
SQLens uses weak supervision over database and LLM error signals to detect clause-level semantic errors in text-to-SQL and fixes them iteratively, improving execution accuracy on BIRD and Spider.
-
Reinforcement Speculative Decoding for Fast Ranking
RSD uses reinforcement learning to train an agent that iteratively modifies an LLM's ranking under a fixed call budget, outperforming single-token and speculative-decoding baselines on IR and RS datasets.
-
Matryoshka Re-Ranker: A Flexible Re-Ranking Architecture With Configurable Depth and Width
A single LLM re-ranker can be configured at runtime to different depths and widths, with training tricks that keep compressed variants close to full-scale accuracy.
-
PaSa: An LLM Agent for Comprehensive Academic Paper Search
PaSa, a two-agent LLM system trained with session-level RL, reports substantially higher recall than existing academic search baselines on complex paper-finding queries.
-
Guiding Retrieval using LLM-based Listwise Rankers
SlideGar adapts graph-based adaptive retrieval to listwise LLM rerankers, improving nDCG@10 by up to 13.23% and recall by up to 28.02% without extra LLM inferences.
-
Sliding Windows Are Not the End: Exploring Full Ranking with Long-Context Large Language Models
With a multi-pass teacher label construction and a position-weighted loss, a fine-tuned long-context LLM can rank 100 passages in one pass, beating sliding-window rerankers in both NDCG@10 and latency.
-
LitLLMs, LLMs for Literature Review: Are we there yet?
LLMs can draft plausible related-work sections when the task is decomposed into keyword-plus-embedding retrieval, attribution-verified reranking, and plan-based generation, but retrieval coverage remains below 10 perc...
-
Geo-Spatial Concept Probing of Large Language Models: Abstraction, Compositionality, and Grounding
A new concept-centric GeoQA benchmark probes LLMs on abstraction, compositionality, and grounding of direction, distance, and topology, finding good abstraction, weak grounding, and a Mistral-family deficit.
-
Scaling Search Relevance: Augmenting App Store Ranking with LLM-Generated Judgments
Using a fine-tuned 3B LLM to generate millions of textual relevance labels for App Store search improves the ranker's behavioral/textual Pareto frontier and lifts conversion by 0.24%.
-
Beyond Sequential Reranking: Reranker-Guided Search Improves Reasoning Intensive Retrieval
Reranker-Guided-Search, a greedy graph search steered by reranker scores, outperforms sequential top-k reranking under a fixed budget on three reasoning-intensive retrieval benchmarks.
-
Leveraging LLMs to Evaluate Usefulness of Document
A cascade of LLM judges, fed with search context and behavior, produces multilevel usefulness labels for clicked documents and improves search satisfaction prediction.
-
K-order Ranking Preference Optimization for Large Language Models
KPO extends the Plackett-Luce preference model used in DPO to top-K partial rankings, with query-adaptive K and curriculum learning, and reports improved LLM ranking accuracy.
-
Iterative Self-Incentivization Empowers Large Language Models as Agentic Searchers
EXSEARCH trains LLMs for agentic search by treating search trajectories as latent variables and optimizing a weighted likelihood via expectation-maximization, yielding gains on NQ, HotpotQA, MuSiQue, and 2WikiQA.
-
REARANK: Reasoning Re-ranking Agent via Reinforcement Learning
Training a listwise reranker with reinforcement learning and explicit reasoning on only 179 annotated queries yields reranking quality comparable to GPT-4.
-
RankLLM: A Python Package for Reranking with LLMs
RankLLM is an open-source Python package that modularly supports pointwise, pairwise, and listwise LLM rerankers, with integrated retrieval, evaluation, training, and response analysis, and reproduces results from Ran...
-
Rankify: A Comprehensive Python Toolkit for Retrieval, Re-Ranking, and Retrieval-Augmented Generation
The paper presents Rankify, a modular open-source toolkit that unifies retrieval, re-ranking, and RAG with 40 pre-retrieved datasets and benchmark results.
-
ASRank: Zero-Shot Re-Ranking with Answer Scent for Document Retrieval
ASRank re-ranks retrieved documents by scoring how well each document supports a zero-shot answer scent generated by a large LLM, beating UPR and RankGPT on several QA datasets.
-
ListConRanker: A Contrastive Text Reranker with Listwise Encoding
ListConRanker combines listwise attention over passage embeddings with Circle Loss to set a new mAP average on the C-MTEB reranking benchmark.
-
ConQRet: Benchmarking Fine-Grained Evaluation of Retrieval Augmented Argumentation with LLM Judges
ConQRet is a new benchmark plus LLM judge prompts for evaluating retrieval-augmented argumentation, with fine-grained document-level scoring that tracks injected irrelevant context better than single-score baselines.
-
MagicSelector: Joint Optimization for Agent Tool Selection via Counterfactual Decomposition and Progressive Reranking
MagicSelector combines RL-trained task decomposition, iterative hard-negative reranking, and adaptive top-k truncation to improve mobile-agent tool retrieval, evaluated on a new MTDTool benchmark.
-
InfluMatch: Frontier-Quality KOL Search at 4B-Model Cost
A 4B-model cascade for Thai KOL matching reaches 94.1% P@5 on 11 queries, matching a frontier model, with pairwise SimPO training transferring end-to-end while pointwise SFT+GRPO does not.
-
SciClaimSeekers at CheckThat! 2026: Retrieving Scientific Sources for Social Media Claims with LLM Reranking
A zero-shot Qwen2.5-14B reranker on top of BM25+E5 hybrid retrieval reaches 64.39 MRR@5 on CLEF-2026 CheckThat! Task 1 English scientific source retrieval, with the LLM contributing most of the gain.
-
Beyond Retrieval: Ensembling Cross-Encoders and GPT Rerankers with LLMs for Biomedical QA
An ensemble of finetuned cross-encoders and GPT-4o rerankers over dense-vector retrieval achieved rank 10 in BioASQ 2025 Phase A retrieval and rank 1 on factoid answers.
-
LineRetriever: Planning-Aware Observation Reduction for Web Agents
LineRetriever uses a small LM to select relevant lines from web page observations, cutting context by up to 73% with only small success-rate drops on web agent benchmarks.
-
DynRank: Improving Passage Retrieval with Dynamic Zero-Shot Prompting Based on Question Classification
DynRank conditions UPR-style passage reranking on an automatically inferred fine-grained question type and reports small gains over static prompting on NQ, TriviaQA, WebQuestions, and BEIR.
Discussion (0). Continue with ORCID to comment.