REVIEW 7 cited by
Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labels
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Zero-shot text rankers powered by recent LLMs achieve remarkable ranking performance by simply prompting. Existing prompts for pointwise LLM rankers mostly ask the model to choose from binary relevance labels like "Yes" and "No". However, the lack of intermediate relevance label options may cause the LLM to provide noisy or biased answers for documents that are partially relevant to the query. We propose to incorporate fine-grained relevance labels into the prompt for LLM rankers, enabling them to better differentiate among documents with different levels of relevance to the query and thus derive a more accurate ranking. We study two variants of the prompt template, coupled with different numbers of relevance levels. Our experiments on 8 BEIR data sets show that adding fine-grained relevance labels significantly improves the performance of LLM rankers.
Forward citations
Cited by 7 Pith papers
-
How Good are LLM-based Rerankers? An Empirical Analysis of State-of-the-Art Reranking Models
On a new benchmark of post-April 2025 queries, LLM rerankers show a 5-15% performance drop compared with familiar benchmarks, and lightweight models match them on efficiency and sometimes accuracy.
-
Are LLMs Reliable Rankers? Rank Manipulation via Two-Stage Token Optimization
RAF, a two-stage token-optimization attack, creates brief natural-sounding text injections that reliably boost a target item's rank in LLM reranking outputs, beating state-of-the-art baselines in effectiveness, stealt...
-
Harnessing Pairwise Ranking Prompting Through Sample-Efficient Ranking Distillation
Pairwise Ranking Distillation trains a pointwise ranker from pairwise LLM teacher labels and matches full-pair distillation using just 2% of pairs on TREC-DL.
-
JointRank: Rank Large Set with Single Pass
JointRank partitions candidates into overlapping blocks, ranks each block in parallel with an LLM, and reconstructs a global ranking by aggregating the resulting pairwise comparisons.
-
MM-R5: MultiModal Reasoning-Enhanced ReRanker via Reinforcement Learning for Document Retrieval
MM-R5, a 7B multimodal re-ranker trained with SFT and GRPO, achieves state-of-the-art page-level recall on MMDocIR by generating per-page reasoning chains.
-
Scaling Search Relevance: Augmenting App Store Ranking with LLM-Generated Judgments
Using a fine-tuned 3B LLM to generate millions of textual relevance labels for App Store search improves the ranker's behavioral/textual Pareto frontier and lifts conversion by 0.24%.
-
Leveraging LLMs to Evaluate Usefulness of Document
A cascade of LLM judges, fed with search context and behavior, produces multilevel usefulness labels for clicked documents and improves search satisfaction prediction.
Discussion (0). Sign in to comment.