Pith. sign in

REVIEW 24 cited by

Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2304.09542 v3 pith:LQC67RXY submitted 2023-04-19 cs.CL cs.IR

Weiwei Sun , Lingyong Yan , Xinyu Ma , Shuaiqiang Wang , Pengjie Ren , Zhumin Chen , Dawei Yin , Zhaochun Ren This is my paper
classification cs.CLcs.IR
keywords llmsrankingchatgptmodelmodelsresultsabilitygenerative
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large Language Models (LLMs) have demonstrated remarkable zero-shot generalization across various language-related tasks, including search engines. However, existing work utilizes the generative ability of LLMs for Information Retrieval (IR) rather than direct passage ranking. The discrepancy between the pre-training objectives of LLMs and the ranking objective poses another challenge. In this paper, we first investigate generative LLMs such as ChatGPT and GPT-4 for relevance ranking in IR. Surprisingly, our experiments reveal that properly instructed LLMs can deliver competitive, even superior results to state-of-the-art supervised methods on popular IR benchmarks. Furthermore, to address concerns about data contamination of LLMs, we collect a new test set called NovelEval, based on the latest knowledge and aiming to verify the model's ability to rank unknown knowledge. Finally, to improve efficiency in real-world applications, we delve into the potential for distilling the ranking capabilities of ChatGPT into small specialized models using a permutation distillation scheme. Our evaluation results turn out that a distilled 440M model outperforms a 3B supervised model on the BEIR benchmark. The code to reproduce our results is available at www.github.com/sunnweiwei/RankGPT.

Discussion (0). Sign in to comment.

Forward citations

Cited by 24 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. How Good are LLM-based Rerankers? An Empirical Analysis of State-of-the-Art Reranking Models

    cs.CL 2025-08 conditional novelty 7.0 of 10

    On a new benchmark of post-April 2025 queries, LLM rerankers show a 5-15% performance drop compared with familiar benchmarks, and lightweight models match them on efficiency and sometimes accuracy.

  2. Frustratingly Simple Retrieval Improves Challenging, Reasoning-Intensive Benchmarks

    cs.CL 2025-07 conditional novelty 7.0 of 10

    A compact, diverse, web-scale datastore plus simple dense retrieval improves performance on MMLU, MMLU Pro, AGI Eval, GPQA, and MATH across multiple LLM families.

  3. Optimizing RAG Rerankers with LLM Feedback via Reinforcement Learning

    cs.CL 2026-04 conditional novelty 6.0 of 10

    RRPO formulates document reranking as a sequential MDP and optimizes a pointwise reranker with PPO using LLM generation rewards and a reference-anchored deterministic baseline.

  4. LLM-guided Hierarchical Search for End-to-end Reasoning Intensive Retrieval

    cs.IR 2025-10 conditional novelty 6.0 of 10

    An LLM directly traverses a hierarchical semantic index of a corpus, using calibrated path-relevance scores to retrieve documents for reasoning-intensive queries.

  5. Characterizing Web Search in The Age of Generative AI

    cs.IR 2025-10 conditional novelty 6.0 of 10

    AI search engines vary greatly in how much they rely on web pages versus internal model knowledge, and these differences shift which sources and concepts users see.

  6. Are LLMs Reliable Rankers? Rank Manipulation via Two-Stage Token Optimization

    cs.CL 2025-10 conditional novelty 6.0 of 10

    RAF, a two-stage token-optimization attack, creates brief natural-sounding text injections that reliably boost a target item's rank in LLM reranking outputs, beating state-of-the-art baselines in effectiveness, stealt...

  7. LLMs for estimating positional bias in logged interaction data

    cs.IR 2025-09 conditional novelty 6.0 of 10

    An LLM-as-a-judge relevance score lets the authors estimate examination propensities from logged clicks, exposing row-column effects in a grid layout and giving an IPS-trained reranker a roughly 2% wNDCG@10 gain.

  8. ERank: Fusing Supervised Fine-Tuning and Reinforcement Learning for Effective and Efficient Text Reranking

    cs.IR 2025-08 conditional novelty 6.0 of 10

    ERank trains a pointwise reasoning-LLM reranker with fine-grained integer scoring and a listwise reinforcement-learning reward, reporting nDCG@10 of 40.2 on BRIGHT for its 32B variant with BM25 hybrid scoring.

  9. LLMCup: Ranking-Enhanced Comment Updating with LLMs

    cs.SE 2025-07 conditional novelty 6.0 of 10

    LLMCup combines multi-prompt LLM comment generation with a learned ranker, achieving 0.383 exact-match accuracy on the CUP test set versus 0.257 for HebCup and 0.177 for CUP.

  10. Shifting from Ranking to Set Selection for Retrieval Augmented Generation

    cs.CL 2025-07 conditional novelty 6.0 of 10

    SETR identifies a query's information requirements with chain-of-thought reasoning and selects a compact passage set, improving multi-hop RAG accuracy over fixed-top-k reranking baselines.

  11. Harnessing Pairwise Ranking Prompting Through Sample-Efficient Ranking Distillation

    cs.IR 2025-07 conditional novelty 6.0 of 10

    Pairwise Ranking Distillation trains a pointwise ranker from pairwise LLM teacher labels and matches full-pair distillation using just 2% of pairs on TREC-DL.

  12. SGIC: A Self-Guided Iterative Calibration Framework for RAG

    cs.CL 2025-06 conditional novelty 6.0 of 10

    SGIC feeds a model's own uncertainty scores back into its prompt for several calibration rounds and improves RAG accuracy on HotpotQA, NQ, and GSM8K.

  13. MM-R5: MultiModal Reasoning-Enhanced ReRanker via Reinforcement Learning for Document Retrieval

    cs.AI 2025-06 conditional novelty 6.0 of 10

    MM-R5, a 7B multimodal re-ranker trained with SFT and GRPO, achieves state-of-the-art page-level recall on MMDocIR by generating per-page reasoning chains.

  14. Towards a Relevance Posterior in Neural Information Access

    cs.IR 2026-07 conditional novelty 5.0 of 10

    Fusing a cached query-independent document-quality prior with BM25 or re-rankers improves TREC DL nDCG, with largest gains for RankZephyr, under a prior–likelihood view of retrieval.

  15. Beyond Sequential Reranking: Reranker-Guided Search Improves Reasoning Intensive Retrieval

    cs.IR 2025-09 conditional novelty 5.0 of 10

    Reranker-Guided-Search, a greedy graph search steered by reranker scores, outperforms sequential top-k reranking under a fixed budget on three reasoning-intensive retrieval benchmarks.

  16. TrackRec: Iterative Alternating Feedback with Chain-of-Thought via Preference Alignment for Recommendation

    cs.IR 2025-08 conditional novelty 5.0 of 10

    TrackRec trains a small LLM to generate user-preference summaries and a validator to score them, alternating the training so each improves the other, and reports gains on public and industrial recommendation benchmarks.

  17. Does UMBRELA Work on Other LLMs?

    cs.IR 2025-07 conditional novelty 5.0 of 10

    UMBRELA relevance judgments made with DeepSeek V3 are close to GPT-4o, and even small models preserve leaderboard rankings although per-document agreement with humans drops.

  18. FlexRAG: A Flexible and Comprehensive Framework for Retrieval-Augmented Generation

    cs.CL 2025-06 conditional novelty 5.0 of 10

    FlexRAG is a modular, open-source RAG framework with text, multimodal, and web retrieval, plus evaluation tools and efficient memory-mapped indexing.

  19. MagicSelector: Joint Optimization for Agent Tool Selection via Counterfactual Decomposition and Progressive Reranking

    cs.IR 2026-07 conditional novelty 4.0 of 10

    MagicSelector combines RL-trained task decomposition, iterative hard-negative reranking, and adaptive top-k truncation to improve mobile-agent tool retrieval, evaluated on a new MTDTool benchmark.

  20. InfluMatch: Frontier-Quality KOL Search at 4B-Model Cost

    cs.CL 2026-07 conditional novelty 4.0 of 10

    A 4B-model cascade for Thai KOL matching reaches 94.1% P@5 on 11 queries, matching a frontier model, with pairwise SimPO training transferring end-to-end while pointwise SFT+GRPO does not.

  21. SciClaimSeekers at CheckThat! 2026: Retrieving Scientific Sources for Social Media Claims with LLM Reranking

    cs.IR 2026-07 conditional novelty 4.0 of 10

    A zero-shot Qwen2.5-14B reranker on top of BM25+E5 hybrid retrieval reaches 64.39 MRR@5 on CLEF-2026 CheckThat! Task 1 English scientific source retrieval, with the LLM contributing most of the gain.

  22. LineRetriever: Planning-Aware Observation Reduction for Web Agents

    cs.CL 2025-06 conditional novelty 4.0 of 10

    LineRetriever uses a small LM to select relevant lines from web page observations, cutting context by up to 73% with only small success-rate drops on web agent benchmarks.

  23. Context-Aware Scientific Knowledge Extraction on Linked Open Data using Large Language Models

    cs.IR 2025-06 reject novelty 4.0 of 10

    WISE combines LLM filtering, word-overlap scoring, and adaptive stopping in a recursive tree search to extract and synthesize knowledge from linked web sources, reporting higher recall and detail than general LLM base...

  24. Evaluating Hybrid Retrieval Augmented Generation using Dynamic Test Sets: LiveRAG Challenge

    cs.CL 2025-06 conditional novelty 3.0 of 10

    A hybrid BM25-plus-dense RAG system finished 4th in faithfulness and 11th in correctness at LiveRAG 2025, and its analysis found vocabulary alignment between questions and documents to be the most important performanc...

Pith tools