Pith. sign in

REVIEW 6 cited by

Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.07712 v2 pith:NWXJW3XJ submitted 2023-10-11 cs.CL cs.LG

Raphael Tang , Xinyu Zhang , Xueguang Ma , Jimmy Lin , Ferhan Ture This is my paper · ORCID
classification cs.CLcs.LG
keywords rankingpromptlistself-consistencybiasimproveslanguagelarge
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Large language models (LLMs) exhibit positional bias in how they use context, which especially complicates listwise ranking. To address this, we propose permutation self-consistency, a form of self-consistency over ranking list outputs of black-box LLMs. Our key idea is to marginalize out different list orders in the prompt to produce an order-independent ranking with less positional bias. First, given some input prompt, we repeatedly shuffle the list in the prompt and pass it through the LLM while holding the instructions the same. Next, we aggregate the resulting sample of rankings by computing the central ranking closest in distance to all of them, marginalizing out prompt order biases in the process. Theoretically, we prove the robustness of our method, showing convergence to the true ranking in the presence of random perturbations. Empirically, on five list-ranking datasets in sorting and passage reranking, our approach improves scores from conventional inference by up to 7-18% for GPT-3.5 and 8-16% for LLaMA v2 (70B), surpassing the previous state of the art in passage reranking. Our code is at https://github.com/castorini/perm-sc.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Set-LLM: A Permutation-Invariant LLM

    cs.LG 2025-05 conditional novelty 7.0 of 10

    A set position encoding and set attention mask make decoder-only LLMs provably invariant to the ordering of options in a prompt.

  2. SGIC: A Self-Guided Iterative Calibration Framework for RAG

    cs.CL 2025-06 conditional novelty 6.0 of 10

    SGIC feeds a model's own uncertainty scores back into its prompt for several calibration rounds and improves RAG accuracy on HotpotQA, NQ, and GSM8K.

  3. RoToR: Towards More Reliable Responses for Order-Invariant Inputs

    cs.CL 2025-02 conditional novelty 6.0 of 10

    RoToR makes a frozen LLM order-invariant by circularly rotating a single global sort of segment position IDs, and Selective Routing combines it with the original model for mixed lists.

  4. PaSa: An LLM Agent for Comprehensive Academic Paper Search

    cs.IR 2025-01 conditional novelty 6.0 of 10

    PaSa, a two-agent LLM system trained with session-level RL, reports substantially higher recall than existing academic search baselines on complex paper-finding queries.

  5. Writing Style Matters: An Examination of Bias and Fairness in Information Retrieval Systems

    cs.IR 2024-11 conditional novelty 6.0 of 10

    Text embedding models used in search are biased by writing style: informal and emotive documents rank lower, and most models match the query style when retrieving.

  6. Lost in the Middle, and In-Between: Enhancing Language Models' Ability to Reason Over Long Contexts in Multi-Hop QA

    cs.CL 2024-12 conditional novelty 5.0 of 10

    In multi-hop QA, language models answer worse when the required evidence documents are separated by distractor documents, not just when the evidence sits in the middle of the context.

Pith tools