Pith. sign in

REVIEW 1 major objections 7 minor 1 cited by

Likert or Not: LLM Absolute Relevance Judgments on Fine-Grained Ordinal Scales

T0 review · 1 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that the gap between pointwise scoring and listwise ranking closes when pointwise scoring uses an 11-point ordinal relevance scale, making the difference statistically insignificant for most LLM-benchmark combinations.

desk verdict A useful, mostly credible empirical challenge to the 'listwise is always better' consensus; the headline significance counts rest on bootstrap CIs that ignore acknowledged LLM nondeterminism, but the core pattern is likely robust and the paper deserves serious review. read the letter →

arxiv 2505.19334 v1 pith:JDQSDU32 submitted 2025-05-25 cs.LG cs.IR

Charles Godfrey , Ping Nie , Natalia Ostapuk , David Ken , Shang Gao , Souheil Inati This is my paper
classification cs.LGcs.IR
keywords LLMrelevancejudgmentspointwisescoringlistwiserankingordinalscalesNDCG@10zero-shotstatisticalsignificancelabels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the widely held view in information retrieval—that large language models are better at making relative relevance judgments (ranking a list of documents) than absolute ones (scoring one document)—rests on comparisons that stacked the deck against pointwise scoring. By scoring each query–document pair on an 11-point ordinal relevance scale with a written semantic rubric for every level, the authors find that pointwise scoring matches listwise ranking within statistical error on most of the 40 model-dataset combinations they test. Listwise ranking significantly improves NDCG@10 in only 9 cases, all on the DBPedia, SciFact, and News datasets, and the result depends more on the dataset than on the LLM or the sorting algorithm. The finding matters because pointwise scoring is simpler, fully parallel, cheaper in tokens, and directly produces the per-document relevance labels needed to evaluate retrieval systems.

What carries the argument

The load-bearing object is the fine-grained ordinal relevance label space: an 11-point (0–10) scale in which each integer carries a written semantic description of the corresponding relevance level, e.g. '5: Average match, partially relevant to the query' up to '10: Perfect match.' Pointwise scoring asks the LLM to map a single query–document pair to one of these integers; listwise ranking asks for a permutation of a window of documents, bootstrapped to 100-document lists by sliding-window bubble sort or a multi-pivot quicksort with telescoping truncations. The comparison machinery is a bootstrap procedure that resamples the query set to place 95% confidence intervals on the difference in NDCG@10 between two methods, treating queries as IID draws. The scale-size effect does the explanatory work: NDCG@10 rises roughly monotonically as the number of bins grows from 2 to 11, which narrows the gap to listwise ranking to the point that most differences are statistically indistinguishable.

What would settle it

A reader could rerun the comparison with confidence intervals that also resample over repeated LLM generations (for example, several temperature-0 calls per query–document pair). If the set of model–dataset combinations on which listwise ranking significantly beats 11-point pointwise scoring—currently 9 of 40 on public benchmarks—changes materially once generation stochasticity is included, the paper's central statistical claim would be overturned; a sharper version would check whether the 95% interval for the NDCG@10 gain on those 9 combinations still excludes zero.

Watch

Extended reading notes

Core claim

The paper's central claim is that the gap between pointwise scoring and listwise ranking, which prior work took as evidence for the 'LLM relative relevance judgments hypothesis,' becomes statistically insignificant when pointwise scoring uses a sufficiently large ordinal label space. On ten public benchmarks and four LLMs, an 11-point (0–10) relevance rubric yields NDCG@10 differences from bubble-sort and multi-pivot quicksort listwise ranking whose 95% bootstrap confidence intervals include zero for 31 of 40 model-dataset combinations; the 9 significant wins are confined to DBPedia, SciFact, and News, and depend more on the dataset than on the model or listwise implementation. The paper also claims that upgrading pure permutation-generation prompts to emit per-document scores alongside ranks does not significantly change ranking quality, and that as relevance classifiers listwise methods significantly decrease AUROC/AUPRC more often than they significantly increase them relative to pointwise scoring. On two proprietary tax-and-accounting datasets with relevance labels collected after all models' training cut-offs, listwise ranking significantly outperforms pointwise scoring on every model-dataset combination, although for most models the absolute advantage is slim.

Load-bearing premise

The load-bearing premise is that the bootstrap confidence intervals are valid: the paper treats queries as independent, identically distributed draws and does not account for the acknowledged non-determinism of LLM outputs at temperature 0, so if those intervals are too narrow or too wide, the count of 'significant' model-dataset combinations would change.

Editorial extensions

If this is right

  • Practitioners can replace listwise re-ranking pipelines with embarrassingly parallel pointwise scoring on an 11-point scale, cutting token usage by roughly a factor of 3.4 compared with the listwise implementations studied.
  • The 'LLM relative relevance judgments hypothesis' needs qualification: relative comparisons are not universally better, and the datasets where they win (DBPedia, SciFact, News) are identifiable in advance.
  • Because ranking+scoring preserves ranking quality while emitting per-document labels, listwise prompts can double as label generators for training data or retrieval-system evaluation.
  • On label-quality metrics (AUROC, AUPRC), listwise methods more often significantly hurt than help relative to pointwise scoring, so conclusions about LLM relevance judgment quality depend heavily on the chosen metric.
  • On genuinely post-training-cutoff data, listwise ranking remains significantly ahead on every model-dataset combination, so the practical value of relative judgments does not disappear on novel distributions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension would re-run the comparison with confidence intervals that also resample over repeated LLM generations; the paper's own footnote acknowledges non-determinism at temperature 0, and such intervals could shift the reported 9-of-40 count.
  • The dataset-dependence of the listwise advantage suggests a testable prediction: datasets with many partially relevant or near-duplicate documents (News, SciFact) reward relative comparisons, while datasets with clear single-answer relevance (Covid) do not; the property could be measured and used to choose the method.
  • The plateau at 11 bins observed on public benchmarks may not transfer to novel domains, since the proprietary tax datasets still favor listwise ranking; searching over the number of bins per domain is a natural follow-up the paper leaves implicit.
  • If pointwise scoring with large ordinal scales is accepted as competitive, LLM-as-judge evaluation of search systems can be simplified and parallelized; the paper's proprietary results, however, imply that on genuinely novel corpora listwise labels may still be worth the extra inference cost.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 7 minor

Summary. The paper studies whether listwise ranking (permutation generation) is truly superior to pointwise scoring for zero-shot LLM relevance judgments. Using four LLMs (Claude 3.5 Sonnet, GPT-4 Omni, Gemini 1.5 Pro, Llama 3.1 405b) and ten datasets (eight from BEIR/TREC-DL plus two proprietary tax/accounting datasets), the authors compare pointwise scoring on ordinal scales of size 2 to 11, bubble-sort and multi-pivot quicksort listwise ranking, and simultaneous ranking-plus-scoring. The main finding is that with an 11-point scale, pointwise scoring is not significantly outperformed by listwise ranking in NDCG@10 for 31 of 40 model-dataset combinations, and significant listwise advantages are confined to DBPedia, SciFact, and News. The paper also reports that listwise ranking-plus-scoring frequently decreases AUPRC and AUROC relative to pointwise scoring. The authors argue that prior work underestimated pointwise scoring by using binary labels.

Significance. If the result holds, it challenges the prevailing community hypothesis that LLMs are better at relative than absolute relevance judgments, with practical implications: pointwise scoring is embarrassingly parallel and consumes fewer tokens than the listwise baselines considered. The evaluation is broad, includes an explicit attempt to give listwise methods a favorable implementation, and addresses data contamination with novel post-cut-off datasets. The paper's main quantitative claims are supported by bootstrap confidence intervals, and the authors are careful to state they do not refute the relative-judgment hypothesis. However, the lack of code/data release and the acknowledged unquantified randomness in the significance analysis are important caveats.

major comments (1)
  1. [Section 2.3, footnote 4, Appendix E (Algorithm 2)] The bootstrap confidence intervals in Section 2.3 resample queries only, treating each per-query NDCG@10 difference as a fixed value. This ignores two acknowledged sources of within-query randomness: LLM output nondeterminism at zero temperature (footnote 4) and the random pivot selection and random batching in the quicksort variant (Algorithm 2 in Appendix E). Because the paper's headline claims, such as the 9/40 count of significant improvements and the statement that the pointwise-listwise gap is 'statistically insignificant for many combinations', are statements about statistical significance, the CIs should incorporate both query-level and within-query variability, for example through a two-level bootstrap over queries and repeated LLM or algorithm runs per query. Without this, the reported CIs may be too narrow and the significance counts could change materially. This issue is load-bearing for the paper's central claim rather than a presentation detail.
minor comments (7)
  1. [Abstract / Section 4.1 / Appendix C] The abstract states that both proprietary datasets have 'relevance labels collected after the training cut-off of all LLMs evaluated', but Appendix C reports that full-document annotation was completed in March 2024, before Claude 3.5 Sonnet's April 2024 cut-off (Table 2). Only the passage-level labels (May 2024) are after all cut-offs. Please correct this overstatement and note the implication for the data-pollution argument on the full-document dataset.
  2. [Section 6] The sentence 'See ??foramoredetailedcomparison' contains a broken cross-reference and should be fixed.
  3. [Appendix B, Table 9] Table 9 is titled 'Difference in AUPRC for pointwise scoring and listwise ranking on Tax&Accounting datasets', but the values listed are identical to the absolute AUROC values in Table 8 and are not differences; the table should be corrected or removed.
  4. [Section 2.3] The paper does not state the number of bootstrap replications B used for the confidence intervals; please report this value (for instance, the n_boot setting passed to seaborn).
  5. [Section 5.1] The tie-breaking rule for pointwise scores is not specified; since 0-10 labels are coarse, ties are likely, and the ordering of tied documents (for example, stable by BM25 score) should be documented because it can affect NDCG@10.
  6. [General] Releasing the run files and prompts for the public benchmarks would substantially improve reproducibility; the proprietary datasets cannot be released, but the public-benchmark results could be shared.
  7. [General] There are minor typographical errors, such as 'expere ments' in Section 2.1 and 'adherance' in Section 3, that should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: this is an empirical comparison with no fitted constants, no load-bearing self-citations, and no definitionally forced predictions.

full rationale

This paper is an empirical comparison, not a derivation, and no step in the paper makes a claimed result equivalent to its inputs by construction. Pointwise scoring, bubble-sort ranking, quicksort ranking, and ranking+scoring are defined independently (Sections 2.1-2.2 and Appendix E), and the headline claim (9/40 model-dataset combinations with a significant NDCG@10 improvement for listwise ranking, Fig. 2) is a measurement over external benchmarks plus two novel post-cutoff datasets. There are no fitted constants and no parameters estimated from the target metric. The bootstrap procedure in Section 2.3 resamples queries to estimate confidence intervals, but that is an inference technique, not a definitional link between input and output; the acknowledged nondeterminism of LLM outputs (footnote 4) and random pivot/batch selection in Algorithm 2 are threats to the validity of those intervals, not circularity. There are no load-bearing self-citations: citations to Sun et al., Ma et al., Zhuang et al., and Guo et al. are external prior work whose choices (e.g., bubble-sort window size 20, overlap 10) are adopted as experimental settings, not as mathematical premises that force the conclusion. The data-pollution limitation (footnote 6) is an external-validity concern and is explicitly addressed with two post-training-cutoff datasets. Accordingly the correct circularity score is 0.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim (11-bin pointwise scoring is competitive with listwise ranking) rests on the listed hyperparameters, the IID query assumption for the bootstrap, the binarization of ground truth for AUPRC/AUROC, and the representativeness of the two listwise algorithms. No new entities are introduced. The relevance rubrics themselves are LLM-generated once per scale, a non-released prompt artifact.

free parameters (6)
  • listwise window size W = 20
    Chosen following prior work (Sun et al. 2023); held fixed across all experiments.
  • sliding window overlap V (bubble sort) = 10
    Chosen following prior work; part of the listwise method configuration.
  • number of pivots P (quicksort) = 10
    Set equal to the bubble-sort overlap so computational costs are comparable; a hand-chosen hyperparameter.
  • telescoping truncation thresholds T1, T2 = 50, 20
    Chosen to favor listwise ranking ('err on the side of over-optimizing listwise'), stated in Appendix E.
  • relevance scale size R = 10 (0-10 scale)
    The central variable of the study; using 11 bins is the paper's main intervention, motivated by Guo et al. 2024.
  • number of BM25 candidates per query = 100
    Top 100 BM25 hits, following prior work; documents truncated to 300 words.
assumptions (5)
  • domain assumption Queries are independent and identically distributed within each benchmark dataset
    Section 2.3 states the IID query assumption as the basis for bootstrap confidence intervals. This is a standard but approximate modelling assumption in IR evaluation.
  • domain assumption LLM outputs at temperature 0 are treated as deterministic for the purpose of significance testing
    Footnote 4 in Section 2.3 acknowledges that LLM outputs remain somewhat non-deterministic but the paper does not quantify this source of randomness in the bootstrap CIs.
  • domain assumption Ground truth labels can be binarized at the documented thresholds (>=2 for MS MARCO-derived, >=1 for others) for AUPRC/AUROC computation
    Section 5.1 describes the binarization; this is a modeling choice that could affect label-quality metrics.
  • domain assumption Truncating documents to 300 words preserves enough relevance signal for fair comparison
    Section 5 states documents and passages are truncated to first 300 words following Sun et al. 2023; this could differentially affect pointwise vs listwise prompting.
  • domain assumption The two listwise algorithms (bubble and multi-pivot quicksort with telescoping) are representative of the state of the art in LLM listwise ranking
    Section 2.2 and Appendix A justify the choice via desiderata D1-D6, but methods requiring logit access (e.g., setwise ranking) are excluded, so the comparison may not generalize to all listwise approaches.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Likert or Not: LLM Absolute Relevance Judgments on Fine-Grained Ordinal Scales." pith.science (2026). https://pith.science/paper/JDQSDU32

@misc{pith2026250519334,
  author       = {Pith},
  title        = {Pith review of: Likert or Not: LLM Absolute Relevance Judgments on Fine-Grained Ordinal Scales},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JDQSDU32}},
  note         = {Machine review of arXiv:2505.19334}
}
read the original abstract

Large language models (LLMs) obtain state of the art zero shot relevance ranking performance on a variety of information retrieval tasks. The two most common prompts to elicit LLM relevance judgments are pointwise scoring (a.k.a. relevance generation), where the LLM sees a single query-document pair and outputs a single relevance score, and listwise ranking (a.k.a. permutation generation), where the LLM sees a query and a list of documents and outputs a permutation, sorting the documents in decreasing order of relevance. The current research community consensus is that listwise ranking yields superior performance, and significant research effort has been devoted to crafting LLM listwise ranking algorithms. The underlying hypothesis is that LLMs are better at making relative relevance judgments than absolute ones. In tension with this hypothesis, we find that the gap between pointwise scoring and listwise ranking shrinks when pointwise scoring is implemented using a sufficiently large ordinal relevance label space, becoming statistically insignificant for many LLM-benchmark dataset combinations (where ``significant'' means ``95\% confidence that listwise ranking improves NDCG@10''). Our evaluations span four LLMs, eight benchmark datasets from the BEIR and TREC-DL suites, and two proprietary datasets with relevance labels collected after the training cut-off of all LLMs evaluated.

Figures

Figures reproduced from arXiv: 2505.19334 by the authors.

Figure 1
Figure 1. Comparison of Pointwise Scoring (left), Ranking (middle) and Scoring and Ranking (right). Here [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Observed differences in NDCG@10 for point [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Comparing relevance classification performance of pointwise scoring vs. simultaneous ranking+scoring [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Observed differences in NDCG@10 for rank [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Observed differences in NDCG@10 for pointwise scoring and listwise ranking methods on Tax&Accounting {Full Document, Passage} datasets. Bootstrap 95% confidence intervals (see section 2.3). For absolute NDCG@10 see table 5. dependent than model dependent – for example,…
Figure 6
Figure 6. Figure 6: Change in NDCG@10 for pointwise scoring on ordinal relevance scales of varying size and listwise [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Difference in area under curve metrics for pointwise scoring and listwise ranking on Tax&Accounting [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Linguistic Nepotism: Trading-off Quality for Language Preference in Multilingual RAG

    cs.CL 2025-09 conditional novelty 6.0 of 10

    In multilingual retrieval-augmented generation, models cite English evidence more accurately than translated evidence, and this language preference can outweigh document relevance.

Reference graph

Works this paper leans on

22 extracted references · 14 canonical work pages · cited by 1 Pith paper

  1. [1]

    Claude 3.5 Sonnet Model Card Addendum

    Anthropic. Claude 3.5 Sonnet Model Card Addendum . Mar. 2024

    work page 2024

  2. [2]

    Y. Chen, Q. Liu, Y. Zhang, W. Sun, D. Shi, J. Mao, and D. Yin. TourRank : Utilizing Large Language Models for Documents Ranking with a Tournament-Inspired Strategy . 2024. doi:10.48550/ARXIV.2406.11678

    arXiv doi:10.48550/arxiv.2406.11678 2024

  3. [3]

    Craswell, B

    N. Craswell, B. Mitra, E. Yilmaz, D. Campos, and E. M. Voorhees. Overview of the TREC 2019 deep learning track, Mar. 2020

    work page 2019

  4. [4]

    Craswell, B

    N. Craswell, B. Mitra, E. Yilmaz, and D. Campos. Overview of the TREC 2020 deep learning track, Feb. 2021

    work page 2020

  5. [5]

    F. Guo, W. Li, H. Zhuang, Y. Luo, Y. Li, Q. Zhu, L. Yan, and Y. Zhang. Generating diverse criteria on-the-fly to improve point-wise llm rankers, 2024. URL https://arxiv.org/abs/2404.11960

    arXiv 2024

  6. [6]

    Khramtsova, S

    E. Khramtsova, S. Zhuang, M. Baktashmotlagh, and G. Zuccon. Leveraging LLMs for Unsupervised Dense Retriever Ranking , May 2024

    work page 2024

  7. [7]

    J. Lin, X. Ma, S.-C. Lin, J.-H. Yang, R. Pradeep, and R. Nogueira. Pyserini : A Python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021), pages 2356--2362, 2021

    work page 2021

  8. [8]

    A. . M. Llama Team. The Llama 3 Herd of Models , Nov. 2024

    work page 2024

Show all 22 references
  1. [9]

    X. Ma, X. Zhang, R. Pradeep, and J. Lin. Zero- Shot Listwise Document Reranking with a Large Language Model . 2023. doi:10.48550/ARXIV.2305.02156

    2023 doi arXiv

  2. [10]

    M. B. A. McDermott, L. H. Hansen, H. Zhang, G. Angelotti, and J. Gallifant. A closer look at auroc and auprc under class imbalance, 2024. URL https://arxiv.org/abs/2401.06091

    2024 arXiv

  3. [11]

    J. Ni, T. Schimanski, M. Lin, M. Sachan, E. Ash, and M. Leippold. DIRAS : Efficient LLM Annotation of Document Relevance in Retrieval Augmented Generation , Jan. 2025. URL http://arxiv.org/abs/2406.14162. arXiv:2406.14162 [cs]

    2025 arXiv

  4. [12]

    Nouriinanloo and M

    B. Nouriinanloo and M. Lamothe. Re- Ranking Step by Step : Investigating Pre-Filtering for Re-Ranking with Large Language Models . 2024. doi:10.48550/ARXIV.2406.18740

    2024 doi arXiv

  5. [13]

    Oosterhuis, R

    H. Oosterhuis, R. Jagerman, Z. Qin, X. Wang, and M. Bendersky. Reliable Confidence Intervals for Information Retrieval Evaluation Using Generative A . I , July 2024

    2024

  6. [14]

    GPT-4o System Card

    OpenAI . GPT-4o System Card . https://openai.com/index/gpt-4o-system-card/, Oct. 2024

    2024

  7. [15]

    Parry, S

    A. Parry, S. MacAvaney, and D. Ganguly. Top- Down Partitioning for Efficient List-Wise Ranking . 2024. doi:10.48550/ARXIV.2405.14589

    2024 doi arXiv

  8. [16]

    Z. Qin, R. Jagerman, K. Hui, H. Zhuang, J. Wu, L. Yan, J. Shen, T. Liu, J. Liu, D. Metzler, X. Wang, and M. Bendersky. Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting . 2023. doi:10.48550/ARXIV.2306.17563

    2023 doi arXiv

  9. [17]

    W. Sun, L. Yan, X. Ma, S. Wang, P. Ren, Z. Chen, D. Yin, and Z. Ren. Is ChatGPT Good at Search ? Investigating Large Language Models as Re-Ranking Agents . 2023. doi:10.48550/ARXIV.2304.09542

    2023 doi arXiv

  10. [18]

    G. Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, Aug. 2024

    2024

  11. [19]

    G. P. Team. PaLM 2 Technical Report , Sept. 2023

    2023

  12. [20]

    Thakur, N

    N. Thakur, N. Reimers, A. R \"u ckl \'e , A. Srivastava, and I. Gurevych. BEIR : A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models , Oct. 2021

    2021

  13. [21]

    Zhuang, Z

    H. Zhuang, Z. Qin, K. Hui, J. Wu, L. Yan, X. Wang, and M. Bendersky. Beyond Yes and No : Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labels , Apr. 2024 a

    2024

  14. [22]

    Zhuang, H

    S. Zhuang, H. Zhuang, B. Koopman, and G. Zuccon. A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Models , May 2024 b

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.