Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

Criteria-Based LLM Relevance Judgments

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that decomposing relevance into four graded criteria—Exactness, Topicality, Coverage, and Contextual Fit—and then aggregating them makes LLM-produced relevance judgments track human system rankings more closely than…

desk verdict A genuine new decomposition method for LLM relevance judgments, but the headline 0.99 correlation is within the authors' own noise threshold of the best baseline and rests on a leak-prone test set. read the letter →

arxiv 2507.09488 v1 pith:FGEJI43I submitted 2025-07-13 cs.IR

Naghmeh Farzi , Laura Dietz This is my paper
classification cs.IR
keywords relevancejudgmentsLLMevaluationmulti-criteriaTRECDeepLearningLLMJudgeleaderboardcorrelationprompt-basedaggregationinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that an LLM produces more reliable and interpretable relevance labels when it grades separate criteria—Exactness, Topicality, Coverage, and Contextual Fit—and then combines those grades, rather than judging relevance in a single unconstrained prompt. In the LLMJudge challenge, this Multi-Criteria approach using the relatively small LLaMA-3-8B model achieved the highest Spearman's rank correlation (0.9919) and the second-highest Kendall's tau among all submissions, outperforming systems that used much larger models or heavy fine-tuning. Across three TREC Deep Learning collections and three LLMs, the full four-criterion set wins more comparisons than any single criterion or subset, and Topicality is necessary but not sufficient. The practical payoff, if the claim holds, is that retrieval researchers can approximate human leaderboards cheaply and transparently, with per-criterion grades a human can audit before the final label is produced. The authors themselves flag that a test-set leak—models having memorized the TREC DL collections—would inflate these correlations, so the result stands on the assumption that the evaluation data was unseen.

What carries the argument

The load-bearing mechanism is a two-phase grading pipeline. Phase One runs four parallel, criterion-specific prompts—one each for Exactness, Topicality, Coverage, and Contextual Fit—so each judgment is a focused single-aspect decision on a 0–3 scale. Phase Two aggregates these four grades into a final relevance label, either by asking the LLM with an aggregator prompt that lists the grades or by summing the grades and mapping the total to a label through thresholds tuned on a held-out development set. The parallel decomposition is the distinctive element: it makes each intermediate grade separately auditable by a human, and it is what the paper credits for the robustness of the final labels.

What would settle it

Run the Multi-Criteria prompts with LLaMA-3-8B on a freshly created query–passage set whose human relevance labels were collected after the model's training cutoff and that has never been part of any public benchmark; if the Spearman correlation with the human system ranking falls well below 0.99, the headline result was partly leakage rather than the method.

Watch

Extended reading notes

Core claim

The central discovery is that decomposing the question 'is this passage relevant?' into four independently graded sub-questions changes the error profile of LLM assessors enough that the resulting system leaderboard matches the human-judgment leaderboard at Spearman 0.9919 and Kendall 0.9483, even when the grader is an 8B model that performs noticeably worse under simpler prompts. The four criteria are defined operationally: Exactness asks how precisely the passage answers the query, Topicality asks whether the passage is about the whole query subject, Coverage asks how much of the passage is dedicated to the topic, and Contextual Fit asks whether the passage provides useful background. Phase One yields four 0–3 grades; Phase Two then prompts the LLM to read those grades and produce a single 0–3 relevance label, with a sum-based variant provided as a deterministic alternative. The paper reads the results as evidence that criterion decomposition gives small models a scaffold that narrows the gap with large models on evaluation tasks, and that the uniform leniency of the grader preserves system ordering even though absolute labels skew high.

Load-bearing premise

The empirical results stand or fall on the assumption that the LLMs had not already memorized the TREC Deep Learning test collections or their human labels; the authors list this precisely as the Test Set Leak Trope and note it could inflate the reported correlations.

Editorial extensions

If this is right

  • An 8B open-weight model can reproduce human system rankings on TREC Deep Learning collections at a Spearman correlation of roughly 0.99, which reduces the need for very large or commercial models when building automated evaluation leaderboards.
  • The full four-criterion configuration is the most reliable choice; the ablation study shows that no single criterion or three-criterion subset consistently beats it, so the complementarity of the criteria is what carries the result.
  • Because the model's leniency toward weak passages applies about equally to all systems, relative rankings stay close to human rankings even though the absolute label distribution is shifted upward.
  • The method's advantage is largest for smaller models, so criterion decomposition is a cheap way to make evaluation accessible without fine-tuning or large-scale inference.
  • The authors recommend continuous meta-evaluation of LLM judges because model updates and potential test-set leakage can change or inflate these correlations over time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the active ingredient is task decomposition rather than the particular four criteria, the same two-phase design should transfer to other graded judgment tasks (answer utility, factuality, safety) whenever a small model is the grader; a testable prediction is that any 3–5 auditable dimensions beat a monolithic prompt.
  • The uniform-leniency result suggests Multi-Criteria labels are safer for comparing systems than for setting absolute thresholds or filtering individual passages, since the paper does not show that labels are calibrated to human strictness.
  • The aggregator prompt effectively receives the model's own intermediate judgments as a compressed reasoning trace, so this design could be combined with per-criterion rationale generation to get explainable labels at little extra cost.
  • A direct way to separate method from measurement is to run the same prompts on a freshly collected query set annotated after the model's training cutoff; if the correlation drops sharply, the headline number was partly test-set leakage.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a Multi-Criteria framework for LLM-based relevance judgments, in which relevance is decomposed into four criteria (Exactness, Coverage, Topicality, and Contextual Fit) that are graded independently in Phase One and then aggregated in Phase Two, either by a prompt-based aggregator or by a summation heuristic ("sumdecompose"). The method is evaluated on the LLMJudge challenge and on TREC DL 2019/2020, using LLaMA-3-8B, LLaMA-3.3-70B, and FLAN-T5-large, against baselines such as Thomas, Faggioli, RUBRIC, and UMBRELA. The headline result is that the prompt-based Multi-Criteria variant with LLaMA-3-8B achieves the highest Spearman correlation (0.9919) and second-highest Kendall's tau (0.9483) in the LLMJudge challenge, and the paper argues that the decomposition provides interpretability and competitive leaderboard reproduction at lower model scale.

Significance. If the empirical claims hold, this is a useful contribution to LLM-based IR evaluation: it demonstrates that decomposing relevance into named criteria can match or exceed monolithic prompting, especially with smaller models, and it provides a more auditable grading process. The paper merits credit for participating in an externally organized challenge (LLMJudge), for evaluating across three datasets and three LLMs, for making prompts and an online appendix available, and for including explicit limitations in Section 6. The main significance is conditional on the measurement being free of test-set contamination and on the reported differences being interpreted without an arbitrary significance convention.

major comments (4)
  1. [Section 6, Section 4.2] The central empirical claim depends on the absence of test-set leakage, but Section 6 only acknowledges the "Test Set Leak Trope" without quantifying exposure. For DL19/DL20 and for LLaMA-3.3-70B, leakage into pretraining data is plausible; for LLaMA-3-8B the paper should state the model's training-data cutoff, since a cutoff preceding TREC DL 2023 would substantially mitigate the concern for the primary LLMJudge result. Because the top entries in Table 5 are separated by less than 0.005 (0.9919 vs 0.9914), even a small contamination effect could change the headline comparison. Please add a contamination check (e.g., query-level memorization probes or a fresh-judgment replication) or explicitly bound the risk.
  2. [Section 4.1.3] The "Significance analysis" paragraph introduces a 0.005/0.025 threshold heuristic, but this is not a statistical significance test, and no confidence intervals or resampling-based tests are reported for the rank correlations. On LLMJudge, with only 25 queries, the Spearman and Kendall estimates have substantial uncertainty. This convention is used to justify claims of ties and of "falling behind" in Table 7, so the paper should either bootstrap/permutation-test the correlations or rephrase the claims as point-estimate comparisons.
  3. [Section 4.4, Table 7] The text states that "apart from two cases, the Multi-Criteria configuration with all four criteria either achieves or ties for the best performance," but Table 7 contradicts this under the paper's own 0.005 convention. For example, with LLaMA-3-8B on DL20, the all-four NDCG@10 values are 0.972/0.871 while single criteria reach 0.984/0.911; on DL19 the all-four values are 0.972/0.871 versus 0.980/0.891 for Exactness. Similar gaps appear for LLaMA-3.3-70B on DL20 (0.927 vs 0.952 for Kendall's tau) and DL19 (0.856 vs 0.904). The caption's "falling behind in only 2 out of 12 experiments" appears to use the 0.025 red threshold, which is not equivalent to "tying" under the 0.005 rule; please reconcile the text and table and define "falling behind" precisely.
  4. [Section 4.2, Table 6] The claim that "the Multi-Criteria approach with prompt-based aggregation achieves the highest score more often than any other method—14 times" is not supported by the Wins column in Table 6, where the authors' own sumdecompose variant records 15 wins. If the comparison is intended to exclude their own variants, that exclusion should be stated explicitly, and the definition of a "win" (strict maximum versus ties, and which of the 36 cells count) should be given.
minor comments (4)
  1. [Table 5 vs Table 6] For the same LLMJudge condition, Multi-Criteria with LLaMA-3-8B is reported as Spearman 0.9919 / Kendall 0.9483 in Table 5 but as 0.994 / 0.952 in Table 6; please explain the discrepancy or use identical computations/rounding.
  2. [Section 3.2.2] The sumdecompose thresholds are tuned on the LLMJudge development set; please report the dev-set performance and a sensitivity analysis, since 25 queries is a small basis for threshold selection.
  3. [Section 4.3] The runtime comparison reports only slowdown factors (5.44-5.45x for LLaMA-3-8B, 1.24-1.33x for FLAN-T5-large); absolute inference times and hardware/config details would make the comparison reproducible.
  4. [Figure 2] The caption refers to a "red diagonal" but the text describes an ascending line of red dots; please align the terminology and clarify whether the line is the identity line or a fitted trend.

Circularity Check

0 steps flagged · score 2.0 of 10

Empirical leaderboard-evaluation paper with no construction-level circularity; self-cited baselines and disclosed leakage risk are not load-bearing.

full rationale

The paper is an empirical meta-evaluation, not a derivation. Multi-Criteria grades each criterion via a prompt and then asks an LLM to aggregate the criterion grades into a relevance label; the label is validated against human judgments on TREC DL 2019/2020 and LLMJudge. No equation in Sections 3.1-3.2 defines the predicted label as a function of the human labels or of the leaderboard metric, so there is no self-definitional reduction. The only explicitly tuned component, the Sumdecompose grade-sum-to-label mapping (Table 3), is fit to the LLMJudge development set held out from evaluation: 'The thresholds used in our experiments are tuned on the LLMJudge development set (held out from evaluation) to maximize correlation with human judgments.' Tables 5-6 then report held-out test results, which is standard model selection rather than fitting-to-target. Self-citations are present (RUBRIC [11,15], UMBRELA reproduction [16], EXAM++ [14]), but they are used as baselines or prior art; the headline 0.9919 Spearman result comes from participation in the external LLMJudge challenge against other teams' submissions, so the central claim does not rest on the authors' own prior claims. Section 6 honestly flags the Test Set Leak Trope (possible LLM pretraining exposure to TREC DL test collections) and an Old System Trope 'risk of circularity' for future LLM-based IR systems; those are contamination and generalization threats to the empirical estimate, not a construction-level equivalence between the method's inputs and outputs. Accordingly, no circular step meets the evidence bar.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The ledger shows one fitted numeric mapping (sumdecompose thresholds), three domain or design assumptions, and no invented entities. The four criteria are adapted from classic relevance theory rather than newly postulated. The central empirical claim rests mainly on the assumption that human labels are ground truth and that no test-set leakage occurred.

free parameters (1)
  • Sumdecompose grade-sum to relevance label thresholds = 10-12 -> 3, 7-9 -> 2, 5-6 -> 1, 0-4 -> 0
    Tuned on the LLMJudge development set, held out from evaluation, to maximize correlation with human judgments. This is a fitted decision rule, not a derived mapping.
assumptions (3)
  • domain assumption Human relevance labels in TREC DL and LLMJudge are valid ground truth for measuring evaluation quality.
    The paper adopts human labels as the gold standard, citing prior work that only humans can arbitrate relevance, and computes all leaderboard correlations against them.
  • ad hoc to paper Exactness, Topicality, Coverage, and Contextual Fit form a sufficient decomposition of relevance for TREC-style ad hoc retrieval, and the LLM can grade each independently.
    The four criteria are introduced by hand in Section 3.1 with rationales from classic IR theory, but their exhaustiveness is not demonstrated. The ablation shows no subset is consistently better, yet the set itself is assumed.
  • domain assumption The LLMs used were not trained on the TREC DL test collections in a way that leaks relevance labels.
    The empirical validity of the high correlations depends on this. The paper explicitly flags it as the Test Set Leak Trope in Section 6.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Criteria-Based LLM Relevance Judgments." pith.science (2026). https://pith.science/paper/FGEJI43I

@misc{pith2026250709488,
  author       = {Pith},
  title        = {Pith review of: Criteria-Based LLM Relevance Judgments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FGEJI43I}},
  note         = {Machine review of arXiv:2507.09488}
}
read the original abstract

Relevance judgments are crucial for evaluating information retrieval systems, but traditional human-annotated labels are time-consuming and expensive. As a result, many researchers turn to automatic alternatives to accelerate method development. Among these, Large Language Models (LLMs) provide a scalable solution by generating relevance labels directly through prompting. However, prompting an LLM for a relevance label without constraints often results in not only incorrect predictions but also outputs that are difficult for humans to interpret. We propose the Multi-Criteria framework for LLM-based relevance judgments, decomposing the notion of relevance into multiple criteria--such as exactness, coverage, topicality, and contextual fit--to improve the robustness and interpretability of retrieval evaluations compared to direct grading methods. We validate this approach on three datasets: the TREC Deep Learning tracks from 2019 and 2020, as well as LLMJudge (based on TREC DL 2023). Our results demonstrate that Multi-Criteria judgments enhance the system ranking/leaderboard performance. Moreover, we highlight the strengths and limitations of this approach relative to direct grading approaches, offering insights that can guide the development of future automatic evaluation frameworks in information retrieval.

Figures

Figures reproduced from arXiv: 2507.09488 by the authors.

Figure 1
Figure 1. Overview the two phases of Multi-Criteria: grading [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. NDCG@10 evaluation scores under manual judg [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Abbreviations: T = Topicality, F = Contextual Fit, [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. mamabench and mamaretrieval: Benchmarks for Evaluating Medical Retrieval-Augmented Generation in Maternal, Neonatal, and Reproductive Health

    cs.CL 2026-06 unverdicted novelty 7.0 of 10

    The authors introduce and release two new benchmarks for maternal-health RAG evaluation, built from expert sources with graded labels and disclosed limitations rather than binary judgments or new question authoring.

  2. SAGE: Scalable AI Governance & Evaluation

    cs.IR 2026-02 conditional novelty 6.0 of 10

    SAGE co-evolves a relevance policy, expert-curated precedents, and a distilled LLM judge to grade search relevance at production scale, reporting 0.72–0.73 linear Cohen's kappa against humans and a 0.25% DAU lift at LinkedIn.

Reference graph

Works this paper leans on

38 extracted references · 34 canonical work pages · cited by 2 Pith papers

  1. [1]

    Can we use large language models to fill relevance judgment holes? arXiv preprint arXiv:2405.05600, 2024

    Zahra Abbasiantaeb, Chuan Meng, Leif Azzopardi, and Mohammad Aliannejadi. Can we use large language models to fill relevance judgment holes? arXiv preprint arXiv:2405.05600, 2024

    arXiv 2024

  2. [2]

    Llms can be fooled into labelling a document as relevant: best café near me; this paper is perfectly relevant

    Marwah Alaofi, Paul Thomas, Falk Scholer, and Mark Sanderson. Llms can be fooled into labelling a document as relevant: best café near me; this paper is perfectly relevant. In Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, SIGIR-AP 2024, page 32–41, New Yor...

    work page 2024

  3. [3]

    Negar Arabzadeh and Charles L. A. Clarke. Benchmarking LLM-based relevance judgment methods. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’25) , Padua, Italy, July 2025. ACM

    work page 2025

  4. [4]

    Barry and Linda Schamber

    Carol L. Barry and Linda Schamber. Users’ criteria for relevance evaluation: A cross-situational comparison. Information Processing & Management , 34(2):219– 236, 1998

    work page 1998

  5. [5]

    The concept of relevance in IR

    Pia Borlund. The concept of relevance in IR. Department of Information Studies, Royal School of Library and Information Science , 2003

    work page 2003

  6. [6]

    Beyond topicality: A two stage view of relevance and the retrieval process

    Bert Boyce. Beyond topicality: A two stage view of relevance and the retrieval process. Information Processing & Management , 18(3):105–109, 1982

    work page 1982

  7. [7]

    Charles L. A. Clarke and Laura Dietz. LLM-based relevance assessment still can’t replace human relevance assessment. In Proceedings of the 10th International Workshop on Evaluating Information Access (EVIA 2025), a Satellite Workshop of the NTCIR-18 Conference, Tokyo, Japan, June 2025. National Institute of Informatics

    work page 2025

  8. [8]

    Dimensions of Relevance

    E Cosijn and P Ingwersen. Dimensions of Relevance. Information Processing and Management, 36:533–550, 2000. Series Number: 4

    work page 2000

Show all 38 references
  1. [9]

    Overview of the TREC 2020 deep learning track

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. Overview of the TREC 2020 deep learning track. In Proceedings of the Twenty-Ninth Text Naghmeh Farzi and Laura Dietz REtrieval Conference (TREC 2020), volume 1266 ofNIST Special Publication, Online (virtual), Novem...

    2020

  2. [10]

    Voorhees

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. Overview of the TREC 2019 deep learning track. In Proceedings of the Twenty-Eighth Text REtrieval Conference (TREC 2019) , volume 1250 of NIST Special Publication, Gaithersburg, MD, USA, November...

    2019

  3. [11]

    A workbench for autograding retrieve/generate systems

    Laura Dietz. A workbench for autograding retrieve/generate systems. InProceed- ings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1963–1972, 2024

    1963

  4. [12]

    Principles and guidelines for the use of llm judges

    Laura Dietz, Oleg Zendel, Peter Bailey, Charles Clarke, Ellese Cotterill, Jeff Dalton, Faegheh Hasibi, Mark Sanderson, and Nick Craswell. Principles and guidelines for the use of llm judges. In Proceedings of the 11th ACM SIGIR / The 15th International Conference on Innovative...

    2025

  5. [13]

    Guglielmo Faggioli, Laura Dietz, Charles L. A. Clarke, Gianluca Demartini, Matthias Hagen, Claudia Hauff, N. Kando, E. Kanoulas, Martin Potthast, Benno Stein, and Henning Wachsmuth. Perspectives on large language models for relevance judgment. Proceedings of the 2023 ACM SIGIR...

    2023

  6. [14]

    EXAM++: LLM-based Answerability Metrics for IR Evaluation

    Naghmeh Farzi and Laura Dietz. EXAM++: LLM-based Answerability Metrics for IR Evaluation. In Proceedings of LLM4Eval: The First Workshop on Large Language Models for Evaluation in Information Retrieval , 2024

    2024

  7. [15]

    Pencils Down! Automatic Rubric-based Eval- uation of Retrieve/Generate Systems

    Naghmeh Farzi and Laura Dietz. Pencils Down! Automatic Rubric-based Eval- uation of Retrieve/Generate Systems. In Proceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval, pages 175–184, Wash- ington DC USA, August 2024. ACM

    2024

  8. [16]

    Naghmeh Farzi and Laura Dietz. Does umbrela work on other llms? InProceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’25) , Padua, Italy, July 2025. ACM

    2025

  9. [17]

    Bruce Croft

    Helia Hashemi, Mohammad Aliannejadi, Hamed Zamani, and W. Bruce Croft. Antique: A non-factoid question answering benchmark. In Advances in Infor- mation Retrieval: 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14–17, 2020, Proceedings, Part II , p...

    2020

  10. [18]

    One-shot labeling for automatic relevance estimation

    Sean MacAvaney and Luca Soldaini. One-shot labeling for automatic relevance estimation. Proceedings of the 46th International ACM SIGIR Conference on Re- search and Development in Information Retrieval , 2023

    2023

  11. [19]

    Crisisfacts: building and evaluating crisis timelines

    Richard McCreadie and Cody Buntain. Crisisfacts: building and evaluating crisis timelines. In Proceedings of the 20th International ISCRAM Conference , pages 320–339, 2023

    2023

  12. [20]

    Query performance prediction using relevance judgments generated by large language models

    Chuan Meng, Negar Arabzadeh, Arian Askari, Mohammad Aliannejadi, and Maarten de Rijke. Query performance prediction using relevance judgments generated by large language models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Inform...

    2024

  13. [21]

    Golbus, and Javed A

    Virgil Pavlu, Shahzad Rajput, Peter B. Golbus, and Javed A. Aslam. IR system evaluation using nugget-based test collections. In Proceedings of the fifth ACM international conference on Web search and data mining , pages 393–402, Seattle Washington USA, February 2012. ACM

    2012

  14. [22]

    Initial nugget evaluation results for the trec 2024 rag track with the autonuggetizer framework, 2024

    Ronak Pradeep, Nandan Thakur, Shivani Upadhyay, Daniel Campos, Nick Craswell, and Jimmy Lin. Initial nugget evaluation results for the trec 2024 rag track with the autonuggetizer framework, 2024

    2024

  15. [23]

    Rahmani, Varsha Ramineni, Nick Craswell, Bhaskar Mitra, and Emine Yilmaz

    Hossein A. Rahmani, Varsha Ramineni, Nick Craswell, Bhaskar Mitra, and Emine Yilmaz. Towards understanding bias in synthetic data for evaluation, 2025

    2025

  16. [24]

    Rahmani, Clemencia Siro, Mohammad Aliannejadi, Nick Craswell, Charles L

    Hossein A. Rahmani, Clemencia Siro, Mohammad Aliannejadi, Nick Craswell, Charles L. A. Clarke, Guglielmo Faggioli, Bhaskar Mitra, Paul Thomas, and Emine Yilmaz. Judging the judges: A collection of llm-generated relevance judgements, 2025

    2025

  17. [25]

    Rahmani, Emine Yilmaz, Nick Craswell, and Bhaskar Mitra

    Hossein A. Rahmani, Emine Yilmaz, Nick Craswell, and Bhaskar Mitra. Judge- blender: Ensembling judgments for automatic relevance assessment. In Compan- ion Proceedings of the ACM Web Conference 2025 (WWW Companion ’25) , Sydney, NSW, Australia, April 2025. ACM

    2025

  18. [26]

    Rahmani, Emine Yilmaz, Nick Craswell, Bhaskar Mitra, Paul Thomas, Charles L

    Hossein A. Rahmani, Emine Yilmaz, Nick Craswell, Bhaskar Mitra, Paul Thomas, Charles L. A. Clarke, Mohammad Aliannejadi, Clemencia Siro, and Guglielmo Faggioli. LLMJudge: LLMs for relevance judgments. In Proceedings of the 47th International ACM SIGIR Conference on Research an...

    2024

  19. [27]

    Swan: A generic framework for auditing textual conversational systems, 2023

    Tetsuya Sakai. Swan: A generic framework for auditing textual conversational systems, 2023

    2023

  20. [28]

    Exam: How to evaluate retrieve-and-generate systems for users who do not (yet) know what they want

    David P Sander and Laura Dietz. Exam: How to evaluate retrieve-and-generate systems for users who do not (yet) know what they want. In DESIRES, pages 136–146, 2021

    2021

  21. [29]

    Relevance: A review of the literature and a framework for thinking on the notion in information science

    Tefko Saracevic. Relevance: A review of the literature and a framework for thinking on the notion in information science. part ii: nature and manifestations of relevance. J. Am. Soc. Inf. Sci. Technol. , 58(13):1915–1933, November 2007

    1915

  22. [30]

    Don’t use LLMs to make relevance judgments.Information Retrieval Research Journal, 1(1):10.54195/irrj.19625, March 2025

    Ian Soboroff. Don’t use LLMs to make relevance judgments.Information Retrieval Research Journal, 1(1):10.54195/irrj.19625, March 2025

    2025 doi

  23. [31]

    Is ChatGPT good at search? investigating large language models as re-ranking agents

    Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Is ChatGPT good at search? investigating large language models as re-ranking agents. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Proces...

    2023

  24. [32]

    Large language models can accurately predict searcher preferences

    Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. Large language models can accurately predict searcher preferences. InAnnual International ACM SIGIR Conference on Research and Development in Information Retrieval , 2023

    2023

  25. [33]

    Together inference api, 2024

    Together AI. Together inference api, 2024. [Online Service] Available at: https://www.together.ai/

    2024

  26. [34]

    Umbrela: Umbrela is the (open-source reproduction of the) bing relevance assessor, 2024

    Shivani Upadhyay, Ronak Pradeep, Nandan Thakur, Nick Craswell, and Jimmy Lin. Umbrela: Umbrela is the (open-source reproduction of the) bing relevance assessor, 2024

    2024

  27. [35]

    Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Huai hsin Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022) , pages 24...

    2022

  28. [36]

    Relevance judgment: What do information users consider beyond topicality? Journal of the American Society for Information Science & Technology, 57(7):961–973, May 2006

    Yunjie Xu and Zhiwei Chen. Relevance judgment: What do information users consider beyond topicality? Journal of the American Society for Information Science & Technology, 57(7):961–973, May 2006. Publisher: Wiley-Blackwell

    2006

  29. [37]

    Hengran Zhang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, and Xueqi Cheng. Are large language models good at utility judgments? In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’24) , pages ...

    1941

  30. [38]

    Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed H. Chi. Least- to-most prompting enables complex reasoning in large language models. In Proceedings of the 11th International Conference...

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.