Limitations of Automatic Relevance Assessments with Large Language Models for Fair and Reliable Retrieval Evaluation

\'Alvaro Barreiro; David Otero; Javier Parapar

REVIEW 3 major objections 5 minor 23 references

Limitations of Automatic Relevance Assessments with Large Language Models for Fair and Reliable Retrieval Evaluation

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLM relevance labels misrank top systems and inflate significance differences.

desk verdict Useful caution for IR evaluation, but the strongest claim about LLM judgment quality is undercut by comparing disjoint topic sets with different qrel depths. read the letter →

arxiv 2411.13212 v3 pith:BNAAHO7D submitted 2024-11-20 cs.IR

David Otero , Javier Parapar , Álvaro Barreiro This is my paper · ORCID

classification cs.IR

keywords relevanceassessmentlargelanguagemodelstestcollectionsretrievalevaluationtop-heavyrankcorrelationstatisticalsignificancefalsepositivesSynDL

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether relevance judgments written by a large language model can stand in for human judgments when evaluating search systems offline. Its answer is that they cannot yet. Across five years of Deep Learning Track test collections, GPT-4-generated labels rank top-performing systems noticeably differently from official human labels: top-weighted correlation measures drop well below standard Kendall's tau, and individual runs move by five to more than forty positions. The paper further shows that pairwise significance tests run on LLM labels mark many pairs of systems as significantly different when the human labels say they are not, an outcome it interprets as a bias toward false positives. The conclusion is that LLM-based relevance assessments are not ready to replace human gold judgments in retrieval evaluation.

What carries the argument

The argument runs on the SynDL synthetic test collection, whose LLM labels were produced by GPT-4, compared against official TREC human judgments across DL-2019 to DL-2023, with disjoint topic sets. Three instruments carry the analysis: top-weighted rank correlations (tau_AP and RBO with p = 0.7) that penalize disagreements near the top of the system ranking; a fairness analysis of per-run rank changes; and a binary classification of pairwise statistical significance decisions from a two-sided Wilcoxon signed-rank test at alpha = 0.05, summarized as true/false positives and negatives, with an undersampling control that equalizes topic counts between human and LLM label sets.

What would settle it

Re-run the comparison on a benchmark where the same topics receive both human and LLM judgments, and compute the pairwise significance agreement on that overlapping set. If the false-positive rate collapses toward zero, the paper's conclusion about LLM labels as a source of bias would fail; if it persists, the conclusion would be confirmed on a cleaner design.

Watch

Extended reading notes

Core claim

Using the SynDL collection, where GPT-4 supplied relevance labels for topics disjoint from the human-judged topics, the authors show three results. First, agreement between LLM and human system rankings shrinks as attention moves to the top: tau_AP and RBO values fall well below Kendall's tau, so the usual correlation numbers overstate how well LLM labels identify the best systems. Second, on a per-run basis most runs change position under LLM labels, with drops of up to around fifty places, implying the evaluation is not fair across systems. Third, pairwise Wilcoxon signed-rank tests with alpha = 0.05 classify far more pairs as significantly different under LLM labels than under human labels; after equalizing the number of topics by undersampling, false positives fall but remain high, and false negatives rise. The authors conclude that LLM-generated qrels do not preserve ranking differences among top systems or statistical significance decisions.

Load-bearing premise

Human and LLM labels are applied to completely different sets of topics, so the comparison assumes those topic sets are interchangeable; if the LLM-judged topics are systematically easier or harder in a way that affects significance, the reported false-positive rates would not be attributable to the labels themselves.

Editorial extensions

If this is right

Researchers who use LLM-generated labels to compare top-performing systems will frequently get the ordering wrong, even when overall Kendall's tau looks acceptable.
Statistical comparisons built on LLM labels will over-report significant improvements, leading to published claims of differences that human labels do not support.
Equalizing the number of topics reduces but does not eliminate the false-positive problem, so the bias is not purely a sample-size artifact.
When false negatives occur they concentrate in a few runs, meaning some systems are systematically disadvantaged by LLM evaluation.
NDCG appears more stable than AP under LLM labels, but both metrics still show substantial disagreement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the disjoint-topic design contributes to the disagreement, an overlapping benchmark where humans and LLMs label the same topics would give a cleaner estimate of label quality; this is a direct test the authors' setup leaves open.
LLM labels might still serve as a cheap screening step for building collections, as long as final leaderboard decisions and significance claims are reserved for human judgments or a calibrated hybrid.
Extending the analysis to other LLMs and graded relevance scales would show whether the unfairness at the top is a GPT-4-specific artifact or a general property of LLM-based assessment.
A practical follow-up could threshold on the LLM's confidence or agreement across repeated samples to flag pairs whose significance decision is unstable.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Worth reading, with a caveat that matters. The paper is a useful caution about using LLM-generated qrels for IR evaluation, but its strongest claim—that LLM judgments are unfair and produce systematic false positives—is not fully identified, because the human and LLM judgments come from disjoint topic sets with different qrel depths.

The paper does something new: instead of stopping at Kendall's tau between human and GPT-4 qrels, it applies top-weighted rank correlations and pairwise significance-testing to the SynDL qrels. The findings are striking. Human/LLM Kendall's tau looks fine, but tau_AP and especially RBO are much lower, meaning top-of-leaderboard distinctions are not preserved. And in the Wilcoxon pairwise comparisons, LLM qrels produce a very high false-positive rate (AP FP 71–89% before undersampling). Those numbers are a useful warning to anyone treating LLM-generated qrels as drop-in replacements.

Credit where due: the experiments are transparent, use public TREC Deep Learning collections and the released SynDL qrels, and the undersampling check in Table 4 is an honest attempt to address the sample-size objection. The citation pattern is clean; the self-citations are methodological and appropriate.

The soft spot is real and load-bearing, and the paper's own text exposes it. T_H and T_LLM are disjoint by construction, and the per-topic judgment depths differ substantially (DL-19: 215 vs. 70). So a ranking change or a significance decision could be due to the label source or due to a different set of topics with different difficulty and qrel depth. The undersampling experiment matches only topic count, not topic identity or per-topic depth, so the confound remains. The abstract's "LLM-based judgements are unfair" is too strong for what is identified. A same-topic comparison, or evidence that the two topic sets are exchangeable in difficulty and depth, would be needed to support the causal claim. Minor issues: no variance around the undersampled averages, and RBO is reported at a single persistence parameter.

Verdict for a desk editor: send it out. The paper is useful to the IR evaluation community despite the confound; it shows empirically that high overall correlation does not imply reliable top-system ranking or trustworthy significance decisions when using SynDL-style qrels. I would not cite the central causal claim as settled, but the cautionary pattern is citable. A serious referee can ask for the same-topic evidence or a framed-down conclusion.

Referee Report

3 major / 5 minor

Summary. The paper compares official TREC human relevance judgments with GPT-4 generated judgments from the SynDL collection across five Deep Learning Track datasets (DL-19 through DL-23). It evaluates top-weighted rank correlations (τ_AP and RBO) alongside Kendall's τ, analyzes per-run changes in ranking positions, and compares pairwise Wilcoxon signed-rank test decisions between human and LLM qrels. The authors report that top-heavy correlations are much lower than Kendall's τ, that many runs change rank substantially, and that LLM qrels yield high false-positive rates in significance testing. An undersampling experiment that equalizes the number of topics reduces but does not eliminate the false-positive rates. The paper concludes that LLMs are not yet ready to replace human relevance judgments in offline IR evaluation.

Significance. If the causal interpretation were valid, this would be an important negative result for the use of LLM-generated test collections, directly challenging the prevailing reliance on rank correlations as evidence of reliability. The paper usefully demonstrates that high Kendall's τ does not imply reliable top-system comparisons or preserved significance decisions, and the use of multiple TREC DL collections lends breadth to the analysis. The authors also make their methodology transparent and their code and data presumably available. However, the paper's central causal claim is undermined by a fundamental design confound: the human and LLM judgments come from disjoint topic sets, so the observed differences in rankings and significance decisions cannot be unambiguously attributed to the label source.

major comments (3)

[Section 3, Table 1] The comparison between Q_H and Q_LLM is confounded by the disjointness of the topic sets. The paper states 'We do not assume any overlap between T_H and T_LLM, in fact, they may be completely disjoint sets,' and Table 1 confirms T_H ∩ T_LLM = ∅ for all five years. The topic sets also differ substantially in size (e.g., DL-19: 43 vs. 157 topics) and in average judgments per topic (e.g., DL-22: 5084 vs. 133). Under this design, differences in per-topic score vectors, and therefore differences in system rankings and pairwise Wilcoxon significance decisions, could arise purely from differences in topic difficulty and judgment depth even if the LLM labels were perfect. The undersampling experiment in Table 4 controls only the number of topics, not their content or depth, so it does not remove this confound. The abstract's causal claims ('LLM-based judgements are unfair at ranking top-performing systems' and 'exceedingly high rate of false positives') are therefore not supported by the evidence as presented. The authors must either demonstrate that T_H and T_LLM are exchangeable in difficulty and depth, or substantially weaken the causal language throughout the paper.
[Table 4 and Section 3.2] The undersampled results are reported as averages over 200 random subsets of T_LLM, but the paper provides no confidence intervals, standard deviations, or other measures of variability for these averages. For datasets with small |T_H| (e.g., 43 in DL-19), the distribution of the false-positive and true-positive rates across random subsets is likely to be wide, so the reader cannot determine whether the differences between Table 3 and Table 4 are stable or within sampling noise. Please report the full distribution, percentile intervals, or standard errors for the undersampled estimates.
[Section 2, Table 2] The RBO(p=0.7) results are strikingly inconsistent across metrics and years, for example DL-19: AP RBO=0.11 vs. NDCG RBO=0.93, and DL-21: AP RBO=0.52 vs. NDCG RBO=0.97. The paper does not discuss these large discrepancies, yet they are directly relevant to the claim that LLM-based judgments fail to preserve top-system rankings; the conclusion appears to be metric-dependent. Please provide an explanation for these inconsistencies and discuss their implications for the overall fairness claim.

minor comments (5)

[Section 3.2] The text 'We plot the distribution of these drops in Figure 1' refers to the boxplot of significance drops, but in the manuscript that figure is numbered Figure 3; the cross-reference is incorrect.
[Tables 3 and 4] The table headers describe the qrels as 'synthetic'; using 'LLM-generated' or 'SynDL' would be more precise and avoid confusion with other types of synthetic qrels.
[Section 2.1] Figure 2 shows per-run rank changes only for DL-2020; the paper does not justify why this year was selected instead of the other years.
[References] Reference [22] contains a typographical error: '57100580' should likely be '571–580'.
[Section 2, footnote 1] The footnote for the RBO parameter p=0.7 cites Reference [15], Section 4.5, but Reference [15] (Thomas et al., 2024) does not appear to discuss RBO persistence; a citation to the RBO paper (Reference [21]) or a direct justification of the parameter choice would be clearer.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's claims are derived empirically from external TREC human qrels and SynDL LLM qrels; the disjoint-topic design is a validity threat, not a circular step.

full rationale

The paper's central assertions about LLM-generated relevance judgements decreasing top-heavy rank correlations and inflating false positives in pairwise significance tests are obtained by direct computation on external benchmarks: official TREC human judgements and the SynDL LLM-generated qrels from Rahmani et al. (2024). No parameter is fitted to the observed outcomes and then presented as a prediction: the LLM labels are produced independently by GPT-4 in the SynDL dataset, and the paper reuses them as-is. The self-citations in references [8]-[11] and [19] are methodological pointers for the fairness plotting convention and for the choice of the Wilcoxon signed-rank test; these choices are externally standard and do not encode the result being claimed. The fact that T_H and T_LLM are disjoint topic sets creates a genuine confounding issue for causal attribution of ranking and p-value differences to the label source, but that is an experimental validity concern, not a self-definitional or fitted-input circularity; the paper even tests its own alternative explanation (sample-size power) by undersampling T_LLM in Table 4. Therefore the derivation chain does not reduce to its inputs.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

Central claim depends on treating human qrels as gold, treating T_H and T_LLM as exchangeable samples, and assuming SynDL GPT-4 labels represent LLM assessment methods generally. No invented entities. Analysis choices RBO p=0.7 and alpha=0.05 are hand-set.

free parameters (3)

RBO persistence p = 0.7
Chosen by hand to focus on the top third or so of the system ranking; affects the top-weighted correlation results in Table 2.
Significance level alpha = 0.05
Standard threshold used for Wilcoxon tests; determines TP/TN/FP/FN classification in Tables 3 and 4.
Undersampling iterations = 200 and 50
Number of random subsamples of T_LLM used to match sample sizes; chosen for computational convenience, results averaged over iterations.

assumptions (4)

domain assumption Human judgments are treated as ground truth.
The paper defines Q_H as gold and computes false positives relative to it; noisy human judgments would change the error rates.
domain assumption T_H and T_LLM are treated as exchangeable topic sets.
The significance comparison computes p-values over disjoint topic sets and attributes differences to label source; this requires the topic sets to be comparable draws.
standard math Wilcoxon signed-rank test assumptions hold for per-topic score arrays.
Test recommended by prior IR literature and applied to paired per-topic scores within each qrel set.
domain assumption SynDL GPT-4 labels are representative of LLM-based relevance judgments.
The paper generalizes from one synthetic collection to LLM assessors broadly.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Limitations of Automatic Relevance Assessments with Large Language Models for Fair and Reliable Retrieval Evaluation." pith.science (2026). https://pith.science/paper/BNAAHO7D

@misc{pith2026241113212,
  author       = {Pith},
  title        = {Pith review of: Limitations of Automatic Relevance Assessments with Large Language Models for Fair and Reliable Retrieval Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BNAAHO7D}},
  note         = {Machine review of arXiv:2411.13212}
}

read the original abstract

Offline evaluation of search systems depends on test collections. These benchmarks provide the researchers with a corpus of documents, topics and relevance judgements indicating which documents are relevant for each topic. While test collections are an integral part of Information Retrieval (IR) research, their creation involves significant efforts in manual annotation. Large language models (LLMs) are gaining much attention as tools for automatic relevance assessment. Recent research has shown that LLM-based assessments yield high systems ranking correlation with human-made judgements. These correlations are helpful in large-scale experiments but less informative if we want to focus on top-performing systems. Moreover, these correlations ignore whether and how LLM-based judgements impact the statistically significant differences among systems with respect to human assessments. In this work, we look at how LLM-generated judgements preserve ranking differences among top-performing systems and also how they preserve pairwise significance evaluation as human judgements. Our results show that LLM-based judgements are unfair at ranking top-performing systems. Moreover, we observe an exceedingly high rate of false positives regarding statistical differences. Our work represents a step forward in the evaluation of the reliability of using LLMs-based judgements for IR evaluation. We hope this will serve as a basis for other researchers to develop more reliable models for automatic relevance assessment.

Figures

Figures reproduced from arXiv: 2411.13212 by the authors.

**Figure 2.** Per-run drops in DL-2020. The x-axis shows the [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Distribution of the drops in significance counts of [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 4 canonical work pages

[1]

Zahra Abbasiantaeb, Chuan Meng, Leif Azzopardi, and Mohammad Aliannejadi
[2]

Marwah Alaofi, Paul Thomas, Falk Scholer, and Mark Sanderson. 2024. Llms can be fooled into labelling a document as relevant: best café near me; this paper is perfectly relevant. In Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP 2024). Association ...

arXiv 2024
[4]

Guglielmo Faggioli et al. 2023. Perspectives on large language models for rele- vance judgment. In Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR ’23). Association for Computing Machinery, Taipei, Taiwan, 39–50. doi: 10.1145/3578337.3605136 (cited on pp. 1, 2)

arXiv 2023
[5]

Maurice G. Kendall. 1948. Rank Correlation Methods. Charles Griffin and Com- pany Limited (cited on p. 1)

work page 1948
[6]

Mucahid Kutlu, Tamer Elsayed, Maram Hasanain, and Matthew Lease. 2018. When rank order isn’t enough: new statistical-significance-aware correlation measures. In Proceedings of the 27th ACM International Conference on Infor- mation and Knowledge Management (CIKM ’18). Association for Computing Machinery, Torino, Italy, 397–406. isbn: 9781450360142. doi: 10...

doi:10.1145/3269206 2018
[7]

Sean MacAvaney and Luca Soldaini. 2023. One-shot labeling for automatic rele- vance estimation. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’23). Association for Computing Machinery, Taipei, Taiwan, 2230–2235. doi: 10.1145/3539618.35 92032 (cited on p. 1)

doi:10.1145/3539618.35 2023
[8]

David Otero, Javier Parapar, and Álvaro Barreiro. 2023. Relevance feedback for building pooled test collections. Journal of Information Science . doi: 10.1177/01 655515231171085 (cited on p. 2)

work page doi:10.1177/01 2023
[9]

David Otero, Javier Parapar, and Álvaro Barreiro. 2025. Towards reliable test- ing for multiple information retrieval system comparisons. In Advances in Information Retrieval. Springer Nature Switzerland, Cham, 424–439. isbn: 978- 3-031-88711-6 (cited on p. 3)

work page 2025

Show all 23 references

[10]

David Otero, Javier Parapar, and Nicola Ferro. 2023. How discriminative are your qrels? how to study the statistical significance of document adjudication methods. In Proceedings of the 32nd ACM International Conference on Infor- mation and Knowledge Management (CIKM ’23). Ass...

2023
[11]

Losada, and Álvaro Barreiro

Javier Parapar, David E. Losada, and Álvaro Barreiro. 2021. Testing the tests: simulation of rankings to compare statistical significance tests in information retrieval evaluation. In Proceedings of the 36th Annual ACM Symposium on Applied Computing (SAC ’21). Association for ...

2021
[12]

Rahmani, Nick Craswell, Emine Yilmaz, Bhaskar Mitra, and Daniel Campos

Hossein A. Rahmani, Nick Craswell, Emine Yilmaz, Bhaskar Mitra, and Daniel Campos. 2024. Synthetic test collections for retrieval evaluation. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’24). Associa...

2024
[13]

Rahmani, Xi Wang, Emine Yilmaz, Nick Craswell, Bhaskar Mitra, and Paul Thomas

Hossein A. Rahmani, Xi Wang, Emine Yilmaz, Nick Craswell, Bhaskar Mitra, and Paul Thomas. 2024. Syndl: a large-scale synthetic test collection for passage retrieval. (2024). https://arxiv.org/abs/2408.16312 arXiv: 2408.16312 [cs.IR] (cited on p. 2)

2024 arXiv
[14]

Mark Sanderson and Justin Zobel. 2005. Information retrieval system eval- uation: effort, sensitivity, and reliability. In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Informa- tion Retrieval (SIGIR ’05). Association for Comp...

2005
[15]

Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2024. Large language models can accurately predict searcher preferences. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’24). Association fo...

2024
[16]

Shivani Upadhyay, Ehsan Kamalloo, and Jimmy Lin. 2024. Llms can patch up missing relevance judgments in evaluation. (2024). https://arxiv.org/abs/2405.0 4727 arXiv: 2405.04727 [cs.IR] (cited on p. 1)

2024 arXiv
[17]

Shivani Upadhyay, Ronak Pradeep, Nandan Thakur, Daniel Campos, Nick Craswell, Ian Soboroff, Hoa Trang Dang, and Jimmy Lin. 2024. A large-scale study of relevance assessments with large language models: an initial look. (2024). https://arxiv.org/abs/2411.08275 arXiv: 2411.08275...

2024 arXiv
[18]

Shivani Upadhyay, Ronak Pradeep, Nandan Thakur, Nick Craswell, and Jimmy Lin. 2024. Umbrela: umbrela is the (open-source reproduction of the) bing relevance assessor. arXiv:2406.06519 (cited on p. 1)

2024 arXiv
[19]

Daniel Valcarce, Alejandro Bellogín, Javier Parapar, and Pablo Castells. 2018. On the robustness and discriminative power of information retrieval met- rics for top-n recommendation. In Proceedings of the 12th ACM Conference on Recommender Systems (RecSys ’18). Association for...

2018
[20]

Voorhees

Ellen M. Voorhees. 2018. On building fair and reusable test collections using bandit techniques. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (CIKM ’18). Association for Comput- ing Machinery, Torino, Italy, 407–416. isbn: 978...

2018 doi
[21]

William Webber, Alistair Moffat, and Justin Zobel. 2010. A similarity measure for indefinite rankings. ACM Transactions on Information Systems, 28, 4, Article 20, 38 pages. doi: 10.1145/1852102.1852106 (cited on p. 2)

2010
[22]

William Webber, Alistair Moffat, and Justin Zobel. 2008. Statistical power in retrieval experimentation. In Proceedings of the 17th ACM Conference on Infor- mation and Knowledge Management (CIKM ’08). Association for Computing Machinery, Napa Valley, California, USA, 57100580....

2008
[23]

Aslam, and Stephen Robertson

Emine Yilmaz, Javed A. Aslam, and Stephen Robertson. 2008. A new rank cor- relation coefficient for information retrieval. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Informa- tion Retrieval (SIGIR ’08). Association for C...

2008
[2024]

https://arxiv.org/abs/2405.05600 arXiv: 2405.05600 [cs.IR] (cited on p

Can we use large language models to fill relevance judgment holes? (2024). https://arxiv.org/abs/2405.05600 arXiv: 2405.05600 [cs.IR] (cited on p. 1)

2024 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Zahra Abbasiantaeb, Chuan Meng, Leif Azzopardi, and Mohammad Aliannejadi

[2] [2]

Marwah Alaofi, Paul Thomas, Falk Scholer, and Mark Sanderson. 2024. Llms can be fooled into labelling a document as relevant: best café near me; this paper is perfectly relevant. In Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP 2024). Association ...

arXiv 2024

[3] [4]

Guglielmo Faggioli et al. 2023. Perspectives on large language models for rele- vance judgment. In Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR ’23). Association for Computing Machinery, Taipei, Taiwan, 39–50. doi: 10.1145/3578337.3605136 (cited on pp. 1, 2)

arXiv 2023

[4] [5]

Maurice G. Kendall. 1948. Rank Correlation Methods. Charles Griffin and Com- pany Limited (cited on p. 1)

work page 1948

[5] [6]

Mucahid Kutlu, Tamer Elsayed, Maram Hasanain, and Matthew Lease. 2018. When rank order isn’t enough: new statistical-significance-aware correlation measures. In Proceedings of the 27th ACM International Conference on Infor- mation and Knowledge Management (CIKM ’18). Association for Computing Machinery, Torino, Italy, 397–406. isbn: 9781450360142. doi: 10...

doi:10.1145/3269206 2018

[6] [7]

Sean MacAvaney and Luca Soldaini. 2023. One-shot labeling for automatic rele- vance estimation. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’23). Association for Computing Machinery, Taipei, Taiwan, 2230–2235. doi: 10.1145/3539618.35 92032 (cited on p. 1)

doi:10.1145/3539618.35 2023

[7] [8]

David Otero, Javier Parapar, and Álvaro Barreiro. 2023. Relevance feedback for building pooled test collections. Journal of Information Science . doi: 10.1177/01 655515231171085 (cited on p. 2)

work page doi:10.1177/01 2023

[8] [9]

David Otero, Javier Parapar, and Álvaro Barreiro. 2025. Towards reliable test- ing for multiple information retrieval system comparisons. In Advances in Information Retrieval. Springer Nature Switzerland, Cham, 424–439. isbn: 978- 3-031-88711-6 (cited on p. 3)

work page 2025