Pith. sign in

REVIEW 3 major objections 4 minor 57 references

Measuring Hypothesis Testing Errors in the Evaluation of Retrieval Systems

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that measuring only false positives when comparing qrels hides Type II errors, and that balanced accuracy and the Matthews correlation coefficient can summarise the full discriminative power of qrels in one number.

desk verdict A useful, small extension to qrel evaluation metrics, but the Type I/II framing overreaches and Table 1's counts do not add up. read the letter →

arxiv 2507.07924 v1 pith:YNWHKP66 submitted 2025-07-10 cs.IR

Jack McKechnie , Graham McDonald , Craig Macdonald This is my paper
classification cs.IR
keywords hypothesistestingerrorsTypeIIdiscriminativepowerqrelsbalancedaccuracyMatthewscorrelationcoefficientsignificanceinformationretrievalevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that evaluations of candidate relevance judgments (qrels) have focused almost exclusively on false positives in significance testing and have neglected false negatives, an omission that can send retrieval research in the wrong direction. It recasts each pair of systems as a binary classification problem—significantly different or not, with ground-truth qrels supplying the true labels—and measures the precision and recall of non-significant system pairs along with the established metrics for significant pairs. It then proposes balanced accuracy and the Matthews correlation coefficient as single-number summaries of discriminative power that weigh both error types. Experiments with LLM-generated, subsampled, and popularity-based qrels on TREC Deep Learning data show these measures expose error patterns that previous metrics hide and recommend the Matthews correlation coefficient as a single interpretable summary.

What carries the argument

The machinery is a binary classification of system pairs driven by the paired randomised Tukey HSD test on nDCG@10 with multiple-comparison correction. For each set of qrels the test splits all system pairs into significant and non-significant sets; comparing candidate-qrels sets against ground-truth sets yields a four-entry confusion matrix of true and false positives and negatives. The formulas $\mathrm{II}P = |NS_{gt} \cap NS_{cand}| / |NS_{cand}|$ and $\mathrm{II}R = |NS_{gt} \cap NS_{cand}| / |NS_{gt}|$ make Type II errors explicit, while balanced accuracy and the Matthews correlation coefficient collapse the matrix into one number. This machinery is what lets the paper claim that error cancellation under sensitivity can be detected and summarised.

What would settle it

Compute Type II precision, Type II recall, balanced accuracy, and MCC for the same TREC runs using two independent sets of human relevance judgments as ground truth. If the proposed error counts diverge materially between the two ground-truth choices, the 'true' Type I and Type II rates are not a stable property of candidate qrels, and the balanced summary inherits that instability.

Watch

Extended reading notes

Core claim

The paper's central claim is that the discriminative power of qrels is two-sided: how well candidate qrels find significant differences and how well they recognise non-significant differences. Casting significance-test outcomes as binary labels, it defines Type II precision and recall over the sets of non-significant pairs, so that a high Type II precision means few false negatives and a high Type II recall means broad coverage of true non-significant pairs. The results show the popularity-biased labeller and the zero-shot LLM differ markedly on these Type II measures even when conventional significant-pair metrics are uninformative, and that balanced accuracy and MCC condense both error types into one number. The paper's recommendation is to use MCC as a single, interpretable summary of overall discriminative power.

Load-bearing premise

The load-bearing premise is that the significance decisions made with human ground-truth qrels are the true statistical status of each system pair, so when candidate qrels disagree they are making real errors; if human qrels are noisy or the multiple-comparison decision is fragile, the measured false-negative rate is a disagreement with a proxy rather than a genuine Type II error.

Editorial extensions

If this is right

  • A zero sensitivity difference no longer guarantees that candidate and ground-truth qrels agree, because the same count of significant pairs can hide many cancelling Type I and Type II errors.
  • Type II precision and recall give interpretable error rates even when significant-pair precision and recall are both zero, so weak candidate qrels can still be characterised rather than written off.
  • Balanced accuracy and MCC let different qrels-generation methods be compared on one scale, avoiding the need to inspect four confusion-matrix entries.
  • Smaller query sets produce more false negatives through lower statistical power, and Type II precision makes that drop visible.
  • The same binary-classification view of hypothesis-testing outcomes transfers to other evaluation artefacts, such as train/test splits in recommender evaluation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implicit extension of the paper's framing is to treat human ground-truth qrels as themselves a noisy measurement; repeating the protocol with an independent human judgement set would show how much of the reported Type II rate is a property of the candidate qrels rather than of the chosen ground truth.
  • Because significance decisions depend on the set of runs included in the Tukey HSD comparison, the proposed metrics are relative to a fixed run pool; comparing qrels across different run sets would require the same stabilising choice.
  • A synthetic benchmark with known system effect sizes could validate the metrics directly: generate systems with known true differences, degrade qrels to controlled levels, and check whether Type II precision and recall and MCC track the injected error rates.
  • Adopting MCC as a reporting standard would make qrels studies comparable across collections, but the single number hides which error type dominates; pairing it with Type II precision and recall gives both a summary and a diagnosis.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper addresses how to evaluate qrels generated by alternative relevance-assessment methods against ground-truth human qrels. It argues that existing measures, namely the delta-sensitivity of Faggioli et al. and the ①P/①R precision-recall of Otero et al., neglect false negatives, and it proposes ②P/②R for non-significant system pairs together with balanced accuracy (BAC) and the Matthews correlation coefficient (MCC) as single summaries of discriminative power. The experiments use DL19 and DL20, three candidate-qrels generation methods (zero-shot LLM, percentage sampling, popularity-biased labelling), and paired randomized Tukey HSD tests. The central claim is that quantifying Type II errors provides new insight and that BAC/MCC give an overall, comparable discriminative-power summary.

Significance. If the empirical results were internally consistent, the paper would make a useful, modest contribution: it fills a gap in qrels comparison by drawing attention to false negatives and by offering easily interpretable aggregate metrics, especially MCC. The metric definitions in Section 3 are simple, reproducible, and parameter-free, and the motivation that delta-sensitivity can hide error cancellation (Figure 1) is well taken. However, the main empirical table is internally inconsistent under the paper's own definitions, and the interpretation of the metrics as true Type I/II error rates presumes that ground-truth-qrels significance decisions are stable true labels, which is not demonstrated. The paper is likely to be of interest to the IR evaluation community once these issues are resolved.

major comments (3)
  1. [Table 1 and Section 5] The confusion-matrix counts in Table 1 cannot be derived from the reported metrics and set sizes. For example, the DL20 popularity-biased row has |S_gt|=714 and ①R=0.455, which implies TP≈325 and hence FN≈389, not the reported FN=77; conversely, ①P=0.994 and FP=2 imply TP≈331, which is inconsistent with ①R=0.455. Similarly, the DL19 row reports 173+465=638 total pairs, exceeding the C(32,2)=496 pairs available from 32 runs. Because Table 1 is the main empirical support for the paper's claims, these inconsistencies must be corrected or explained.
  2. [Section 3, sets S_gt and NS_gt; Section 4] The paper treats the significance decisions obtained with ground-truth qrels as true binary labels for Type I/II errors. However, these decisions are themselves outcomes of a stochastic hypothesis test (paired randomized Tukey HSD, Section 4) and will vary with the topic sample, the set of submitted runs, and the multiple-comparison correction. Without a stability analysis (e.g., bootstrap over topics, split-half replication, or a sensitivity analysis over the run set), the metrics ②P, BAC, and MCC measure agreement with a possibly noisy proxy rather than with a known ground truth. Since the paper's central claim is about real Type II errors, this limitation should be explicitly addressed or the claims should be softened.
  3. [Table 2 and Section 5, RQ1] The columns ②P and ②R in Table 2 are swapped relative to the definitions in Section 3. For the #Queries=10 row, TP=443, TN=303, FP=36, FN=929, so ②P = TN/(TN+FP) = 303/339 ≈ 0.894 and ②R = TN/(TN+FN) = 303/1232 ≈ 0.246; the table reports ②P=0.2459 and ②R=0.8938. This inversion directly affects the RQ1 discussion about the 10-query versus 30-query comparison and needs to be fixed.
minor comments (4)
  1. [Section 5, RQ1] The sentence comparing subsets of 10 and 30 queries says 'an increase in ①P (0.2459→0.2542)'; the values 0.2459 and 0.2542 are ②P in Table 2, not ①P.
  2. [Section 5, DL19 popularity-biased row] The phrase '~36% of non-significant differences identified are in fact Type II errors' should be stated more precisely as '1−②P = 0.374, i.e., 37.4% of the pairs predicted non-significant by the candidate qrels are significant according to the ground-truth qrels.' The current wording could be misread as an overall error rate.
  3. [Section 4] Please clarify what exactly the 'Paired Randomised Tukey HSD test' is; Tukey HSD is not normally described as paired, and the details are necessary to reproduce the significance decisions.
  4. [Figure 1] The y-axis label 'Sampled Qrels nDCG@10' is not defined in the text; specify that it is the mean nDCG@10 according to the candidate qrels and describe how the example Type I and Type II regions are derived.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the proposed metrics are standard classification definitions applied post hoc to externally obtained ground-truth and candidate qrels, with no fitted parameter or self-referential equation driving the claims.

full rationale

The paper's derivation chain is self-contained and does not reduce to its own inputs. In Section 3, the sets S_gt, S_cand, NS_gt, and NS_cand are defined from significance-test outcomes using ground-truth and candidate qrels, and the metrics ①P, ①R, ②P, ②R, BAC, and MCC are then defined directly from these sets using standard classification formulas. The candidate qrels are produced by independent methods (zero-shot LLM prompting, percentage sampling, popularity-biased labelling) and are compared against human ground-truth qrels, so no parameter is fitted to the quantity being predicted and no equation's output coincides with its input. The central claims—that Type II errors are worth quantifying and that BAC/MCC summarise both error types—are analytic properties of the confusion matrix applied to this external comparison, not circular derivations. The only self-citation ([30]) appears in related work as background on LLM-based relevance assessment and is not load-bearing. The concern that ground-truth significance labels are themselves a noisy proxy for true statistical status is a validity limitation, not a circularity, and the apparent numerical inconsistencies in Table 1 would be a correctness issue, not a circularity issue. Under the stated rules, no circular step can be identified, so the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The measurement framework rests on treating ground-truth qrels significance outcomes as an oracle, and on the validity of the pairwise significance test. No numeric parameters are fitted to data; the only tuning choices are experimental conditions (sampling percentages, p in the popularity labeller), which come from the data or design rather than being fitted to obtain the reported conclusions.

assumptions (4)
  • domain assumption Human ground-truth qrels are a gold standard; significance outcomes computed on them are the true labels against which candidate qrels are judged.
    Section 1 states human judgements are the gold standard; Section 3 defines S_gt and NS_gt as ground truth. The entire Type I/II error computation depends on this oracle assumption.
  • domain assumption The paired randomised Tukey HSD test with multiple-comparison correction yields valid and reliable pairwise significance decisions.
    Section 4 selects this test; all error metrics inherit its validity. If this test is miscalibrated, the confusion matrix labels are wrong regardless of qrels quality.
  • domain assumption Candidate and ground truth qrels are applied to the same system runs and queries, so outcome differences are attributable to qrels.
    Section 4 fixes DL19/DL20 official runs for both; if run sets or topic sets differed, the comparison would conflate qrel quality with sampling effects.
  • domain assumption Binary classification of system-pair significance adequately captures discriminative power.
    Section 3 casts significance testing as binary classification; this discards effect sizes, p-value magnitudes, and the direction of differences, so the summary metrics inherit this reduction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Measuring Hypothesis Testing Errors in the Evaluation of Retrieval Systems." pith.science (2026). https://pith.science/paper/YNWHKP66

@misc{pith2026250707924,
  author       = {Pith},
  title        = {Pith review of: Measuring Hypothesis Testing Errors in the Evaluation of Retrieval Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YNWHKP66}},
  note         = {Machine review of arXiv:2507.07924}
}
read the original abstract

The evaluation of Information Retrieval (IR) systems typically uses query-document pairs with corresponding human-labelled relevance assessments (qrels). These qrels are used to determine if one system is better than another based on average retrieval performance. Acquiring large volumes of human relevance assessments is expensive. Therefore, more efficient relevance assessment approaches have been proposed, necessitating comparisons between qrels to ascertain their efficacy. Discriminative power, i.e. the ability to correctly identify significant differences between systems, is important for drawing accurate conclusions on the robustness of qrels. Previous work has measured the proportion of pairs of systems that are identified as significantly different and has quantified Type I statistical errors. Type I errors lead to incorrect conclusions due to false positive significance tests. We argue that also identifying Type II errors (false negatives) is important as they lead science in the wrong direction. We quantify Type II errors and propose that balanced classification metrics, such as balanced accuracy, can be used to portray the discriminative power of qrels. We perform experiments using qrels generated using alternative relevance assessment methods to investigate measuring hypothesis testing errors in IR evaluation. We find that additional insights into the discriminative power of qrels can be gained by quantifying Type II errors, and that balanced classification metrics can be used to give an overall summary of discriminative power in one, easily comparable, number.

Figures

Figures reproduced from arXiv: 2507.07924 by the authors.

Figure 1
Figure 1. Illustration of hypothesis testing errors. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Effect of sampling qrels on classification metrics. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

57 extracted references · 50 canonical work pages

  1. [1]

    Zahra Abbasiantaeb, Chuan Meng, Leif Azzopardi, and Mohammad Aliannejadi

  2. [2]

    Javed A Aslam and Robert Savell. 2003. On the effectiveness of evaluating retrieval systems in the absence of relevance judgments. In Proc. of SIGIR

    work page 2003

  3. [3]

    Leonid Boytsov, Anna Belova, and Peter Westfall. 2013. Deciding on an adjust- ment for multiplicity in IR experiments. In Proc. of SIGIR

    work page 2013

  4. [4]

    Kay Henning Brodersen, Cheng Soon Ong, Klaas Enno Stephan, and Joachim M Buhmann. 2010. The balanced accuracy and its posterior distribution. In Proc. of ICPR

    work page 2010

  5. [5]

    Rocío Cañamares and Pablo Castells. 2020. On target item sampling in offline recommender system evaluation. In Proc. of RecSys

    work page 2020

  6. [6]

    Cyril Cleverdon. 1967. The Cranfield tests on index language devices. In ASLIB, Vol. 19

    work page 1967

  7. [7]

    Jacob Cohen. 1960. A coefficient of agreement for nominal scales. EPM 20, 1 (1960)

    work page 1960

  8. [8]

    William S Cooper. 1971. A definition of relevance for information retrieval. Information storage and retrieval 7, 1 (1971)

    work page 1971

Show all 57 references
  1. [9]

    Gordon V Cormack, Christopher R Palmer, and Charles LA Clarke. 1998. Efficient construction of large test collections. In Proc. of SIGIR

    1998

  2. [10]

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2020. Overview of the TREC 2020 deep learning track. arXiv preprint arXiv:2102.07662 (2020)

    2020 arXiv

  3. [11]

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. 2020. Overview of the TREC 2019 deep learning track. arXiv preprint arXiv:2003.07820 (2020)

    2020 arXiv

  4. [12]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  5. [13]

    Guglielmo Faggioli, Laura Dietz, Charles LA Clarke, Gianluca Demartini, Matthias Hagen, Claudia Hauff, Noriko Kando, Evangelos Kanoulas, Martin Potthast, Benno Stein, et al. 2023. Perspectives on large language models for relevance judgment. In Proc. of SIGIR

    2023

  6. [14]

    Guglielmo Faggioli and Nicola Ferro. 2021. System effect estimation by sharding: A comparison between anova approaches to detect significant differences. In Proc. of ECIR

    2021

  7. [15]

    Yu-chen Fan, Yitong Ji, Jie Zhang, and Aixin Sun. 2024. Our model achieves excellent performance on movielens: what does it mean? TOIS 42, 6 (2024)

    2024

  8. [16]

    Nicola Ferro, Yubin Kim, and Mark Sanderson. 2019. Using collection shards to study retrieval performance effect sizes. TOIS 37, 3 (2019)

    2019

  9. [17]

    Nicola Ferro and Mark Sanderson. 2017. Sub-corpora impact on system effective- ness. In Proc. of SIGIR

    2017

  10. [18]

    Nicola Ferro and Mark Sanderson. 2019. Improving the accuracy of system performance estimation by using shards. In Proc. of SIGIR

    2019

  11. [19]

    Nicola Ferro and Mark Sanderson. 2022. How do you test a test? A multifaceted examination of significance tests. In Proc. of WSDM

    2022

  12. [20]

    Douglas A Fitts. 2011. Ethics and animal numbers: informal analyses, uncertain sample sizes, inefficient replications, and type I errors. JAALAS 50, 4 (2011)

    2011

  13. [21]

    Norbert Fuhr. 2018. Some common mistakes in IR evaluation, and how they can be avoided. In ACM SIGIR Forum, Vol. 51

    2018

  14. [22]

    Lei Han, Eddy Maddalena, Alessandro Checco, Cristina Sarasua, Ujwal Gadi- raju, Kevin Roitero, and Gianluca Demartini. 2020. Crowd worker strategies in relevance judgment tasks. In Proc. of WSDM

    2020

  15. [23]

    Claudia Hauff, Djoerd Hiemstra, Leif Azzopardi, and Franciska De Jong. 2010. A case for automatic system evaluation. In Proc. of ECIR

    2010

  16. [24]

    Jason Hsu. 1996. Multiple comparisons: theory and methods . CRC Press

    1996

  17. [25]

    Georg Jäger, Laura S Zilian, Christian Hofer, and Manfred Füllsack. 2019. Crowd- working: working with or against the crowd? J. Econ. Interact. Coord. 14 (2019)

    2019

  18. [26]

    Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika 30, 1-2 (1938)

    1938

  19. [27]

    David E Losada, Javier Parapar, and Álvaro Barreiro. 2016. Feeling lucky? Multi- armed bandits for ordering judgements in pooling-based evaluation. In Proc. of SAC

    2016

  20. [28]

    Sean MacAvaney and Luca Soldaini. 2023. One-shot labeling for automatic relevance estimation. In Proc. of SIGIR

    2023

  21. [29]

    Brian W Matthews. 1975. Comparison of the predicted and observed secondary structure of T4 phage lysozyme. BBA-Protein Structure 405, 2 (1975)

    1975

  22. [30]

    Jack McKechnie, Graham McDonald, and Craig Macdonald. 2025. Context Exam- ple Selection for LLM Generated Relevance Assessments. In Proc. of ECIR

    2025

  23. [31]

    Zaiqiao Meng, Richard McCreadie, Craig Macdonald, and Iadh Ounis. 2020. Ex- ploring data splitting strategies for the evaluation of recommendation models. In Proc. of RecSys

    2020

  24. [32]

    David Otero, Javier Parapar, and Nicola Ferro. 2023. How Discriminative Are Your Qrels? How To Study the Statistical Significance of Document Adjudication Methods. In Proc. of CIKM

    2023

  25. [33]

    Taemin Kim Park. 1993. The nature of relevance in information retrieval: An empirical study. LQ 63, 3 (1993)

    1993

  26. [34]

    Filip Radlinski and Nick Craswell. 2010. Comparing the sensitivity of information retrieval metrics. In Proc. of SIGIR

    2010

  27. [35]

    Hossein A Rahmani, Clemencia Siro, Mohammad Aliannejadi, Nick Craswell, Charles LA Clarke, Guglielmo Faggioli, Bhaskar Mitra, Paul Thomas, and Emine Yilmaz. 2024. Report on the 1st Workshop on Large Language Model for Eval- uation in Information Retrieval (LLM4Eval 2024) at SI...

    2024 arXiv

  28. [36]

    Alan Said and Alejandro Bellogín. 2014. Comparative recommender system evaluation: benchmarking recommendation frameworks. In Proc. of RecSys

    2014

  29. [37]

    Tetsuya Sakai. 2016. Statistical significance, power, and sample sizes: A systematic review of SIGIR and TOIS, 2006-2015. In Proc. of SIGIR

    2016

  30. [38]

    Tetsuya Sakai. 2016. Two sample t-tests for ir evaluation: Student or welch?. In Proc. of SIGIR

    2016

  31. [39]

    Tetsuya Sakai. 2018. Multiple comparison procedures. Laboratory Experiments in Information Retrieval: Sample Sizes, Effect Sizes, and Statistical Power (2018)

    2018

  32. [40]

    Tetsuya Sakai. 2021. On Fuhr’s guideline for IR evaluation. In ACM SIGIR Forum, Vol. 54

    2021

  33. [41]

    Tetsuya Sakai, Noriko Kando, Chuan-Jie Lin, Teruko Mitamura, Hideki Shima, Donghong Ji, Kuang-Hua Chen, and Eric Nyberg. 2008. Overview of the NTCIR-7 ACLIA IR4QA Task.. In NTCIR

    2008

  34. [42]

    Ian Soboroff, Charles Nicholas, and Patrick Cahan. 2001. Ranking retrieval systems without relevance judgments. In Proc. of SIGIR

    2001

  35. [43]

    Karen Sparck Jones and Cornelis Joost Van Rijsbergen. 1976. Information retrieval test collections. J. Doc 32, 1 (1976)

    1976

  36. [44]

    Charles Spearman. 1987. The proof and measurement of association between two things. AJP 100, 3/4 (1987)

    1987

  37. [45]

    Aixin Sun. 2023. Take a fresh look at recommender systems from an evaluation standpoint. In Proc. of SIGIR

    2023

  38. [46]

    Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2024. Large language models can accurately predict searcher preferences. In Proc. of SIGIR

    2024

  39. [47]

    Shivani Upadhyay, Ehsan Kamalloo, and Jimmy Lin. 2024. LLMs Can Patch Up Missing Relevance Judgments in Evaluation. arXiv preprint arXiv:2405.04727 (2024)

    2024 arXiv

  40. [48]

    Shivani Upadhyay, Ronak Pradeep, Nandan Thakur, Nick Craswell, and Jimmy Lin. 2024. UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor. arXiv preprint arXiv:2406.06519 (2024)

    2024 arXiv

  41. [49]

    Julián Urbano, Mónica Marrero, and Diego Martín. 2013. A comparison of the optimality of statistical significance tests for information retrieval evaluation. In Proc. of SIGIR

    2013

  42. [50]

    Julián Urbano, Mónica Marrero, and Diego Martín. 2013. On the measurement of test collection reliability. In Proc. of SIGIR

    2013

  43. [51]

    C Van Rijsbergen. 1979. Information retrieval: Theory and Practice. In Proc. of the joint IBM/University of Newcastle upon Tyne seminar on data base systems , Vol. 79

    1979

  44. [52]

    Voorhees and Donna Harman

    Ellen M. Voorhees and Donna Harman. 1997. Overview of the Fifth Text REtrieval Conference (TREC-5). InProc. of TREC-5, Ellen M. Voorhees and Donna K. Harman (Eds.)

    1997

  45. [53]

    William Webber, Alistair Moffat, and Justin Zobel. 2008. Statistical power in retrieval experimentation. In Proc. of CIKM

    2008

  46. [54]

    John L Williams, Christopher A Hathaway, Kaia L Kloster, and Benjamin H Layne. 1997. Low power, type II errors, and other statistical problems in recent cardiovascular research. AJP-Heart and Circulatory Physiology. 273, 1 (1997)

    1997

  47. [55]

    Shengli Wu and Fabio Crestani. 2003. Methods for ranking information retrieval systems without relevance judgments. In Proc. of SAC

    2003

  48. [56]

    Justin Zobel. 1998. How reliable are the results of large-scale information retrieval experiments?. In Proc. of SIGIR

    1998

  49. [2024]

    Can We Use Large Language Models to Fill Relevance Judgment Holes? arXiv preprint arXiv:2405.05600 (2024)

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.