{"id":"59272f06-15dd-445e-b216-3435290213a5","arxiv_id":"2507.07924","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper adds Type II error metrics to the evaluation of relevance judgment sets and shows that balanced accuracy and Matthews correlation can summarize qrels' discriminative power in one number.","lead":"This paper proposes measuring false-negative statistical errors, not just false-positive ones, when judging whether alternative relevance assessments can tell good search systems from bad ones. It shows that balanced metrics can summarize a set of relevance judgments' discriminative power in a single number.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'Type II error' labels are themselves significance-test outcomes from ground-truth qrels; without showing stability of S_gt, BAC/MCC summarise agreement with a noisy proxy, not qrel discriminative power.","rationale":"The reader's weakest assumption identifies exactly the point on which the central claim rests: ground-truth qrels are treated as the true statistical status of each system pair. I agree that this is the most load-bearing concern. The paper's proposed Type II precision/recall and balanced metrics are defined against S_gt, yet S_gt is itself an output of a significance-testing procedure that depends on the finite topic sample, the run set, and the multiple-comparison correction. If S_gt is unstable, then the 'errors' being measured are partly artifacts of the significance test rather than properties of the candidate qrels. This does not make the proposed metrics useless; it means the paper must demonstrate stability or soften its Type I/II language. The Table 1 inconsistency strengthens the case for a conditional verdict, because the empirical tables currently cannot be checked or trusted as reported. I would keep the reader's CONDITIONAL verdict: the conceptual contribution is reasonable, but the paper needs a stability analysis of S_gt, corrected/released results, and more careful wording distinguishing significance-test outcomes from true population status.","tokens_in":9806,"tokens_out":5899,"duration_ms":62538,"concrete_test":"Take the DL19/DL20 runs and ground-truth qrels, and generate 100 bootstrap resamples of the query set. In each resample, recompute nDCG@10, run the same paired Tukey HSD with the same multiple-comparison correction, and rebuild S_gt/NS_gt; then recompute ①P/①R/②P/②R, BAC and MCC for the same three candidate qrel sets. If the ordering of candidate qrels by BAC/MCC changes across resamples, or if the spread of Type II error rates is large (>0.1), then the metrics are measuring sampling instability of the significance test, not a stable property of qrel quality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is in Section 3: S_gt and NS_gt are defined as the pairs found significantly/non-significantly different using ground-truth qrels, and these are then used as the true binary labels for Type I/II errors. This conflates an estimated significance pattern with the true statistical status of a system pair. A paired Tukey HSD decision (Section 4) is stochastic: with finite topics, the set of significant pairs changes under query resampling, when the set of submitted runs changes, or when the multiple-comparison correction changes. Candidate qrels are therefore being scored for their agreement with a noisy proxy. The central claim that BAC/MCC capture discriminative power is exactly as good as that proxy; until S_gt is shown to be stable, the reported Type II rates and balanced metrics cannot be interpreted as error rates against ground truth. Additionally, Table 1's FP/FN columns cannot be reconciled with the reported ①P/①R/②P/②R values (e.g., DL20 popularity-biased has ①R=0.455 and |S_gt|=714, implying FN≈389, not 77), so the empirical support for the claim is currently not internally consistent.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses how to evaluate qrels generated by alternative relevance-assessment methods against ground-truth human qrels. It argues that existing measures, namely the delta-sensitivity of Faggioli et al. and the ①P/①R precision-recall of Otero et al., neglect false negatives, and it proposes ②P/②R for non-significant system pairs together with balanced accuracy (BAC) and the Matthews correlation coefficient (MCC) as single summaries of discriminative power. The experiments use DL19 and DL20, three candidate-qrels generation methods (zero-shot LLM, percentage sampling, popularity-biased labelling), and paired randomized Tukey HSD tests. The central claim is that quantifying Type II errors provides new insight and that BAC/MCC give an overall, comparable discriminative-power summary.","tokens_in":10023,"tokens_out":6856,"duration_ms":66990,"significance":"If the empirical results were internally consistent, the paper would make a useful, modest contribution: it fills a gap in qrels comparison by drawing attention to false negatives and by offering easily interpretable aggregate metrics, especially MCC. The metric definitions in Section 3 are simple, reproducible, and parameter-free, and the motivation that delta-sensitivity can hide error cancellation (Figure 1) is well taken. However, the main empirical table is internally inconsistent under the paper's own definitions, and the interpretation of the metrics as true Type I/II error rates presumes that ground-truth-qrels significance decisions are stable true labels, which is not demonstrated. The paper is likely to be of interest to the IR evaluation community once these issues are resolved.","major_comments":[{"comment":"The confusion-matrix counts in Table 1 cannot be derived from the reported metrics and set sizes. For example, the DL20 popularity-biased row has |S_gt|=714 and ①R=0.455, which implies TP≈325 and hence FN≈389, not the reported FN=77; conversely, ①P=0.994 and FP=2 imply TP≈331, which is inconsistent with ①R=0.455. Similarly, the DL19 row reports 173+465=638 total pairs, exceeding the C(32,2)=496 pairs available from 32 runs. Because Table 1 is the main empirical support for the paper's claims, these inconsistencies must be corrected or explained.","section":"Table 1 and Section 5"},{"comment":"The paper treats the significance decisions obtained with ground-truth qrels as true binary labels for Type I/II errors. However, these decisions are themselves outcomes of a stochastic hypothesis test (paired randomized Tukey HSD, Section 4) and will vary with the topic sample, the set of submitted runs, and the multiple-comparison correction. Without a stability analysis (e.g., bootstrap over topics, split-half replication, or a sensitivity analysis over the run set), the metrics ②P, BAC, and MCC measure agreement with a possibly noisy proxy rather than with a known ground truth. Since the paper's central claim is about real Type II errors, this limitation should be explicitly addressed or the claims should be softened.","section":"Section 3, sets S_gt and NS_gt; Section 4"},{"comment":"The columns ②P and ②R in Table 2 are swapped relative to the definitions in Section 3. For the #Queries=10 row, TP=443, TN=303, FP=36, FN=929, so ②P = TN/(TN+FP) = 303/339 ≈ 0.894 and ②R = TN/(TN+FN) = 303/1232 ≈ 0.246; the table reports ②P=0.2459 and ②R=0.8938. This inversion directly affects the RQ1 discussion about the 10-query versus 30-query comparison and needs to be fixed.","section":"Table 2 and Section 5, RQ1"}],"minor_comments":[{"comment":"The sentence comparing subsets of 10 and 30 queries says 'an increase in ①P (0.2459→0.2542)'; the values 0.2459 and 0.2542 are ②P in Table 2, not ①P.","section":"Section 5, RQ1"},{"comment":"The phrase '~36% of non-significant differences identified are in fact Type II errors' should be stated more precisely as '1−②P = 0.374, i.e., 37.4% of the pairs predicted non-significant by the candidate qrels are significant according to the ground-truth qrels.' The current wording could be misread as an overall error rate.","section":"Section 5, DL19 popularity-biased row"},{"comment":"Please clarify what exactly the 'Paired Randomised Tukey HSD test' is; Tukey HSD is not normally described as paired, and the details are necessary to reproduce the significance decisions.","section":"Section 4"},{"comment":"The y-axis label 'Sampled Qrels nDCG@10' is not defined in the text; specify that it is the mean nDCG@10 according to the candidate qrels and describe how the example Type I and Type II regions are derived.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a SIGIR short paper and the proposed metrics are a plausible incremental contribution. The internal inconsistencies in Tables 1 and 2 are the main obstacle; they suggest that the experimental pipeline needs to be re-run or carefully documented. The oracle assumption about ground-truth qrels is not fatal for the contribution, but the authors should either provide a stability analysis or explicitly limit the claims to agreement with the chosen ground-truth qrels."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the pair of Type II precision/recall metrics (②P/②R) and the suggestion to use balanced accuracy or MCC as a single-number summary of qrel discriminative power. That is a sensible, incremental extension of Otero et al., and the paper is honest that it builds on that prior work. The definitions are clear, and the demonstration that Δ sensitivity can hide error cancellation is a point worth making.\n\nWhat the paper does well: the experimental setup is straightforward, the sampling experiment in Figure 2 shows the proposed metrics behave reasonably, and the authors correctly identify a real gap in how alternative qrels are validated.\n\nThe soft spots are real, though. First, Table 1 is internally inconsistent. For DL20, |S_gt| + |NS_gt| = 714 + 1711 = 2425, but with 59 runs there are only 1711 pairs; DL19 shows 173 + 465 = 638 vs. 496 pairs. The FP/FN columns also do not match the reported ①R values: with |S_gt| = 714 and ①R = 0.455, FN should be about 389, not 77. This is a load-bearing presentation problem because the table is the main empirical support for RQ1.\n\nSecond, the Type I/II language is too strong. S_gt is an estimated significance pattern from a finite topic set, not a fixed population-level truth. Treating disagreements as genuine Type I/II errors is only valid if S_gt is stable, and no stability analysis is provided. The metrics themselves survive this critique, but the claim that BAC/MCC portray 'discriminative power' in an absolute sense needs qualification: they measure agreement with a possibly noisy proxy. A rephrasing to 'disagreement with ground-truth significance outcomes' or a topic-splitting stability check would fix this.\n\nMinor: the text in Section 5 says '①P' where the numbers 0.2459 and 0.2542 are clearly ②P. Also, no code or data is released, which is a shame for a metrics-focused paper.\n\nThis is a five-page SIGIR resource paper, and the target audience is IR evaluators working with LLM or sampled qrels. It deserves serious review, not desk rejection, but it needs major revision before acceptance: correct the table, soften the error-rate framing or show S_gt stability, and release the evaluation scripts. I'd bring it to a reading group, and I'd cite the ②P/②R definitions once the numbers are trustworthy.","headline":"A useful, small extension to qrel evaluation metrics, but the Type I/II framing overreaches and Table 1's counts do not add up.","tokens_in":10525,"tokens_out":3956,"would_cite":true,"duration_ms":42152,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that measuring only false positives when comparing qrels hides Type II errors, and that balanced accuracy and the Matthews correlation coefficient can summarise the full discriminative power of qrels in one number.","keywords":["hypothesis testing errors","Type II errors","discriminative power","qrels","balanced accuracy","Matthews correlation coefficient","significance testing","information retrieval evaluation"],"falsifier":"Compute Type II precision, Type II recall, balanced accuracy, and MCC for the same TREC runs using two independent sets of human relevance judgments as ground truth. If the proposed error counts diverge materially between the two ground-truth choices, the 'true' Type I and Type II rates are not a stable property of candidate qrels, and the balanced summary inherits that instability.","tokens_in":9612,"feed_emoji":"📊","tokens_out":9974,"duration_ms":98929,"temperature":0.7,"pith_summary":"This paper argues that evaluations of candidate relevance judgments (qrels) have focused almost exclusively on false positives in significance testing and have neglected false negatives, an omission that can send retrieval research in the wrong direction. It recasts each pair of systems as a binary classification problem—significantly different or not, with ground-truth qrels supplying the true labels—and measures the precision and recall of non-significant system pairs along with the established metrics for significant pairs. It then proposes balanced accuracy and the Matthews correlation coefficient as single-number summaries of discriminative power that weigh both error types. Experiments with LLM-generated, subsampled, and popularity-based qrels on TREC Deep Learning data show these measures expose error patterns that previous metrics hide and recommend the Matthews correlation coefficient as a single interpretable summary.","feed_headline":"New metrics quantify false negatives in retrieval evaluation","feed_subtitle":"Precision and recall of non-significant pairs reveal when candidate judgments wrongly discard effective systems.","key_machinery":"The machinery is a binary classification of system pairs driven by the paired randomised Tukey HSD test on nDCG@10 with multiple-comparison correction. For each set of qrels the test splits all system pairs into significant and non-significant sets; comparing candidate-qrels sets against ground-truth sets yields a four-entry confusion matrix of true and false positives and negatives. The formulas $\\mathrm{II}P = |NS_{gt} \\cap NS_{cand}| / |NS_{cand}|$ and $\\mathrm{II}R = |NS_{gt} \\cap NS_{cand}| / |NS_{gt}|$ make Type II errors explicit, while balanced accuracy and the Matthews correlation coefficient collapse the matrix into one number. This machinery is what lets the paper claim that error cancellation under sensitivity can be detected and summarised.","core_discovery":"The paper's central claim is that the discriminative power of qrels is two-sided: how well candidate qrels find significant differences and how well they recognise non-significant differences. Casting significance-test outcomes as binary labels, it defines Type II precision and recall over the sets of non-significant pairs, so that a high Type II precision means few false negatives and a high Type II recall means broad coverage of true non-significant pairs. The results show the popularity-biased labeller and the zero-shot LLM differ markedly on these Type II measures even when conventional significant-pair metrics are uninformative, and that balanced accuracy and MCC condense both error types into one number. The paper's recommendation is to use MCC as a single, interpretable summary of overall discriminative power.","pith_inferences":["An implicit extension of the paper's framing is to treat human ground-truth qrels as themselves a noisy measurement; repeating the protocol with an independent human judgement set would show how much of the reported Type II rate is a property of the candidate qrels rather than of the chosen ground truth.","Because significance decisions depend on the set of runs included in the Tukey HSD comparison, the proposed metrics are relative to a fixed run pool; comparing qrels across different run sets would require the same stabilising choice.","A synthetic benchmark with known system effect sizes could validate the metrics directly: generate systems with known true differences, degrade qrels to controlled levels, and check whether Type II precision and recall and MCC track the injected error rates.","Adopting MCC as a reporting standard would make qrels studies comparable across collections, but the single number hides which error type dominates; pairing it with Type II precision and recall gives both a summary and a diagnosis."],"forward_implications":["A zero sensitivity difference no longer guarantees that candidate and ground-truth qrels agree, because the same count of significant pairs can hide many cancelling Type I and Type II errors.","Type II precision and recall give interpretable error rates even when significant-pair precision and recall are both zero, so weak candidate qrels can still be characterised rather than written off.","Balanced accuracy and MCC let different qrels-generation methods be compared on one scale, avoiding the need to inspect four confusion-matrix entries.","Smaller query sets produce more false negatives through lower statistical power, and Type II precision makes that drop visible.","The same binary-classification view of hypothesis-testing outcomes transfers to other evaluation artefacts, such as train/test splits in recommender evaluation."],"supporting_citations":[{"why":"Introduces precision and recall of significant system pairs, the Type I-only baseline that the paper extends, and the percentage-sampling candidate qrels approach.","marker":"[32]"},{"why":"Defines the sensitivity metric and its difference version, whose error cancellation motivates the paper's new measures.","marker":"[13]"},{"why":"Supplies the paired randomised Tukey HSD procedure used to decide which system pairs are significantly different.","marker":"[39]"},{"why":"Provides the popularity-biased labeller used as a strawman candidate qrels baseline.","marker":"[42]"},{"why":"Gives the prompt design that the paper adapts for the zero-shot LLM candidate qrels.","marker":"[46]"},{"why":"Is the Llama 3 model used to instantiate the zero-shot LLM relevance assessments.","marker":"[12]"},{"why":"Defines balanced accuracy, one of the two proposed single-number summaries of discriminative power.","marker":"[4]"},{"why":"Defines the Matthews correlation coefficient, the paper's recommended single-number summary.","marker":"[29]"}],"fun_headline_variants":["Type II errors exposed in retrieval evaluation","False negatives in qrels now measurable","Balanced metrics reveal hidden qrels power","Two-sided error checks for search system tests","Type II precision and recall for IR evaluation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the significance decisions made with human ground-truth qrels are the true statistical status of each system pair, so when candidate qrels disagree they are making real errors; if human qrels are noisy or the multiple-comparison decision is fragile, the measured false-negative rate is a disagreement with a proxy rather than a genuine Type II error.","fun_headline_variants_meta":{"raw":{"variants":["Type II errors exposed in retrieval evaluation","False negatives in qrels now measurable","Balanced metrics reveal hidden qrels power","Two-sided error checks for search system tests","Type II precision and recall for IR evaluation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000177,"raw_usage":{"total_tokens":1286,"prompt_tokens":930,"completion_tokens":356,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":292}},"tokens_in":546,"tokens_out":356,"duration_ms":4684,"temperature":1.0,"reasoning_tokens":292,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:29:48.261425+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute Type II precision, Type II recall, balanced accuracy, and MCC for the same TREC runs using two independent sets of human relevance judgments as ground truth. If the proposed error counts diverge materially between the two ground-truth choices, the 'true' Type I and Type II rates are not a stable property of candidate qrels, and the balanced summary inherits that instability.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces precision and recall of significant system pairs, the Type I-only baseline that the paper extends, and the percentage-sampling candidate qrels approach."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the sensitivity metric and its difference version, whose error cancellation motivates the paper's new measures."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the paired randomised Tukey HSD procedure used to decide which system pairs are significantly different."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the popularity-biased labeller used as a strawman candidate qrels baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the prompt design that the paper adapts for the zero-shot LLM candidate qrels."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines balanced accuracy, one of the two proposed single-number summaries of discriminative power."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Matthews correlation coefficient, the paper's recommended single-number summary."}],"review_version":1}