{"id":"503fbf05-a727-4f8b-a302-83bb606eb6f4","arxiv_id":"2411.13212","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"LLM-generated relevance judgments misrank top retrieval systems and produce many false significant differences relative to human judgments.","lead":"This paper compares human-made and LLM-generated relevance judgments for search systems on TREC Deep Learning collections. It finds that LLM judgments misorder top systems and inflate the number of statistically significant performance differences.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's central causal claim is confounded: human and LLM judgements come from disjoint topic sets with different qrel depths, so observed ranking and significance differences cannot be attributed to LLM label quality without a same-topic comparison.","rationale":"The paper is transparent about the disjointness, which is good, but that transparency does not fix the identification problem. The central conclusion in the abstract goes beyond what the data can support. In a controlled comparison, one would hold topics fixed and vary only the label source. Here both vary, and the qrel depths also vary. The paper's own Table 4 shows that equalizing topic counts reduces false positives but does not eliminate them; however, this is expected under any topic-set effect and cannot distinguish label bias from topic bias. The proposed same-topic check would settle the matter: if discrepancies persist on identical topics, the LLM label source is implicated; if not, the paper's claims need to be substantially softened. I agree with the reader's identification of the topic-set confound as the weakest assumption, and add the per-topic judgment depth as a second, closely related confound. The verdict should remain CONDITIONAL: the paper has a useful measurement approach and transparent reporting, but its headline claim should not be accepted without this additional experiment.","tokens_in":8580,"tokens_out":6461,"duration_ms":69887,"concrete_test":"Run the same SynDL GPT-4 labeling protocol on the 43 DL-19 topics that have human judgements (T_H), producing LLM qrels on exactly those topics. Recompute the top-weighted rank correlations (tau_AP, RBO p=0.7) and the pairwise two-sided Wilcoxon decisions for AP and NDCG, comparing human qrels vs. LLM qrels on the identical topic set. Also subsample human qrels to the same per-topic judgment depth as the LLM qrels for a depth-matched comparison. If the FP rates and correlation drops persist, the paper's claims about LLM label quality are supported; if they largely disappear, the disjoint topic sets and qrel depth differences explain the results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim — that LLM-based judgements are unfair at ranking top systems and produce excessive false positives — requires that observed differences between human- and LLM-based evaluations are caused by the label source. Section 3 explicitly states 'We do not assume any overlap between T_H and T_LLM, in fact, they may be completely disjoint sets,' and Table 1 confirms T_H ∩ T_LLM = ∅ across all five years. Additionally, the average number of judgements per topic differs substantially (e.g., DL-19: 215 vs. 70), so the two qrel sets differ in topic selection, topic count, and judgment depth. Any of these factors can change per-topic AP/NDCG vectors and therefore system rankings and Wilcoxon significance decisions, even if LLM labels were perfect. The undersampling experiment in Table 4 matches only the number of topics, not topic content or per-topic depth, so it does not remove the confound. Unless the authors show that T_H and T_LLM are exchangeable in difficulty and in qrel depth, the abstract's causal language is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares official TREC human relevance judgments with GPT-4 generated judgments from the SynDL collection across five Deep Learning Track datasets (DL-19 through DL-23). It evaluates top-weighted rank correlations (τ_AP and RBO) alongside Kendall's τ, analyzes per-run changes in ranking positions, and compares pairwise Wilcoxon signed-rank test decisions between human and LLM qrels. The authors report that top-heavy correlations are much lower than Kendall's τ, that many runs change rank substantially, and that LLM qrels yield high false-positive rates in significance testing. An undersampling experiment that equalizes the number of topics reduces but does not eliminate the false-positive rates. The paper concludes that LLMs are not yet ready to replace human relevance judgments in offline IR evaluation.","tokens_in":8812,"tokens_out":6084,"duration_ms":57642,"significance":"If the causal interpretation were valid, this would be an important negative result for the use of LLM-generated test collections, directly challenging the prevailing reliance on rank correlations as evidence of reliability. The paper usefully demonstrates that high Kendall's τ does not imply reliable top-system comparisons or preserved significance decisions, and the use of multiple TREC DL collections lends breadth to the analysis. The authors also make their methodology transparent and their code and data presumably available. However, the paper's central causal claim is undermined by a fundamental design confound: the human and LLM judgments come from disjoint topic sets, so the observed differences in rankings and significance decisions cannot be unambiguously attributed to the label source.","major_comments":[{"comment":"The comparison between Q_H and Q_LLM is confounded by the disjointness of the topic sets. The paper states 'We do not assume any overlap between T_H and T_LLM, in fact, they may be completely disjoint sets,' and Table 1 confirms T_H ∩ T_LLM = ∅ for all five years. The topic sets also differ substantially in size (e.g., DL-19: 43 vs. 157 topics) and in average judgments per topic (e.g., DL-22: 5084 vs. 133). Under this design, differences in per-topic score vectors, and therefore differences in system rankings and pairwise Wilcoxon significance decisions, could arise purely from differences in topic difficulty and judgment depth even if the LLM labels were perfect. The undersampling experiment in Table 4 controls only the number of topics, not their content or depth, so it does not remove this confound. The abstract's causal claims ('LLM-based judgements are unfair at ranking top-performing systems' and 'exceedingly high rate of false positives') are therefore not supported by the evidence as presented. The authors must either demonstrate that T_H and T_LLM are exchangeable in difficulty and depth, or substantially weaken the causal language throughout the paper.","section":"Section 3, Table 1"},{"comment":"The undersampled results are reported as averages over 200 random subsets of T_LLM, but the paper provides no confidence intervals, standard deviations, or other measures of variability for these averages. For datasets with small |T_H| (e.g., 43 in DL-19), the distribution of the false-positive and true-positive rates across random subsets is likely to be wide, so the reader cannot determine whether the differences between Table 3 and Table 4 are stable or within sampling noise. Please report the full distribution, percentile intervals, or standard errors for the undersampled estimates.","section":"Table 4 and Section 3.2"},{"comment":"The RBO(p=0.7) results are strikingly inconsistent across metrics and years, for example DL-19: AP RBO=0.11 vs. NDCG RBO=0.93, and DL-21: AP RBO=0.52 vs. NDCG RBO=0.97. The paper does not discuss these large discrepancies, yet they are directly relevant to the claim that LLM-based judgments fail to preserve top-system rankings; the conclusion appears to be metric-dependent. Please provide an explanation for these inconsistencies and discuss their implications for the overall fairness claim.","section":"Section 2, Table 2"}],"minor_comments":[{"comment":"The text 'We plot the distribution of these drops in Figure 1' refers to the boxplot of significance drops, but in the manuscript that figure is numbered Figure 3; the cross-reference is incorrect.","section":"Section 3.2"},{"comment":"The table headers describe the qrels as 'synthetic'; using 'LLM-generated' or 'SynDL' would be more precise and avoid confusion with other types of synthetic qrels.","section":"Tables 3 and 4"},{"comment":"Figure 2 shows per-run rank changes only for DL-2020; the paper does not justify why this year was selected instead of the other years.","section":"Section 2.1"},{"comment":"Reference [22] contains a typographical error: '57100580' should likely be '571–580'.","section":"References"},{"comment":"The footnote for the RBO parameter p=0.7 cites Reference [15], Section 4.5, but Reference [15] (Thomas et al., 2024) does not appear to discuss RBO persistence; a citation to the RBO paper (Reference [21]) or a direct justification of the parameter choice would be clearer.","section":"Section 2, footnote 1"}],"recommendation":"reject","confidential_remarks":"The authors are transparent about the disjointness of the topic sets, but the paper's conclusions overreach the evidence. The confound is structural and cannot be fixed with the existing data; even a major revision that reframes all causal claims would leave the contribution as a documentation of differences between two topic sets rather than a demonstration of LLM-label deficiencies. As a result, I do not see a viable path to publication at a major IR venue without a new experimental design that includes overlapping topics or a direct exchangeability analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth reading, with a caveat that matters. The paper is a useful caution about using LLM-generated qrels for IR evaluation, but its strongest claim—that LLM judgments are unfair and produce systematic false positives—is not fully identified, because the human and LLM judgments come from disjoint topic sets with different qrel depths.\n\nThe paper does something new: instead of stopping at Kendall's tau between human and GPT-4 qrels, it applies top-weighted rank correlations and pairwise significance-testing to the SynDL qrels. The findings are striking. Human/LLM Kendall's tau looks fine, but tau_AP and especially RBO are much lower, meaning top-of-leaderboard distinctions are not preserved. And in the Wilcoxon pairwise comparisons, LLM qrels produce a very high false-positive rate (AP FP 71–89% before undersampling). Those numbers are a useful warning to anyone treating LLM-generated qrels as drop-in replacements.\n\nCredit where due: the experiments are transparent, use public TREC Deep Learning collections and the released SynDL qrels, and the undersampling check in Table 4 is an honest attempt to address the sample-size objection. The citation pattern is clean; the self-citations are methodological and appropriate.\n\nThe soft spot is real and load-bearing, and the paper's own text exposes it. T_H and T_LLM are disjoint by construction, and the per-topic judgment depths differ substantially (DL-19: 215 vs. 70). So a ranking change or a significance decision could be due to the label source or due to a different set of topics with different difficulty and qrel depth. The undersampling experiment matches only topic count, not topic identity or per-topic depth, so the confound remains. The abstract's \"LLM-based judgements are unfair\" is too strong for what is identified. A same-topic comparison, or evidence that the two topic sets are exchangeable in difficulty and depth, would be needed to support the causal claim. Minor issues: no variance around the undersampled averages, and RBO is reported at a single persistence parameter.\n\nVerdict for a desk editor: send it out. The paper is useful to the IR evaluation community despite the confound; it shows empirically that high overall correlation does not imply reliable top-system ranking or trustworthy significance decisions when using SynDL-style qrels. I would not cite the central causal claim as settled, but the cautionary pattern is citable. A serious referee can ask for the same-topic evidence or a framed-down conclusion.","headline":"Useful caution for IR evaluation, but the strongest claim about LLM judgment quality is undercut by comparing disjoint topic sets with different qrel depths.","tokens_in":9249,"tokens_out":4798,"would_cite":true,"duration_ms":55491,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM relevance labels misrank top systems and inflate significance differences.","keywords":["relevance assessment","large language models","test collections","retrieval evaluation","top-heavy rank correlation","statistical significance","false positives","SynDL"],"falsifier":"Re-run the comparison on a benchmark where the same topics receive both human and LLM judgments, and compute the pairwise significance agreement on that overlapping set. If the false-positive rate collapses toward zero, the paper's conclusion about LLM labels as a source of bias would fail; if it persists, the conclusion would be confirmed on a cleaner design.","tokens_in":8424,"feed_emoji":"📉","tokens_out":3680,"duration_ms":36234,"temperature":0.7,"pith_summary":"The paper asks whether relevance judgments written by a large language model can stand in for human judgments when evaluating search systems offline. Its answer is that they cannot yet. Across five years of Deep Learning Track test collections, GPT-4-generated labels rank top-performing systems noticeably differently from official human labels: top-weighted correlation measures drop well below standard Kendall's tau, and individual runs move by five to more than forty positions. The paper further shows that pairwise significance tests run on LLM labels mark many pairs of systems as significantly different when the human labels say they are not, an outcome it interprets as a bias toward false positives. The conclusion is that LLM-based relevance assessments are not ready to replace human gold judgments in retrieval evaluation.","feed_headline":"LLM labels misrank top systems and inflate significance","feed_subtitle":"Top-heavy correlations drop and pairwise tests flag differences human labels say are not there.","key_machinery":"The argument runs on the SynDL synthetic test collection, whose LLM labels were produced by GPT-4, compared against official TREC human judgments across DL-2019 to DL-2023, with disjoint topic sets. Three instruments carry the analysis: top-weighted rank correlations (tau_AP and RBO with p = 0.7) that penalize disagreements near the top of the system ranking; a fairness analysis of per-run rank changes; and a binary classification of pairwise statistical significance decisions from a two-sided Wilcoxon signed-rank test at alpha = 0.05, summarized as true/false positives and negatives, with an undersampling control that equalizes topic counts between human and LLM label sets.","core_discovery":"Using the SynDL collection, where GPT-4 supplied relevance labels for topics disjoint from the human-judged topics, the authors show three results. First, agreement between LLM and human system rankings shrinks as attention moves to the top: tau_AP and RBO values fall well below Kendall's tau, so the usual correlation numbers overstate how well LLM labels identify the best systems. Second, on a per-run basis most runs change position under LLM labels, with drops of up to around fifty places, implying the evaluation is not fair across systems. Third, pairwise Wilcoxon signed-rank tests with alpha = 0.05 classify far more pairs as significantly different under LLM labels than under human labels; after equalizing the number of topics by undersampling, false positives fall but remain high, and false negatives rise. The authors conclude that LLM-generated qrels do not preserve ranking differences among top systems or statistical significance decisions.","pith_inferences":["If the disjoint-topic design contributes to the disagreement, an overlapping benchmark where humans and LLMs label the same topics would give a cleaner estimate of label quality; this is a direct test the authors' setup leaves open.","LLM labels might still serve as a cheap screening step for building collections, as long as final leaderboard decisions and significance claims are reserved for human judgments or a calibrated hybrid.","Extending the analysis to other LLMs and graded relevance scales would show whether the unfairness at the top is a GPT-4-specific artifact or a general property of LLM-based assessment.","A practical follow-up could threshold on the LLM's confidence or agreement across repeated samples to flag pairs whose significance decision is unstable."],"forward_implications":["Researchers who use LLM-generated labels to compare top-performing systems will frequently get the ordering wrong, even when overall Kendall's tau looks acceptable.","Statistical comparisons built on LLM labels will over-report significant improvements, leading to published claims of differences that human labels do not support.","Equalizing the number of topics reduces but does not eliminate the false-positive problem, so the bias is not purely a sample-size artifact.","When false negatives occur they concentrate in a few runs, meaning some systems are systematically disadvantaged by LLM evaluation.","NDCG appears more stable than AP under LLM labels, but both metrics still show substantial disagreement."],"supporting_citations":[{"why":"Supplies the SynDL dataset with GPT-4-generated labels for topics disjoint from the human-judged topics, the experimental ground for all results.","marker":"[13]"},{"why":"A large-scale prior study showing high ranking correlation of LLM relevance assessments, which is the baseline claim this paper presses on.","marker":"[17]"},{"why":"Earlier perspectives on using LLMs for relevance judgment, providing the motivation and comparison point for the field.","marker":"[4]"},{"why":"Source of the LLM-judgment approach whose RBO parameter and top-focused analysis are used in the paper.","marker":"[15]"},{"why":"Defines RBO, the top-weighted rank correlation used to measure agreement on top systems.","marker":"[21]"},{"why":"Defines tau_AP, the top-weighted correlation used alongside RBO.","marker":"[23]"},{"why":"Supports the choice of the Wilcoxon signed-rank test for pairwise significance testing.","marker":"[11]"},{"why":"Supports the claim that more topics increase statistical power, motivating the undersampling control.","marker":"[3]"},{"why":"Supports the sample-size and power argument and the reliability framing of the evaluation.","marker":"[14]"}],"fun_headline_variants":["LLM relevance labels bias top-system rankings","LLM assessments mislead on top systems and significance","LLM labels hurt fair ranking and inflate significance","LLM qrels misrank top systems and create false significance","Automatic LLM judgments unfair for top retrieval systems"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Human and LLM labels are applied to completely different sets of topics, so the comparison assumes those topic sets are interchangeable; if the LLM-judged topics are systematically easier or harder in a way that affects significance, the reported false-positive rates would not be attributable to the labels themselves.","fun_headline_variants_meta":{"raw":{"variants":["LLM relevance labels bias top-system rankings","LLM assessments mislead on top systems and significance","LLM labels hurt fair ranking and inflate significance","LLM qrels misrank top systems and create false significance","Automatic LLM judgments unfair for top retrieval systems"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00023,"raw_usage":{"total_tokens":1483,"prompt_tokens":946,"completion_tokens":537,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":461}},"tokens_in":562,"tokens_out":537,"duration_ms":5742,"temperature":1.0,"reasoning_tokens":461,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:41:11.479438+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the comparison on a benchmark where the same topics receive both human and LLM judgments, and compute the pairwise significance agreement on that overlapping set. If the false-positive rate collapses toward zero, the paper's conclusion about LLM labels as a source of bias would fail; if it persists, the conclusion would be confirmed on a cleaner design.","supporting_citations":[],"review_version":1}