Pith. sign in

REVIEW 3 major objections 5 minor 22 references

Impact of Shallow vs. Deep Relevance Judgments on BERT-based Reranking Models

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Shallow relevance judgments beat deep ones for training BERT rerankers in this study.

desk verdict Useful negative-sampling finding, but the deep-vs-shallow headline is built on shallow test sets and mismatched query pools. read the letter →

arxiv 2506.23191 v1 pith:IFAMP4A6 submitted 2025-06-29 cs.IR

Gabriel Iturra-Bocaz , Danny Vo , Petra Galuscakova This is my paper · ORCID
classification cs.IR
keywords neuralinformationretrievalBERTrerankingrelevancejudgmentsdeepshallownegativesamplingMSMARCOLongEval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether training data for neural rerankers should contain many queries with one or two relevance judgments each (shallow) or fewer queries with many judgments each (deep). Using BERT-based rerankers over BM25 top-10 candidates, it compares matched-size training sets drawn from MS MARCO V1, MS MARCO V2, and LongEval. It finds that shallow-judged sets consistently outperform deep-judged sets at equal training-instance counts, and that shallow sets need far fewer instances—around 1,000 to 2,000 in these settings—to beat the BM25 baseline. The paper also shows that adding negative samples improves deep-set training up to a point, with a positive-to-negative ratio near 1:8 giving the best gains. If the finding holds, annotation budgets for training data are better spent on broad query coverage than on exhaustive per-query judgment.

What carries the argument

The load-bearing object is the query-to-judgment ratio of a training set, expressed as 'query count / judgments per query' (for example, shallow 2,500/2 versus deep 50/100) while holding the total number of training instances fixed. The reranker is MonoBERT, a BERT sequence-classification model that scores each query-document pair pointwise, fine-tuned on top-10 BM25 candidates so that reranking happens at depth 10. The comparison's validity rests on matching total training-instance counts across the deep and shallow regimes, and on drawing negatives from BM25 results with the top 10 excluded to avoid false negatives.

What would settle it

Take a single set of queries from one corpus, split them randomly into two groups of matched difficulty, and build deep and shallow training sets from those groups with equal total instance counts (for example, 50 queries with 100 judgments each versus 2,500 queries with 2 judgments each, drawn from the same query distribution). Train the same BERT reranker on both and evaluate on the same test queries; if the deep-set model matches or beats the shallow-set model, the paper's central claim fails. A simpler version would rerun the paper's protocol but subsample the deep queries so their pool matches the shallow queries' source and difficulty.

Watch

Extended reading notes

Core claim

The central claim is that, for training a BERT-based reranker, shallow relevance judgments are a more effective use of a fixed annotation budget than deep ones. In every experiment, when the total number of training instances is held equal, the model trained on many lightly judged queries outperforms the model trained on few heavily judged queries, and the deep-trained models even fall below the BM25 baseline in most MS MARCO configurations. The paper attributes the effect to the broader range of contexts and information needs that shallow sets expose the model to, improving generalization. A secondary claim is that negative judgments—cheap to generate automatically—can partially compensate for judgment depth: adding negatives helps the deep sets most, with performance peaking around a 1:8 positive-to-negative ratio before degrading at more imbalanced ratios.

Load-bearing premise

The deep and shallow training sets are assumed to differ only in the number of queries versus judgments per query, but the two kinds actually come from different query pools—evaluation-campaign queries for deep sets and search-engine log queries for shallow sets—so the reported gaps could reflect query difficulty or topic distribution rather than judgment depth.

Editorial extensions

If this is right

  • Broad query coverage, not exhaustive per-query judgment, is what drives reranker performance when training budgets are fixed.
  • Annotation projects aiming to train neural rerankers should prioritize collecting one or two judgments for many diverse queries over hundreds of judgments for a few queries.
  • Existing deep evaluation collections can be reused for training only if supplemented with enough negative samples, at a ratio around 1:8 in the MS MARCO V1 setting.
  • Shallow sets of roughly 1,000 to 2,000 instances are enough to surpass the BM25 baseline in the tested collections, a lower threshold than earlier estimates.
  • Performance gains from adding negatives are not monotonic: overly imbalanced positive-to-negative ratios add noise and hurt both shallow and deep sets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer that the query-source confound is the main threat: if deep and shallow queries were drawn from the same pool, the performance gap could shrink or disappear, which would change the recommendation about annotation budgets.
  • A testable extension would apply the same shallow-versus-deep comparison to a reranker with stronger few-shot capabilities; the paper notes more advanced models might need even fewer instances, which could alter the break-even point.
  • We infer that evaluation pools could be made slightly shallower without losing their evaluative power, letting a single annotation effort serve both training and evaluation—an idea the paper hints at but does not test.
  • A useful follow-up would measure annotation cost directly (time or money per judgment) alongside performance, converting the effectiveness gap into a cost-effectiveness comparison.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper compares BERT-based reranking models (MonoBERT) trained on 'shallow' relevance-judgment datasets (many queries, few judgments per query, e.g., MS MARCO Train) with models trained on 'deep' datasets (fewer queries, many judgments per query, e.g., TREC DL track qrels), using MS MARCO V1/V2 and LongEval. Training-set sizes are matched in terms of total positive training instances in several experiments. The authors report that shallow-trained rerankers outperform deep-trained ones at equal instance counts, that shallow training reaches the BM25 baseline with fewer instances, and that increasing negative samples partially mitigates the disadvantage of deep judgments. They conclude that shallow training sets consistently outperform deep training sets and generalize better.

Significance. If the central finding held as stated, it would be practically valuable: it would justify building training sets with broad query coverage and few judgments per query, and would inform how to reuse expensive TREC-style pooling judgments for neural ranking training. The paper ships code and uses standard collections, significance tests, and a simple, reproducible MonoBERT setup. However, the causal interpretation is weakened by the confounding of judgment depth with query-pool origin and by the fact that all evaluation is done on shallow test collections; the overbroad wording of the conclusion is also contradicted by one of the paper's own tables. With appropriate caveats and additional analyses, the empirical observations remain useful for the IR community.

major comments (3)
  1. [Section 3.1, 3.2; Tables 3, 6, 8] The deep and shallow training sets do not differ only in the number of judgments per query; they also come from different query pools (TREC Deep Learning track topics or LongEval Heldout for deep, MS MARCO Train query logs or LongEval Train for shallow). These pools may differ in topic distribution, query difficulty, and annotation methodology (pooling versus user clicks vs. single-label collection). Consequently, the performance gaps in Tables 3, 6, and 8 could reflect query selection rather than judgment depth. The manuscript does not control for query distribution or provide evidence that the query pools are comparable. This is load-bearing for the causal claim that 'shallow judgments' cause better generalization, and it should be addressed by either matching queries across conditions, analyzing query difficulty, or substantially tempering the causal language.
  2. [Section 3.1, 3.2, 5; Tables 3, 6, 8] Every evaluation in the paper uses test collections that are themselves shallow: MS MARCO V1 Dev2 and V2 Dev1 have on average one relevant passage per query, and the LongEval test sets report roughly four relevant documents per query (Section 3.1, 3.2). A deep-trained reranker that correctly retrieves an additional relevant document not present in the shallow qrels is scored as a false positive, which systematically disadvantages deep training. The authors acknowledge that MS MARCO V2 may contain many unjudged relevant documents and is unreliable for evaluation (Section 5), yet no deep test set (e.g., TREC DL 2019/2020 official graded qrels) is used. This makes the claim that shallow training 'consistently outperforms' deep training a claim about performance on shallow test labels only, not a general property. The paper should either add an evaluation on a deep test set or explicitly restrict the conclusion to shallow test environments.
  3. [Section 6 and Table 8] The concluding sentence 'Shallow training sets consistently outperform deep training sets in all our experiments' is not supported by the paper's own data. In Table 8, the deep 31/100 model (3,100 instances) outperforms the shallow 754/2 model (1,508 instances) on both LongEval ST and LT across all metrics. While the instance counts differ, this is one of the experiments reported, and the universal wording is therefore inaccurate. The conclusion should be restated to reflect the matched-instance comparisons on MS MARCO and the more qualified LongEval result, e.g., 'shallow training is generally more sample-efficient in our experiments,' along with the caveat about test-set depth.
minor comments (5)
  1. [Section 3.1, Table 1 footnote] The exclusion of two development queries with more than 40,000 relevant passages is mentioned only in a footnote; since this materially changes the deep dataset statistics, the exclusion criterion and its impact on the results should be stated in the main text.
  2. [Section 4.2 and Table 3] Significance tests are reported only against the BM25 baseline (asterisks), not as pairwise comparisons between deep and shallow conditions. Adding pairwise significance tests (or at least confidence intervals) would strengthen the claim that observed differences between deep and shallow models are meaningful.
  3. [Table 7 caption] The caption contains a duplicated word: 'depending depending on the number of training instances.'
  4. [References] Reference [15] lists the arXiv ID as '11910.14424'; this appears to be a typo for '1910.14424'.
  5. [Appendix, Figure 1 caption] The caption mentions '2011-2022' when the intended range is presumably '2021-2022' for MS MARCO V2; please correct.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical comparison with no derivation chain that reduces to its own inputs.

full rationale

The paper is an empirical study, not a derivation-based argument. It compares BERT rerankers fine-tuned on hand-constructed shallow and deep training subsets and evaluated on standard held-out MS MARCO Dev sets and LongEval Test sets. No equation maps a fitted parameter onto a predicted quantity, no quantity is defined in terms of the claimed outcome, and no fitted value is later renamed as a prediction. The central conclusion in Section 6, that shallow training sets outperform deep training sets, is a measured generalization result rather than an identity forced by construction. The only self-adjacent element is that one co-author is also an organizer/author of LongEval, cited as [10], but using that publicly available benchmark as an external evaluation collection is not load-bearing in a circular way: the paper does not invoke the citation to define its independent variable or to rule out alternatives. Limitations stated in Section 5, such as the scarcity of collections containing both deep and shallow judgments and the warning that MS MARCO V2 may contain many unjudged relevant documents making it unreliable for evaluation, are validity threats rather than circularity; indeed, the paper itself reports a result that contradicts its universal conclusion in Table 8, where deep 3,100/100 outperforms shallow 1,508 on both LongEval test sets. That overgeneralization is a correctness concern, not a circular reduction. No self-citation chain, uniqueness theorem, or ansatz is used to force the claim, so the honest verdict is no significant circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No fitted parameters enter the comparison. Hyperparameters such as batch size 8, learning rate 2e-5, 10 epochs, and BM25 depth 10 are standard and held constant across conditions; they do not change the relative shallow-vs-deep comparison.

assumptions (5)
  • domain assumption The TREC DL and LongEval Heldout query sets are representative of the deep-judged paradigm, and the MS MARCO Train and LongEval Train query sets are representative of the shallow-judged paradigm.
    The paper's central comparison assumes these two groups differ only in judgment depth, not in query distribution or annotation methodology (Section 3.1, Section 3.2).
  • domain assumption Matching the total number of training instances across deep and shallow sets isolates the effect of judgment depth.
    The paper constructs subsets with equal instance counts (Tables 3 and 6); the alternative of matching annotation budget is not considered, so the comparison is conditional on this choice.
  • domain assumption MS MARCO V2 Dev judgments are sufficiently complete for evaluating rerankers.
    Section 5 states MS MARCO V2 'may contain many unjudged relevant documents, making it unreliable for evaluation' [9,17], yet V2 results are used in Tables 6 and 7.
  • domain assumption BERT fine-tuned from scratch on these subsets is representative of neural rerankers.
    The paper uses a single architecture (bert-base-uncased, MonoBERT); conclusions may not generalize to other rankers (Section 4.1).
  • domain assumption Negative samples drawn from BM25 results, with the top 10 excluded, are true negatives.
    Section 4.1 assumes this sampling avoids false negatives; if top-ranked BM25 documents are actually relevant, the negative sampling is noisy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Impact of Shallow vs. Deep Relevance Judgments on BERT-based Reranking Models." pith.science (2026). https://pith.science/paper/IFAMP4A6

@misc{pith2026250623191,
  author       = {Pith},
  title        = {Pith review of: Impact of Shallow vs. Deep Relevance Judgments on BERT-based Reranking Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IFAMP4A6}},
  note         = {Machine review of arXiv:2506.23191}
}
read the original abstract

This paper investigates the impact of shallow versus deep relevance judgments on the performance of BERT-based reranking models in neural Information Retrieval. Shallow-judged datasets, characterized by numerous queries each with few relevance judgments, and deep-judged datasets, involving fewer queries with extensive relevance judgments, are compared. The research assesses how these datasets affect the performance of BERT-based reranking models trained on them. The experiments are run on the MS MARCO and LongEval collections. Results indicate that shallow-judged datasets generally enhance generalization and effectiveness of reranking models due to a broader range of available contexts. The disadvantage of the deep-judged datasets might be mitigated by a larger number of negative training examples.

Figures

Figures reproduced from arXiv: 2506.23191 by the authors.

Figure 1
Figure 1. Distribution of the number of positive relevance passages (Qrels) per query in the TREC DL datasets from 2019-2020 [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Distribution of the number of positive Qrels per query in the MS MARCO V1 and V2 collections, for both training and [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Distribution of the number of positive relevance passages (Qrels) per query in the LongEval datasets. Each histogram [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 19 canonical work pages

  1. [1]

    Sophia Althammer, Guido Zuccon, Sebastian Hofstätter, Suzan Verberne, and Allan Hanbury. 2023. Annotating Data for Fine-Tuning a Neural Ranker? Current Active Learning Strategies are not Better than Random Selection. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Reg...

    work page 2023

  2. [2]

    Yinqiong Cai, Jiafeng Guo, Yixing Fan, Qingyao Ai, Ruqing Zhang, and Xueqi Cheng. 2022. Hard Negatives or False Negatives: Correcting Pooling Bias in Training Neural Ranking Models. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management (CIKM’22) . 118–127

    work page 2022

  3. [3]

    Manning, Prabhakar Raghavan

    Hinrich Schütze Christopher D. Manning, Prabhakar Raghavan. 2008.Introduction to Information Retrieval. Cambridge University Press

    work page 2008

  4. [4]

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Danie Campos. 2020. Overview of the TREC 2020 Deep Learning Track. In Text REtrieval Conference (TREC’20). NIST

    work page 2020

  5. [5]

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Jimmy Lin. 2021. Overview of the TREC 2021 Deep Learning Track. In Text REtrieval Conference (TREC’21). NIST

    work page 2021

  6. [6]

    Voorhees, and Ian Soboroff

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, Jimmy Lin, Ellen M. Voorhees, and Ian Soboroff. 2022. Overview of the TREC 2022 Deep Learning Track. In Text REtrieval Conference (TREC’22). NIST

    work page 2022

  7. [7]

    Voorhees

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. 2019. Overview of the TREC 2019 Deep Learning Track. In Text REtrieval Conference (TREC’19). NIST

    work page 2019

  8. [8]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. (2019). arXiv:1810.04805 [cs.CL]

    arXiv 2019

Show all 22 references
  1. [9]

    Maik Fröbe, Christopher Akiki, Martin Potthast, and Matthias Hagen. 2022. Noise- Reduction for Automatically Transferred Relevance Judgments. In Experimental IR Meets Multilinguality, Multimodality, and Interaction (CLEF’22) . 48–61

    2022

  2. [10]

    Petra Galuščáková, Romain Deveaud, Gabriela González Sáez, Philippe Mulhem, Lorraine Goeuriot, Florina Piroi, and Martin Popel. 2023. LongEval-Retrieval: French-English Dynamic Test Collection for Continuous Web Search Evaluation. In Proceedings of the 46th International ACM S...

    2023

  3. [11]

    Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini: A Python Toolkit for Reproducible Informa- tion Retrieval Research with Sparse and Dense Representations. In Proceedings of the 44th Annual International ACM SIGIR Co...

    2021

  4. [12]

    Shane Culpepper

    Xiaolu Lu, Alistair Moffat, and J. Shane Culpepper. 2017. Can Deep Effectiveness Metrics Be Evaluated Using Shallow Judgment Pools?. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’17) . 35–44

    2017

  5. [13]

    Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage Re-ranking with BERT. (2019). arXiv:1901.04085 [cs.IR]

    2019 arXiv

  6. [14]

    Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Docu- ment Ranking with a Pretrained Sequence-to-Sequence Model. In Findings of the Association for Computational Linguistics (EMNLP’20) . 708–718

    2020

  7. [15]

    Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. 2019. Multi-Stage Document Ranking with BERT. (2019). arXiv:11910.14424 [cs.IR]

    2019 arXiv

  8. [16]

    Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxi- ang Dong, Hua Wu, and Haifeng Wang. 2021. RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2021 Conference of the North Am...

    2021

  9. [17]

    Voorhees, Nick Craswell, and Jimmy Lin

    Ellen M. Voorhees, Nick Craswell, and Jimmy Lin. 2022. Too Many Relevants: Whither Cranfield Test Collections?. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’22). 2970–2980

    2022

  10. [18]

    Bennett, Junaid Ahmed, and Arnold Overwijk

    Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate Nearest Neighbor Neg- ative Contrastive Learning for Dense Text Retrieval. In International Conference on Learning Representations (ICLR’21)

    2021

  11. [19]

    Andrew Yates, Rodrigo Nogueira, and Jimmy Lin. 2021. Pretrained transformers for text ranking: BERT and beyond. In Proceedings of the 14th ACM International Conference on web search and data mining (WSDM’21) . 1154–1156

    2021

  12. [20]

    Emine Yilmaz and Stephen Robertson. 2009. Deep Versus Shallow Judgments in Learning to Rank. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’09) . 662–663

    2009

  13. [21]

    Xinyu Zhang, Andrew Yates, and Jimmy Lin. 2020. A Little Bit is Worse than None: Ranking with Limited Training Data. InProceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing (SustaiNLP’20) . 107–112

    2020

  14. [22]

    Justin Zobel. 1998. How reliable are the results of large-scale information retrieval experiments?. In Association for Computing Machinery (SIGIR ’98) . 307–314. ICTIR ’25, July 18, 2025, Padua, Italy Gabriel Iturra-Bocaz, Danny Vo, and Petra Galuščáková APPENDIX Figure 1: Dis...

    1998

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.