{"id":"480c5e88-a63e-45a4-97ae-7c63adab57be","arxiv_id":"2506.23191","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Shallow-judged training data (many queries, few judgments per query) produces better BERT rerankers than deep-judged data (few queries, many judgments per query) when total training instances are matched, across MS MARCO and LongEval.","lead":"This paper tests whether training a BERT ranking model on many queries with few labeled answers each works better than training on a few queries with many labeled answers each. The experiments on MS MARCO and LongEval show the many-query approach generally wins, and that adding negative examples can partly close the gap.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper evaluates every deep-versus-shallow comparison on shallow test sets (MS MARCO dev and LongEval test, one-to-four judged relevant documents), so the claimed general advantage of shallow training may be an artifact of test-set judgment depth.","rationale":"The strongest reading of the paper is that the effect is about the trade-off between query count and judgments per query. For that to be the right reading, the deep and shallow conditions should differ only in that trade-off. The least secure point is that they differ in at least two additional, evaluation-relevant ways: the training query pools come from different sources (TREC DL / LongEval Heldout versus MS MARCO Train / LongEval Train), and every test set is a shallow set with one-to-few judged relevant documents. The test-set asymmetry is the more decisive problem because it creates a direct mechanism for shallow training to win even when deep training produces better models: a model trained on deep judgments will tend to recognize multiple relevant documents, and on a shallow test qrel with a single relevant label those extra relevant documents are counted as errors. The authors' own discussion of unjudged relevant documents in MS MARCO V2 (Section 5) acknowledges this measurement risk. A feasible check is to evaluate the already-trained or reproduced models on the TREC DL test qrels, which are public and contain graded deep judgments. This would separate the training-data effect from the evaluation-label effect. The internal contradiction with Table 8 reinforces that the Section 6 wording is too broad. Therefore the conditional verdict stands, but the revision conditions should explicitly require either a deep-test evaluation or a careful narrowing of the conclusion.","tokens_in":11060,"tokens_out":8222,"duration_ms":91829,"concrete_test":"Using the released code, fine-tune the same MonoBERT configurations from Tables 3 and 6 (for example, MS MARCO V1 deep 50/100 versus shallow 2,500/2) and score both models on the official TREC DL 2019 and 2020 test topics with their graded qrels, available through ir-datasets or Pyserini, in addition to the MS MARCO dev sets. If shallow-trained models still outperform deep-trained models on those deep test qrels, the judgment-depth claim is supported; if deep-trained models win or tie, the conclusion must be narrowed to 'shallow training helps on shallow test collections' and the manuscript should be revised before acceptance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central conclusion in Section 6 ('Shallow training sets consistently outperform deep training sets in all our experiments') is only tested on test collections that are themselves shallow: Section 3.1 reports that the MS MARCO V1 Dev2 and V2 Dev1 test sets have on average one relevant passage per query, and Section 3.2 reports LongEval test sets with roughly four relevant documents per query. Deep models are trained on TREC DL or LongEval-Heldout queries with many graded relevance judgments, then scored against these shallow single-relevant-per-query labels; shallow models are trained on MS MARCO Train or LongEval-Train queries and tested on the same query and label distribution. This aligns both training and test conditions with the shallow side. A deep-trained reranker that correctly promotes an additional relevant passage not present in the shallow qrels is scored as a false positive, and the authors themselves flag in Section 5 that MS MARCO V2 contains many unjudged relevant documents and is unreliable for evaluation. The paper never evaluates on any deep test set, such as the TREC DL 2019/2020 test topics with their official graded qrels. In addition, Table 8 contradicts the universal wording: the deep 31/100 model beats the shallow 754/2 model on both LongEval test sets, so 'consistently outperform in all experiments' is not supported even within the paper.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares BERT-based reranking models (MonoBERT) trained on 'shallow' relevance-judgment datasets (many queries, few judgments per query, e.g., MS MARCO Train) with models trained on 'deep' datasets (fewer queries, many judgments per query, e.g., TREC DL track qrels), using MS MARCO V1/V2 and LongEval. Training-set sizes are matched in terms of total positive training instances in several experiments. The authors report that shallow-trained rerankers outperform deep-trained ones at equal instance counts, that shallow training reaches the BM25 baseline with fewer instances, and that increasing negative samples partially mitigates the disadvantage of deep judgments. They conclude that shallow training sets consistently outperform deep training sets and generalize better.","tokens_in":11342,"tokens_out":2186,"duration_ms":22772,"significance":"If the central finding held as stated, it would be practically valuable: it would justify building training sets with broad query coverage and few judgments per query, and would inform how to reuse expensive TREC-style pooling judgments for neural ranking training. The paper ships code and uses standard collections, significance tests, and a simple, reproducible MonoBERT setup. However, the causal interpretation is weakened by the confounding of judgment depth with query-pool origin and by the fact that all evaluation is done on shallow test collections; the overbroad wording of the conclusion is also contradicted by one of the paper's own tables. With appropriate caveats and additional analyses, the empirical observations remain useful for the IR community.","major_comments":[{"comment":"The deep and shallow training sets do not differ only in the number of judgments per query; they also come from different query pools (TREC Deep Learning track topics or LongEval Heldout for deep, MS MARCO Train query logs or LongEval Train for shallow). These pools may differ in topic distribution, query difficulty, and annotation methodology (pooling versus user clicks vs. single-label collection). Consequently, the performance gaps in Tables 3, 6, and 8 could reflect query selection rather than judgment depth. The manuscript does not control for query distribution or provide evidence that the query pools are comparable. This is load-bearing for the causal claim that 'shallow judgments' cause better generalization, and it should be addressed by either matching queries across conditions, analyzing query difficulty, or substantially tempering the causal language.","section":"Section 3.1, 3.2; Tables 3, 6, 8"},{"comment":"Every evaluation in the paper uses test collections that are themselves shallow: MS MARCO V1 Dev2 and V2 Dev1 have on average one relevant passage per query, and the LongEval test sets report roughly four relevant documents per query (Section 3.1, 3.2). A deep-trained reranker that correctly retrieves an additional relevant document not present in the shallow qrels is scored as a false positive, which systematically disadvantages deep training. The authors acknowledge that MS MARCO V2 may contain many unjudged relevant documents and is unreliable for evaluation (Section 5), yet no deep test set (e.g., TREC DL 2019/2020 official graded qrels) is used. This makes the claim that shallow training 'consistently outperforms' deep training a claim about performance on shallow test labels only, not a general property. The paper should either add an evaluation on a deep test set or explicitly restrict the conclusion to shallow test environments.","section":"Section 3.1, 3.2, 5; Tables 3, 6, 8"},{"comment":"The concluding sentence 'Shallow training sets consistently outperform deep training sets in all our experiments' is not supported by the paper's own data. In Table 8, the deep 31/100 model (3,100 instances) outperforms the shallow 754/2 model (1,508 instances) on both LongEval ST and LT across all metrics. While the instance counts differ, this is one of the experiments reported, and the universal wording is therefore inaccurate. The conclusion should be restated to reflect the matched-instance comparisons on MS MARCO and the more qualified LongEval result, e.g., 'shallow training is generally more sample-efficient in our experiments,' along with the caveat about test-set depth.","section":"Section 6 and Table 8"}],"minor_comments":[{"comment":"The exclusion of two development queries with more than 40,000 relevant passages is mentioned only in a footnote; since this materially changes the deep dataset statistics, the exclusion criterion and its impact on the results should be stated in the main text.","section":"Section 3.1, Table 1 footnote"},{"comment":"Significance tests are reported only against the BM25 baseline (asterisks), not as pairwise comparisons between deep and shallow conditions. Adding pairwise significance tests (or at least confidence intervals) would strengthen the claim that observed differences between deep and shallow models are meaningful.","section":"Section 4.2 and Table 3"},{"comment":"The caption contains a duplicated word: 'depending depending on the number of training instances.'","section":"Table 7 caption"},{"comment":"Reference [15] lists the arXiv ID as '11910.14424'; this appears to be a typo for '1910.14424'.","section":"References"},{"comment":"The caption mentions '2011-2022' when the intended range is presumably '2021-2022' for MS MARCO V2; please correct.","section":"Appendix, Figure 1 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant practical question and the experiments are reproducible, but the main causal claim needs to be qualified to account for the query-pool confound and the shallow-only test conditions. A revision that adds a deep-test-set evaluation or an analysis controlling for query distribution, and that revises the overbroad conclusion, would be within scope. The topic fits ICTIR well."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Plainly: this is a clean empirical study with a confounded central comparison. The authors train BERT rerankers on deep versus shallow judgments and find shallow wins, but they only evaluate on shallow test sets. MS MARCO dev sets have about one relevant passage per query, and LongEval test sets about four. That aligns training and test conditions with the shallow side. A deep-trained model that promotes an extra relevant passage not in the shallow qrels gets scored as wrong. So \"shallow consistently outperforms deep in all our experiments\" is not supported as stated. Also, deep and shallow training queries come from different pools (TREC DL topics versus MS MARCO train queries), so the gap could be query selection rather than judgment depth. Even the paper's own Table 8 undercuts the universal wording: the deep 31/100 model beats the shallow 754/2 model on both LongEval test sets. The authors do flag the MS MARCO V2 evaluation problem in Section 5, but that only adds to the uncertainty.\n\nWhat is genuinely new: the negative sampling result. They show that adding more negative examples per query narrows the deep-shallow gap, with performance peaking around a 1:8 positive-to-negative ratio and declining after. That is a cost-effective insight for anyone building training collections, and it is not confounded by the test-set asymmetry in the same way, since it is a within-dataset comparison. Credit also for matching total training-instance counts, using standard collections, and shipping code.\n\nThe soft spots beyond the test-set issue: no multiple runs or confidence intervals on the differences, the MS MARCO V2 evaluation is unreliable by their own admission, and the LongEval test set is not public. The query-pool confound is acknowledged nowhere.\n\nWho this is for: people who build IR training collections and care about annotation budget. They will get a useful hint that shallow queries plus heavy negative sampling is cheap and effective. But the headline should be toned down to \"when evaluated on shallow test sets, shallow training wins under matched instance counts.\"\n\nMy recommendation: send it to peer review — the negative sampling pattern and the cost-benefit framing deserve scrutiny — but the authors need to either evaluate on a deep test set (e.g., TREC DL 2019/2020 graded qrels) or substantially soften the claim. Without that, the central result is a conditional observation, not a general law.","headline":"Useful negative-sampling finding, but the deep-vs-shallow headline is built on shallow test sets and mismatched query pools.","tokens_in":11838,"tokens_out":1782,"would_cite":false,"duration_ms":18424,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Shallow relevance judgments beat deep ones for training BERT rerankers in this study.","keywords":["neural information retrieval","BERT reranking","relevance judgments","deep judgments","shallow judgments","negative sampling","MS MARCO","LongEval"],"falsifier":"Take a single set of queries from one corpus, split them randomly into two groups of matched difficulty, and build deep and shallow training sets from those groups with equal total instance counts (for example, 50 queries with 100 judgments each versus 2,500 queries with 2 judgments each, drawn from the same query distribution). Train the same BERT reranker on both and evaluate on the same test queries; if the deep-set model matches or beats the shallow-set model, the paper's central claim fails. A simpler version would rerun the paper's protocol but subsample the deep queries so their pool matches the shallow queries' source and difficulty.","tokens_in":10883,"feed_emoji":"🔍","tokens_out":5357,"duration_ms":51512,"temperature":0.7,"pith_summary":"This paper asks whether training data for neural rerankers should contain many queries with one or two relevance judgments each (shallow) or fewer queries with many judgments each (deep). Using BERT-based rerankers over BM25 top-10 candidates, it compares matched-size training sets drawn from MS MARCO V1, MS MARCO V2, and LongEval. It finds that shallow-judged sets consistently outperform deep-judged sets at equal training-instance counts, and that shallow sets need far fewer instances—around 1,000 to 2,000 in these settings—to beat the BM25 baseline. The paper also shows that adding negative samples improves deep-set training up to a point, with a positive-to-negative ratio near 1:8 giving the best gains. If the finding holds, annotation budgets for training data are better spent on broad query coverage than on exhaustive per-query judgment.","feed_headline":"Shallow relevance judgments beat deep ones for BERT reranking","feed_subtitle":"Across MS MARCO and LongEval, BERT rerankers train better on many lightly judged queries than on a few deeply judged ones.","key_machinery":"The load-bearing object is the query-to-judgment ratio of a training set, expressed as 'query count / judgments per query' (for example, shallow 2,500/2 versus deep 50/100) while holding the total number of training instances fixed. The reranker is MonoBERT, a BERT sequence-classification model that scores each query-document pair pointwise, fine-tuned on top-10 BM25 candidates so that reranking happens at depth 10. The comparison's validity rests on matching total training-instance counts across the deep and shallow regimes, and on drawing negatives from BM25 results with the top 10 excluded to avoid false negatives.","core_discovery":"The central claim is that, for training a BERT-based reranker, shallow relevance judgments are a more effective use of a fixed annotation budget than deep ones. In every experiment, when the total number of training instances is held equal, the model trained on many lightly judged queries outperforms the model trained on few heavily judged queries, and the deep-trained models even fall below the BM25 baseline in most MS MARCO configurations. The paper attributes the effect to the broader range of contexts and information needs that shallow sets expose the model to, improving generalization. A secondary claim is that negative judgments—cheap to generate automatically—can partially compensate for judgment depth: adding negatives helps the deep sets most, with performance peaking around a 1:8 positive-to-negative ratio before degrading at more imbalanced ratios.","pith_inferences":["We infer that the query-source confound is the main threat: if deep and shallow queries were drawn from the same pool, the performance gap could shrink or disappear, which would change the recommendation about annotation budgets.","A testable extension would apply the same shallow-versus-deep comparison to a reranker with stronger few-shot capabilities; the paper notes more advanced models might need even fewer instances, which could alter the break-even point.","We infer that evaluation pools could be made slightly shallower without losing their evaluative power, letting a single annotation effort serve both training and evaluation—an idea the paper hints at but does not test.","A useful follow-up would measure annotation cost directly (time or money per judgment) alongside performance, converting the effectiveness gap into a cost-effectiveness comparison."],"forward_implications":["Broad query coverage, not exhaustive per-query judgment, is what drives reranker performance when training budgets are fixed.","Annotation projects aiming to train neural rerankers should prioritize collecting one or two judgments for many diverse queries over hundreds of judgments for a few queries.","Existing deep evaluation collections can be reused for training only if supplemented with enough negative samples, at a ratio around 1:8 in the MS MARCO V1 setting.","Shallow sets of roughly 1,000 to 2,000 instances are enough to surpass the BM25 baseline in the tested collections, a lower threshold than earlier estimates.","Performance gains from adding negatives are not monotonic: overly imbalanced positive-to-negative ratios add noise and hurt both shallow and deep sets."],"supporting_citations":[{"why":"Introduces the deep-versus-shallow judgment trade-off that this paper re-examines for neural reranker training.","marker":"[20]"},{"why":"Defines the BERT architecture that the reranking model is fine-tuned from.","marker":"[8]"},{"why":"Proposes the MonoBERT pointwise reranking setup adopted throughout the experiments.","marker":"[15]"},{"why":"Reports the earlier estimate that about 10,000 training instances are needed to beat BM25, which this paper revisits with shallow sets.","marker":"[14]"},{"why":"Introduces the LongEval collection whose train and heldout sets provide the shallow and deep judgments used here.","marker":"[10]"},{"why":"Provides the BM25 retrieval toolkit that generates the top-10 candidate documents for reranking.","marker":"[11]"}],"fun_headline_variants":["Shallow beats deep for BERT reranker training","BERT rerankers prefer shallow relevance judgments","Many light judgments beat few deep ones for BERT","In BERT reranking, shallow relevance wins out"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The deep and shallow training sets are assumed to differ only in the number of queries versus judgments per query, but the two kinds actually come from different query pools—evaluation-campaign queries for deep sets and search-engine log queries for shallow sets—so the reported gaps could reflect query difficulty or topic distribution rather than judgment depth.","fun_headline_variants_meta":{"raw":{"variants":["Shallow beats deep for BERT reranker training","BERT rerankers prefer shallow relevance judgments","Many light judgments beat few deep ones for BERT","In BERT reranking, shallow relevance wins out"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000572,"raw_usage":{"total_tokens":2643,"prompt_tokens":824,"completion_tokens":1819,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":440,"completion_tokens_details":{"reasoning_tokens":1758}},"tokens_in":440,"tokens_out":1819,"duration_ms":13296,"temperature":1.0,"reasoning_tokens":1758,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:47:46.326428+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a single set of queries from one corpus, split them randomly into two groups of matched difficulty, and build deep and shallow training sets from those groups with equal total instance counts (for example, 50 queries with 100 judgments each versus 2,500 queries with 2 judgments each, drawn from the same query distribution). Train the same BERT reranker on both and evaluate on the same test queries; if the deep-set model matches or beats the shallow-set model, the paper's central claim fails. A simpler version would rerun the paper's protocol but subsample the deep queries so their pool matches the shallow queries' source and difficulty.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the deep-versus-shallow judgment trade-off that this paper re-examines for neural reranker training."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Reports the earlier estimate that about 10,000 training instances are needed to beat BM25, which this paper revisits with shallow sets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the LongEval collection whose train and heldout sets provide the shallow and deep judgments used here."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the BM25 retrieval toolkit that generates the top-10 candidate documents for reranking."}],"review_version":1}