Pith. sign in

REVIEW 2 major objections 6 minor 1 cited by

Overview of the TREC 2023 deep learning track

T0 review · 2 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLM-prompted rerankers outscored the four-year-best nnlm approach at TREC 2023, and human-filtered synthetic queries reproduced human-query system rankings with Kendall tau 0.8487.

desk verdict Solid track overview; the synthetic-query reliability claim in the abstract is overstated given it rests on only 31 human-filtered queries. read the letter →

arxiv 2507.08890 v1 pith:JYPIFE6V submitted 2025-07-10 cs.IR cs.AIcs.CL

Nick Craswell , Bhaskar Mitra , Emine Yilmaz , Hossein A. Rahmani , Daniel Campos , Jimmy Lin , Ellen M. Voorhees , Ian Soboroff This is my paper · ORCID
classification cs.IRcs.AIcs.CL
keywords TRECDeepLearningtrackpassagerankingdocumentMSMARCOv2syntheticqueriesLLMpromptingtestcollectionreusabilityKendalltau
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

On its fifth and final run, the TREC Deep Learning track set out to build another reusable test collection on the MS MARCO v2 passage and document corpora and to ask whether synthetic queries could stand in for real user queries during evaluation. The track reports two headline results. First, runs that use an LLM through prompting outperformed runs built on the nnlm recipe, a fine-tuned neural language model stack that was the best approach for the previous four years; the best prompt run scores NDCG@10 0.6994 on passage ranking. Second, after human assessors filtered unusable synthetic queries, evaluating systems on the remaining 31 T5- and GPT-4-generated queries agreed closely with evaluation on 51 human queries, with a Kendall tau of 0.8487, and the track found no clear sign that a query generator favored systems built with the same model. The paper presents this as evidence that synthetic queries can be a reliable component of test collection construction, while cautioning that human filtering was required.

What carries the argument

The load-bearing mechanism is the reusable test collection: 82 judged queries spanning three query types, with four-point relevance judgments and expanded qrels that copy a judged passage's label to every near-duplicate passage in its cluster. The comparison engine is NDCG@10, with Kendall's tau measuring whether systems keep their order when human queries are swapped for synthetic ones; tau = 0.8487 is the number that carries the synthetic-query conclusion. The synthetic pipeline has four stages: a GPT-4 prompt scores sampled passages for self-containedness, T5 generates many candidate queries stratified to match human query length and lexical overlap, GPT-4 generates one zero-shot query per seed passage, and human assessors reject most candidates, 66 of 97, before judging. The prompt-run result rests on comparing prompt-classified runs against nnlm runs at the system level and on per-query comparisons.

What would settle it

Take all 97 T5- and GPT-4-generated queries that went to the assessors, including the 66 rejected ones, judge them under the same relevance-assessment protocol, and recompute Kendall's tau between system NDCG@10 on those synthetic queries and on the 51 human queries; if the tau drops materially below 0.8487, the reported agreement is an artifact of the human filter rather than a property of synthetic queries in general.

Watch

Extended reading notes

Core claim

The paper's central claim is that prompt-based LLM ranking overtook fine-tuned nnlm stacks as the best-performing approach on the TREC 2023 Deep Learning track, and that synthetic queries, once filtered by human assessors, rank systems almost as well as real user queries do. The track judged 82 passage queries on the MS MARCO v2 collections: 51 from held-out human queries, 13 produced by a fine-tuned T5 query generator, and 18 by a GPT-4 prompt; labels were then propagated from passages to source documents for the document task, and expanded qrels spread each judgment through its cluster of near-duplicate passages. The best prompt run reaches NDCG@10 of 0.6994 on passage ranking versus 0.5972 for the best nnlm run, and the prompt methods win on most individual queries. Across submitted systems, Kendall's tau between system orderings on human queries and on synthetic queries is 0.8487, and 0.9395 when the collection is evaluated on all real plus synthetic track queries. The authors conclude that synthetic queries can be reliably used in test collection construction, but only after human selection, and they find no clear evidence that GPT-4-generated queries inflate GPT-based systems or that T5-generated queries inflate T5-based systems.

Load-bearing premise

The load-bearing premise is that the 31 synthetic queries that survived human filtering are representative of synthetic queries in general; if the filter removes exactly the difficult, ambiguous, or biased queries, the observed agreement of tau = 0.8487 will not carry over to unfiltered synthetic queries.

Editorial extensions

If this is right

  • Prompt-based LLM ranking now becomes the reference point for retrieval track benchmarks, replacing the fine-tuned nnlm stacks that had dominated the previous four years.
  • Test collection builders can generate candidate queries with T5 or GPT-4 and still obtain system orderings close to those from human queries, provided a human quality filter is applied first.
  • Passage-level judgments remain sufficient for document ranking: propagating passage labels to source documents yields usable document qrels without separate document judging.
  • With two consecutive years of harder held-out query sets, year-over-year comparisons on this collection are more discriminative than the earlier MS MARCO test queries.
  • Even the best prompt runs mostly self-reported using MS MARCO training data somewhere in the stack, so prompting does not remove the need for fine-tuned retrieval components.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the 0.8487 tau is measured on only 31 accepted synthetic queries, and 66 of 97 generated queries were discarded by human assessors, so the practical cost saving depends on automating or outsourcing that filter, which this paper does not attempt.
  • Editorial inference: GPT-4 queries are nearly twice as long as human queries, and both synthetic types return fewer relevant documents per query, so a synthetic-query collection is a harder and somewhat different evaluation surface, not a free replacement for the historical human-query series.
  • Editorial inference: the no-bias conclusion is drawn from very few systems in each family, for example one document-ranking prompt run, so the absence of clear bias is a weak bound; a larger paired sampling of generators and system families could detect smaller or conditional biases.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper is the overview of the TREC 2023 Deep Learning Track. It describes the passage and document ranking tasks, the MS MARCO v2 datasets, the submitted runs, and the main evaluation results. The authors report two headline findings: (1) runs that use large language model (LLM) prompting in some part of the pipeline outperformed runs using the previous best "nnlm" approach, and (2) evaluation using synthetically generated queries (from T5 and GPT-4) produced system orderings similar to those from human queries, with a Kendall tau of 0.8487. The paper also analyzes potential bias from using T5 or GPT-4 generated queries toward systems based on the same model family, finding no clear bias. The track is the final year of the Deep Learning Track, and the paper emphasizes the construction of a reusable test collection.

Significance. If the findings hold, they are of considerable significance to the IR community. The prompt-vs-nnlm result suggests that LLM prompting is now a state-of-the-art technique on this benchmark, while the synthetic-query analysis speaks directly to the cost and feasibility of building test collections without human queries. The paper is transparent about its query generation pipeline, including the prompts used, the filtering rates, and the final query counts, and it makes the official NIST judgments the basis of all reported metrics. These are genuine strengths. However, the synthetic-query claim is based on a small, human-filtered subset of the generated queries, and the prompt-vs-nnlm comparison relies on self-classified run categories without significance testing. Both limitations constrain the strength of the conclusions as stated in the abstract, although the paper itself uses cautious language in places (e.g., "initial results suggest" and "more analysis is needed").

major comments (2)
  1. [Section 4, Table 2, Figure 2] The headline agreement tau = 0.8487 is computed on only 31 synthetic queries (13 T5-generated and 18 GPT-4-generated) that survived a human filtering step. The paper states that assessors removed queries that "do not look reasonable" or that contain "too few or too many relevant documents" because they are "noisy or not very informative for evaluation purposes." This filtering is not random, and it plausibly removes exactly the difficult, ambiguous, or atypical synthetic queries where system agreement with human-query evaluation could be lowest. The reported Kendall tau therefore may substantially overstate the agreement that an unfiltered synthetic-query test collection would achieve. The conclusion in Section 5 that "test collections consisting of synthetically generated queries could be reliably used" is not supported for unfiltered synthetic generation. The authors should either report the agreement on the full set of generated queries (if judgments exist), provide a confidence interval or significance test for the tau value, or explicitly qualify the claim to apply only to synthetic queries that pass a human quality filter. The abstract's substitution claim currently goes beyond what the data show.
  2. [Section 4, Table 2, Figure 2] The headline claim about prompt runs outperforming nnlm runs is not supported by significance testing, and the self-classified run categories plus the concentration of top runs in two groups make the comparison less controlled than the abstract implies.
minor comments (6)
  1. [Section 4] There is a typo in "one or mpre phases" (should be "more"), and the sentence "The best 'prompt' run outperforms the best 'nnlm' on on the majority of queries" has a duplicated "on."
  2. [Figure 2 caption] The caption says "As in the previous two years, 'nnlm' runs continue to outperform over 'trad' runs for both tasks," but the figure includes the 'prompt' category; the caption should describe all three run types shown.
  3. [Figure 4 caption] The caption refers to "mean performance between 'prompt' and 'llm' runs," but the correct category name is "nnlm".
  4. [Section 5, Table 6] Table 6 reports the final number of queries per type (82 total) but not the numbers of queries initially generated and provided to assessors (200 human, 250 T5, 250 GPT-4, of which 147/48/49 were provided to assessors). Adding the initial counts and the selection rates would make the filtering step clearer.
  5. [Section 5] The sentence "For all query types, depth-10 pooling was used to select the documents to be judged by the NIST assessors" is potentially confusing because the track judged passages, not documents, and document labels were inferred from passage labels; please clarify that the depth-10 pooling applies to passage pooling, with document labels propagated afterward.
  6. [Conclusion] The Conclusion contains a typo: "we repeated the updats that were first introduced last year" should be "updates."

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline claims are empirical comparisons against independent NIST judgments, not derivations from their own inputs.

full rationale

This paper is an evaluation overview, not a derivation. The two headline claims—that LLM-prompted runs outperformed the previous best 'nnlm' approach, and that synthetic queries yield system orderings close to real queries (tau = 0.8487)—are both empirical measurements against NIST relevance judgments. The prompt-vs-nnlm comparison is computed from submitted runs and externally judged relevance labels; no parameter is fitted to the outcome and then reported as a prediction. The synthetic-query analysis compares system NDCG@10 computed on real queries with the same systems' NDCG@10 computed on separately generated and separately judged synthetic queries; one set of scores is not an input to the other. The paper even tests the most plausible circularity threat—that query generators from a given model family bias evaluation in favor of systems from that family—and reports no clear evidence of such bias, which is an explicit check rather than an assumption. The human filtering of synthetic queries (13 of 48 T5 and 18 of 49 GPT-4 queries retained) is a real validity limitation on generalization, but it is not a circular step: the retained synthetic queries are still assessed independently, and the agreement statistic is not constructed to equal any input. Citations to the track's own previous overviews provide empirical context and dataset provenance, but no load-bearing mathematical claim in this paper reduces to those citations. No equation, fitted parameter, renamed result, or self-citation chain is used to manufacture the stated conclusions.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical entities or physical models. It relies on standard IR evaluation practice (pooling, NIST judgments) and on a set of hand-chosen thresholds and prompts in the synthetic query pipeline. The key unvalidated assumption is that the surviving synthetic queries represent the full synthetic query distribution.

free parameters (4)
  • passage quality score threshold = 50
    Chosen by hand in Section 5; passages with GPT-4 quality scores below 50 (14.6%) were discarded as seed passages before query generation.
  • GPT-4 sampling temperature for passage quality scoring = 0.8
    Set in Section 5 for the one-off passage quality filter; stochasticity affects which passages pass the threshold.
  • T5 query generation count per passage = 100
    Section 5: the BeIR T5 query generator produced 100 queries per seed passage; 250 were later sampled for the test set.
  • T5 query sampling stratification = matched to 2022 NIST qrels on query length and lexical overlap
    Section 5: sampled 250 T5 query-passage pairs to match the distribution of 2022 human qrels; a design choice that shapes the synthetic query set.
assumptions (4)
  • domain assumption NIST assessor judgments are ground-truth relevance labels for both human and synthetic queries
    All metrics (NDCG, AP) are computed against NIST judgments (Sections 2 and 4); the track's conclusions inherit the quality of these judgments.
  • domain assumption Depth-10 pooling over submitted runs yields a sufficiently complete set of relevant documents for fair evaluation
    Section 5 uses depth-10 pooling; the track assumes the pooled set covers the relevant results needed for reusable evaluation.
  • domain assumption The stratification of T5 queries by query length and lexical overlap with the passage makes them comparable to human queries
    Section 5 assumes that matching these two statistics to 2022 NIST qrels is enough to make T5 queries representative of human queries.
  • domain assumption The GPT-4 and T5 prompts used for query generation produce queries similar in intent to real user queries
    Section 5 relies on these prompts without an external validation of query realism beyond the NIST filter.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Overview of the TREC 2023 deep learning track." pith.science (2026). https://pith.science/paper/JYPIFE6V

@misc{pith2026250708890,
  author       = {Pith},
  title        = {Pith review of: Overview of the TREC 2023 deep learning track},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JYPIFE6V}},
  note         = {Machine review of arXiv:2507.08890}
}
abstract

This is the fifth year of the TREC Deep Learning track. As in previous years, we leverage the MS MARCO datasets that made hundreds of thousands of human-annotated training labels available for both passage and document ranking tasks. We mostly repeated last year's design, to get another matching test set, based on the larger, cleaner, less-biased v2 passage and document set, with passage ranking as primary and document ranking as a secondary task (using labels inferred from passage). As we did last year, we sample from MS MARCO queries that were completely held out, unused in corpus construction, unlike the test queries in the first three years. This approach yields a more difficult test with more headroom for improvement. Alongside the usual MS MARCO (human) queries from MS MARCO, this year we generated synthetic queries using a fine-tuned T5 model and using a GPT-4 prompt. The new headline result this year is that runs using Large Language Model (LLM) prompting in some way outperformed runs that use the "nnlm" approach, which was the best approach in the previous four years. Since this is the last year of the track, future iterations of prompt-based ranking can happen in other tracks. Human relevance assessments were applied to all query types, not just human MS MARCO queries. Evaluation using synthetic queries gave similar results to human queries, with system ordering agreement of $\tau=0.8487$. However, human effort was needed to select a subset of the synthetic queries that were usable. We did not see clear evidence of bias, where runs using GPT-4 were favored when evaluated using synthetic GPT-4 queries, or where runs using T5 were favored when evaluated on synthetic T5 queries.

Figures

Figures reproduced from arXiv: 2507.08890 by the authors.

Figure 1
Figure 1. Crowd task used to generate the original MS MARCO natural language generation leaderboard. This same [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. NDCG@10 results by run type. As in the previous two years, “nnlm” runs continue to outperform over [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Comparison of the best “prompt” and “nnlm” runs on individual test queries for the passage ranking task. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Comparison of the best “prompt” and “nnlm” runs on individual test queries for the document ranking task. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Comparison of system performance on real queries versus (left) on all (real+synthetic) track queries, and [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Comparison of system performance on real queries versus (left) on queries generated using GPT-4, and [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Lighting the Way for BRIGHT: Reproducible Baselines with Anserini, Pyserini, and RankLLM

    cs.IR 2025-09 conditional novelty 5.0 of 10

    BRIGHT's BM25 results come from query-side BM25 weighting rather than standard bag-of-words, and the paper shows this variant helps on the benchmark's long queries while shipping reproducible toolkits and a corpus audit.

Reference graph

Works this paper leans on

13 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [2]

    Overview of the trec 2019 deep learning track

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen Voorhees. Overview of the trec 2019 deep learning track. In Proceedings of Text REtrieval Conference (TREC), 2020

    work page 2019

  2. [3]

    Overview of the trec 2020 deep learning track

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. Overview of the trec 2020 deep learning track. In Proceedings of Text REtrieval Conference (TREC), 2021 a

    work page 2020

  3. [4]

    Ms marco: Benchmarking ranking models in the large-data regime

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Jimmy Lin. Ms marco: Benchmarking ranking models in the large-data regime. In Proc. SIGIR, pages 1566--1576, 2021 b

    work page 2021

  4. [5]

    Overview of the trec 2021 deep learning track

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Jimmy Lin. Overview of the trec 2021 deep learning track. In Proceedings of Text REtrieval Conference (TREC), 2022

    work page 2021

  5. [6]

    Voorhees, and Ian Soboroff

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, Jimmy Lin, Ellem M. Voorhees, and Ian Soboroff. Overview of the trec 2022 deep learning track. In Proceedings of Text REtrieval Conference (TREC), 2023

    work page 2022

  6. [7]

    arvelin and J. Kek\

    K. J\"arvelin and J. Kek\"al\"ainen. Cumulated gain-based evaluation of IR techniques. ACM TOIS, 20 0 (4): 0 422--446, 2002

    work page 2002

  7. [8]

    Significant improvements over the state of the art? a case study of the ms marco document ranking leaderboard

    Jimmy Lin, Daniel Campos, Nick Craswell, Bhaskar Mitra, and Emine Yilmaz. Significant improvements over the state of the art? a case study of the ms marco document ranking leaderboard. In Proc. SIGIR, pages 2283--2287, 2021 a

    work page 2021

  8. [9]

    Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations

    Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, page 2356–2362, New Y...

    work page 2021

Show all 13 references
  1. [10]

    Fostering coopetition while plugging leaks: The design and implementation of the ms marco leaderboards

    Jimmy Lin, Daniel Campos, Nick Craswell, Bhaskar Mitra, and Emine Yilmaz. Fostering coopetition while plugging leaks: The design and implementation of the ms marco leaderboards. In Proc. SIGIR, 2022

    2022

  2. [11]

    MS MARCO: A human generated machine reading comprehension dataset

    Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. MS MARCO: A human generated machine reading comprehension dataset. CoRR, abs/1611.09268, 2016. URL http://arxiv.org/abs/1611.09268

    2016 arXiv

  3. [12]

    TREC: Experiment and evaluation in information retrieval, volume 1

    Ellen M Voorhees, Donna K Harman, et al. TREC: Experiment and evaluation in information retrieval, volume 1. MIT press Cambridge, 2005

    2005

  4. [13]

    Too many relevants: Whither cranfield test collections? In Proc

    Ellen M Voorhees, Nick Craswell, and Jimmy Lin. Too many relevants: Whither cranfield test collections? In Proc. SIGIR, 2022

    2022

  5. [14]

    Investigating passage-level relevance and its role in document-level relevance judgment

    Zhijing Wu, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma. Investigating passage-level relevance and its role in document-level relevance judgment. SIGIR'19, page 605–614, New York, NY, USA, 2019. Association for Computing Machinery

    2019

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.