Pith. sign in

REVIEW 3 major objections 6 minor 62 references

Hypothetical Documents or Knowledge Leakage? Rethinking LLM-based Query Expansion

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLM query expansion helps only when models leak benchmark answers

desk verdict A useful diagnostic for when QE helps, but the leakage conclusion is overclaimed; the matched/unmatched gap is the QE mechanism doing its job. read the letter →

arxiv 2504.14175 v2 pith:25UTICER submitted 2025-04-19 cs.CL cs.IR

Yejun Yoon , Jaeyoon Jung , Seunghyun Yoon , Kunwoo Park This is my paper · ORCID
classification cs.CLcs.IR
keywords queryexpansionknowledgeleakagefactverificationhypotheticaldocumentsnaturallanguageinferenceinformationretrievalLLMmemorizationzero-shot
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper challenges the assumption behind LLM-based query expansion (QE) methods like HyDE and Query2doc: that a language model can generate a useful hypothetical document to help retrieve real evidence. Using fact verification as a testbed, the authors ask whether the apparent success of such methods is actually driven by knowledge leakage, where the LLM has seen the benchmark's gold evidence during pretraining and simply reproduces it. Across three benchmarks (FEVER, SciFact, AVeriTeC) and seven backbone LLMs, they find that QE improves retrieval and verdict prediction only when the generated documents contain sentences entailed by the gold evidence. When no such entailment is detected, QE often performs worse than the baseline retriever. The authors conclude that knowledge leakage may inflate the perceived effectiveness of LLM-based query expansion and that these methods may be of limited use in real-world fact-checking of novel or niche knowledge.

What carries the argument

The central mechanism is an NLI-based matching algorithm that classifies each claim into a matched or unmatched condition. The algorithm segments the LLM-generated document into sentences, removes sentences that merely reproduce the claim, and runs a natural language inference model (GPT-4o-mini) on each pair of gold evidence sentence and generated sentence; if any pair is labeled entailment, the claim is matched. This binary condition is then used to stratify retrieval and verification performance, and the paper's finding is the consistent gap between the two conditions.

What would settle it

Construct synthetic fact-verification claims from documents published after the target LLM's training cutoff, then run HyDE and Query2doc with that model. If query expansion still improves retrieval and verdict prediction on matched claims, or if the matched/unmatched performance gap persists without any possible pretraining exposure, the leakage interpretation is wrong.

Watch

Extended reading notes

Core claim

The paper establishes a conditional pattern: LLM-based query expansion is effective exactly when the LLM-generated document reproduces information entailed by the benchmark's gold evidence. Using an NLI-based matching algorithm to label each claim as matched (at least one generated sentence entailed by gold evidence) or unmatched, the authors show that across all tested LLMs and datasets, matched claims enjoy significant gains over both all claims and unmatched claims, while unmatched claims frequently fall below the baseline. The matched proportion is high, often between 40% and 80% depending on dataset, model, and method. The authors interpret this as evidence that the LLMs were exposed to benchmark knowledge sources during pretraining, and that the hypothetical documents are in large part recall of memorized content rather than genuine extrapolation.

Load-bearing premise

The load-bearing premise is that an NLI entailment match between an LLM-generated sentence and gold evidence indicates the LLM was exposed to that specific evidence during pretraining, rather than that the model reconstructed the fact from general world knowledge or from the claim itself.

Editorial extensions

If this is right

  • If QE effectiveness depends on leakage, then reported gains on standard fact-verification benchmarks overestimate real-world value for novel or niche claims.
  • On unmatched claims, QE can actively hurt retrieval compared to using the claim alone, so practitioners should use generation-based expansion cautiously.
  • Benchmark creators should treat contamination as a confound and build evaluation sets with post-cutoff or synthetic knowledge.
  • The methodology can be adapted to other knowledge-intensive tasks as a leakage diagnostic, though the paper does not test those settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A causal test of the leakage interpretation could be built by generating claims exclusively from documents published after the target LLM's training cutoff and checking whether QE gains disappear or the matched/unmatched gap persists.
  • The matched/unmatched split offers a practical sanity check for any retrieval benchmark: if a generation-based method only helps on instances where the generator already knows the answer, the benchmark's difficulty is not where it appears.
  • The same NLI matching could audit closed-book QA and retrieval-augmented generation pipelines to see whether performance gains are similarly concentrated in leaked instances.
  • The matched condition may be confounded with claim difficulty, since claims whose answers are easily inferred from the claim itself could be both more likely to be matched and easier to retrieve; a design controlling for claim-inherent answerability would sharpen the leakage interpretation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The manuscript investigates whether the effectiveness of LLM-based query expansion (QE) in zero-shot evidence retrieval is partly an artifact of knowledge leakage. Using fact verification as a testbed, it applies Query2doc and HyDE over FEVER, SciFact, and AVeriTeC with seven backbone LLMs, and labels each claim as matched if the generated pseudo-document contains a sentence entailed by the gold evidence under a GPT-4o-mini NLI judge. The paper reports that both QE methods improve retrieval and verdict prediction on average, but that the gains concentrate on matched claims: performance on unmatched claims is often below the non-expanded baseline. From this correlation the paper concludes that knowledge leakage may be present in these benchmarks and may inflate the perceived performance of LLM-based query expansion. The Limitations section explicitly disclaims a causal link between leakage and generation, but the abstract, Discussion, and Conclusion present the correlation as suggestive evidence of leakage.

Significance. If the claimed leakage effect were established, the result would be important for interpreting a large body of zero-shot retrieval and retrieval-augmented generation evaluations, and it would motivate new benchmark designs and QE methods robust to unseen knowledge. The paper's strengths are its breadth and reproducibility: the matched-versus-unmatched pattern is consistent across three datasets, seven LLMs, two QE methods, and two values of k, with eight repetitions and standard errors, and the automatic NLI-based matching is validated against manual annotation on a 500-sample subset. However, the central inferential step from the matched/unmatched performance gap to knowledge leakage is not currently supported, because the matched condition is defined using the same gold evidence against which retrieval is scored. The empirical pattern is valuable as a benchmark finding, but the leakage interpretation requires additional controls that the manuscript does not provide.

major comments (3)
  1. [Section 3.3, Table 4] The matched condition is defined by NLI entailment between an LLM-generated sentence and gold evidence, while the retrieval metrics in Table 4 are evaluated against that same gold evidence. Under the standard hypothesis-document account of QE, this is exactly the condition under which expansion should help: a generated document semantically aligned with the gold evidence should make that evidence easier to retrieve, whether or not the LLM ever saw the benchmark during pretraining. Conversely, an inaccurate generated document should hurt retrieval. The M-versus-unmatched gap in Table 4 is therefore predicted by the intended QE mechanism even with zero leakage, and the gap cannot by itself indicate knowledge leakage. The paper needs a control in which leakage is impossible, such as claims whose gold evidence was published after the model's training cutoff or evidence constructed synthetically, and should show that the matched/unmatched gap disappears or changes in that setting. This control should be added or the leakage claim should be substantially softened.
  2. [Section 5, Table 4] The matched/unmatched distinction is also confounded with claim difficulty. Claims for which the LLM can produce gold-entailed sentences are likely to be claims that are well covered in general knowledge, that contain the answer lexically, or that are otherwise easier retrieval targets; the unmatched set may therefore be proxying for hard claims rather than for absence of leakage. Since the manuscript does not control for baseline retrievability or claim difficulty, the claim that 'performance improvements from query expansion were consistent only when LLM-generated documents contained sentences entailed by gold evidence' (Section 5) conflates the leakage hypothesis with a difficulty confound. A concrete remedy would be to compare matched and unmatched claims with matched baseline retrieval difficulty, or to use a post-hoc matched-pair analysis where gold evidence difficulty is held fixed.
  3. [Section 3.3, Table 3, Section 5] An NLI entailment match between a generated sentence and gold evidence establishes semantic equivalence, not pretraining exposure. The phrasing in Section 3.3 ('it may indicate that the backbone LLM was exposed to knowledge related to the corresponding evidence during training') is appropriately hedged, but Section 5 states that the seven LLMs 'were likely exposed to knowledge sources from the three benchmarks during training.' That stronger statement does not follow from the matched proportions in Table 3, because the same outputs could arise from general world knowledge, from the claim containing most of the evidence content, or from the model reconstructing a well-known fact without having memorized the specific benchmark document. The Discussion should explicitly acknowledge this alternative explanation or add a control that distinguishes memorization from reasoning over general knowledge.
minor comments (6)
  1. [Section 1] In the sentence 'These methods assume that LLMs can generate hypothetical documents,' the phrase 'hypotheticaldocuments' appears without a space in the manuscript text; this should be corrected.
  2. [Throughout] The dataset name is inconsistently typeset as 'A VeriTeC', 'AVeriTeC', and 'AVeriTeC' across the body, tables, and appendix; the spelling should be standardized.
  3. [Appendix C] The Mann-Whitney U test is mentioned without specifying whether it is applied to per-claim scores, per-repetition aggregate scores, or to the retrieval metrics directly; since only eight repetitions were run, the level of analysis affects interpretation of the reported p<0.001 values.
  4. [Section 3.3] The ROUGE-2 threshold of 0.95 for removing claim reproductions is described as based on manual inspection but no sensitivity analysis is reported; a short description of how the threshold was chosen would improve reproducibility.
  5. [Appendix B, Equation (3)] Equation (3) uses the notation max_X with a boolean function X but does not define the constraint set for X; please specify that X is an assignment matrix and state the constraints used in the Hungarian algorithm.
  6. [Appendix C] The text refers to 'Pyserini' in one place and 'PySerini' in another; the spelling should be made consistent.

Circularity Check

1 steps flagged · score 6.0 of 10

The matched/unmatched performance gap is partly tautological: 'matched' is defined as generated text entailing the same gold evidence used to score retrieval, so higher performance on M reflects the intended QE mechanism rather than independent evidence of leakage.

  1. self definitional [Section 3.3 (Matching Algorithm); Section 4, Table 4; Section 5 (Discussion)]
    "If such a sentence exists, it may indicate that the backbone LLM was exposed to knowledge related to the corresponding evidence during training. ... (3) Label Aggregation: ... If there exists l(i,j) labeled as entailment, assign matched, otherwise assign unmatched. ... Table 4 presents fact-verification performance based on whether LLM-generated documents contained sentences entailed by gold evidence."

    The condition 'matched' is defined by entailment between the LLM-generated sentence and the gold evidence E, and the retrieval metrics (Recall@5, NDCG@5, METEOR/BERTScore) are scored against that same E. For Query2doc, appending a document that lexically or semantically reproduces E directly boosts BM25 scores for E; for HyDE, averaging g(d) into the query vector moves the query toward the gold evidence. Hence higher retrieval on matched claims is the expected operation of QE, not an independent signal of pretraining exposure. The leakage conclusion therefore reduces to the matched-condition definition plus an untested assumption that entailment implies memorization; the paper's own Limitations disclaim a causal link.

full rationale

The paper's measurement of the proportion of matched claims is a legitimate empirical observation, and the matched/unmatched performance gap is real and consistently reproduced. However, as evidence for knowledge leakage, the central step is circular: matched claims are, by construction, claims for which the expanded query contains the gold evidence or its paraphrase, and the same gold evidence defines the retrieval target. Under the standard hypothesis-document account of QE, this is exactly the condition in which expansion should help. Without a control (e.g., evidence published after the model cutoff, or synthetic evidence that cannot be memorized), the correlation cannot distinguish leakage from the intended mechanism of QE or from general-knowledge generation. The paper's Limitations statement ('we do not claim a causal link between data leakage and generation') appropriately narrows the claim, but the abstract and Discussion still present the correlation as suggesting leakage. No load-bearing self-citation or fitted-parameter-as-prediction issues were found; the circularity is in the matched-condition/evaluation-target alignment.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper is an empirical analysis with no fitted parameters in the usual sense. The single hand-chosen threshold (ROUGE-2 = 0.95) influences the central matched/unmatched classification. The main load-bearing assumptions are interpretive: that NLI entailment with gold evidence is a proxy for pretraining exposure, and that the matched/unmatched split isolates leakage rather than claim difficulty. No new entities are postulated.

free parameters (1)
  • ROUGE-2 threshold for claim-reproduction filtering = 0.95
    Sentences in the generated document that overlap the original claim with ROUGE-2 above 0.95 are removed before NLI matching. The threshold was set by manual inspection (Section 3.3), and it changes which sentences enter the matching step, so it affects the matched/unmatched split that drives the main result.
assumptions (3)
  • domain assumption NLI entailment between an LLM-generated sentence and gold evidence indicates the LLM was exposed to that gold evidence during pretraining.
    Used to define matched claims in Section 3.3 and to interpret Table 3 as evidence of knowledge leakage in the Discussion. It ignores the possibility that the LLM reconstructs the same fact from general knowledge or from the claim itself.
  • domain assumption Matched and unmatched claims differ only in leakage, not in claim difficulty or other confounds.
    The comparison in Table 4 assumes the matched/unmatched split isolates knowledge leakage; no controls for claim complexity, entity popularity, or retrieval difficulty are reported.
  • domain assumption Gold evidence in these benchmarks is representative of pretraining corpora.
    The inference from overlap with gold evidence to pretraining exposure assumes the gold evidence itself (e.g., Wikipedia sentences in FEVER) was in the training data of all seven LLMs, which is plausible but unverified for the proprietary models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hypothetical Documents or Knowledge Leakage? Rethinking LLM-based Query Expansion." pith.science (2026). https://pith.science/paper/25UTICER

@misc{pith2026250414175,
  author       = {Pith},
  title        = {Pith review of: Hypothetical Documents or Knowledge Leakage? Rethinking LLM-based Query Expansion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/25UTICER}},
  note         = {Machine review of arXiv:2504.14175}
}
read the original abstract

Query expansion methods powered by large language models (LLMs) have demonstrated effectiveness in zero-shot retrieval tasks. These methods assume that LLMs can generate hypothetical documents that, when incorporated into a query vector, enhance the retrieval of real evidence. However, we challenge this assumption by investigating whether knowledge leakage in benchmarks contributes to the observed performance gains. Using fact verification as a testbed, we analyze whether the generated documents contain information entailed by ground-truth evidence and assess their impact on performance. Our findings indicate that, on average, performance improvements consistently occurred for claims whose generated documents included sentences entailed by gold evidence. This suggests that knowledge leakage may be present in fact-verification benchmarks, potentially inflating the perceived performance of LLM-based query expansion methods.

Figures

Figures reproduced from arXiv: 2504.14175 by the authors.

Figure 1
Figure 1. Illustration of potential knowledge leakage in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 15 canonical work pages

  1. [1]

    Gianni Amati and Cornelis Joost Van Rijsbergen. 2002. Probabilistic models of information retrieval based on measuring the divergence from randomness. ACM Transactions on Information Systems (TOIS), 20(4):357--389

    work page 2002

  2. [2]

    Hiteshwar Kumar Azad and Akshay Deepak. 2019. Query expansion techniques for information retrieval: a survey. Information Processing & Management, 56(5):1698--1735

    work page 2019

  3. [3]

    Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268

    arXiv 2016

  4. [4]

    Satanjeev Banerjee and Alon Lavie. 2005. https://aclanthology.org/W05-0909/ METEOR : An automatic metric for MT evaluation with improved correlation with human judgments . In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , pages 65--72, Ann Arbor, Michigan. Association for Compu...

    2005

  5. [5]

    O'Reilly Media, Inc

    Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. " O'Reilly Media, Inc."

    2009

  6. [6]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  7. [7]

    Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. 2022. Discovering latent knowledge in language models without supervision. arXiv preprint arXiv:2212.03827

    arXiv 2022

  8. [8]

    Jifan Chen, Grace Kim, Aniruddh Sriram, Greg Durrett, and Eunsol Choi. 2024 a . https://doi.org/10.18653/v1/2024.naacl-long.196 Complex claim verification with evidence retrieved in the wild . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pa...

    doi:10.18653/v1/2024.naacl-long.196 2024

Show all 62 references
  1. [9]

    Jifan Chen, Aniruddh Sriram, Eunsol Choi, and Greg Durrett. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.229 Generating literal and implied subquestions to fact-check complex claims . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processin...

    2022 doi

  2. [10]

    Xinran Chen, Xuanang Chen, Ben He, Tengfei Wen, and Le Sun. 2024 b . https://doi.org/10.18653/v1/2024.findings-acl.708 Analyze, generate and refine: Query expansion with LLM s for zero-shot open-domain QA . In Findings of the Association for Computational Linguistics: ACL 2024...

    2024 doi

  3. [11]

    Qinyuan Cheng, Tianxiang Sun, Xiangyang Liu, Wenwei Zhang, Zhangyue Yin, Shimin Li, Linyang Li, Zhengfu He, Kai Chen, and Xipeng Qiu. 2024. https://openreview.net/forum?id=girxGkdECL Can AI assistants know what they don't know? In Forty-first International Conference on Machin...

    2024

  4. [12]

    Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. 2023. Benchmark probing: Investigating data leakage in large language models. In NeurIPS 2023 Workshop on Backdoors in Deep Learning-The Good, the Bad, and the Ugly

    2023

  5. [13]

    Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. 2024. https://doi.org/10.18653/v1/2024.naacl-long.482 Investigating data contamination in modern benchmarks for large language models . In Proceedings of the 2024 Conference of the North American Chapter ...

    2024 doi

  6. [14]

    Qingxiu Dong, Jingjing Xu, Lingpeng Kong, Zhifang Sui, and Lei Li. 2024. Statistical knowledge assessment for large language models. Advances in Neural Information Processing Systems, 36

    2024

  7. [15]

    Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. 2022. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, pages 5547--5569. PMLR

    2022

  8. [16]

    Angela Fan, Aleksandra Piktus, Fabio Petroni, Guillaume Wenzek, Marzieh Saeidi, Andreas Vlachos, Antoine Bordes, and Sebastian Riedel. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.580 Generating fact checking briefs . In Proceedings of the 2020 Conference on Empirical Met...

    2020 doi

  9. [17]

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027

    2020 arXiv

  10. [18]

    Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2023. https://doi.org/10.18653/v1/2023.acl-long.99 Precise zero-shot dense retrieval without relevance labels . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers...

    2023 doi

  11. [19]

    Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tusha...

    2024

  12. [20]

    Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos. 2022. https://doi.org/10.1162/tacl_a_00454 A survey on automated fact-checking . Transactions of the Association for Computational Linguistics, 10:178--206

    2022 doi

  13. [21]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    2020 arXiv

  14. [22]

    Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118

    2021 arXiv

  15. [23]

    Rolf Jagerman, Honglei Zhuang, Zhen Qin, Xuanhui Wang, and Michael Bendersky. 2023. Query expansion by prompting large language models. arXiv preprint arXiv:2305.03653

    2023 arXiv

  16. [24]

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. https://doi.org/10.18653/v1/P17-1147 T rivia QA : A large scale distantly supervised challenge dataset for reading comprehension . In Proceedings of the 55th Annual Meeting of the Association for Computational...

    2017 doi

  17. [25]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221

    2022 arXiv

  18. [26]

    Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning, pages 15696--15707. PMLR

    2023

  19. [27]

    Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A Smith, Yejin Choi, Kentaro Inui, et al. 2024. Realtime qa: what's the answer right now? Advances in Neural Information Processing Systems, 36

    2024

  20. [28]

    Harold W Kuhn. 1955. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83--97

    1955

  21. [29]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019 doi

  22. [30]

    Bruce Croft

    Victor Lavrenko and W. Bruce Croft. 2001. https://doi.org/10.1145/383952.383972 Relevance based language models . In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '01, page 120–127, New York, NY, U...

    2001

  23. [31]

    Yibin Lei, Yu Cao, Tianyi Zhou, Tao Shen, and Andrew Yates. 2024. Corpus-steered query expansion with large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pages 393--401

    2024

  24. [32]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Proc...

    2020

  25. [33]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81

    2004

  26. [34]

    Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini : A Python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th Annual International ACM SIGIR Con...

    2021

  27. [35]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. https://doi.org/10.1162/tacl_a_00638 Lost in the middle: How language models use long contexts . Transactions of the Association for Computational Linguistics, 12...

    2024 doi

  28. [36]

    Bill MacCartney. 2009. Natural language inference. Stanford University

    2009

  29. [37]

    Iain Mackie, Shubham Chatterjee, and Jeffrey Dalton. 2023 a . https://doi.org/10.1145/3539618.3591992 Generative relevance feedback with large language models . In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, ...

    2023

  30. [38]

    Iain Mackie, Shubham Chatterjee, and Jeffrey Dalton. 2023 b . Generative relevance feedback with large language models. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2026--2031

    2023

  31. [39]

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. https://doi.org/10.18653/v1/2023.acl-long.546 When not to trust language models: Investigating effectiveness of parametric and non-parametric memories . In Proceedings of the 6...

    2023 doi

  32. [40]

    Christopher D Manning. 2009. An introduction to information retrieval

    2009

  33. [41]

    Sebasti \ a o Miranda, David Nogueira, Afonso Mendes, Andreas Vlachos, Andrew Secker, Rebecca Garrett, Jeff Mitchel, and Zita Marinho. 2019. Automated fact checking in the news room. In The world wide web conference, pages 3579--3583

    2019

  34. [42]

    Preslav Nakov, David Corney, Maram Hasanain, Firoj Alam, Tamer Elsayed, Alberto Barr \'o n-Cede \ n o, Paolo Papotti, Shaden Shaar, Giovanni Da San Martino, et al. 2021. Automated fact-checking for assisting human fact-checkers. In Proceedings of the Thirtieth International Jo...

    2021

  35. [43]

    JJ Rocchio. 1971. Relevance feedback in information retrieval. The SMART Retrieval System-Experiments in Automatic Document Processing/Prentice Hall

    1971

  36. [44]

    Vinay Samuel, Yue Zhou, and Henry Peng Zou. 2025. https://aclanthology.org/2025.coling-main.338/ Towards data contamination detection for modern large language models: Limitations, inconsistencies, and oracle challenges . In Proceedings of the 31st International Conference on ...

    2025

  37. [45]

    Michael Schlichtkrull, Zhijiang Guo, and Andreas Vlachos. 2023. Averitec: a dataset for real-world claim verification with evidence from the web. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 65128--65167

    2023

  38. [46]

    Nandan Thakur, Nils Reimers, Andreas R \"u ckl \'e , Abhishek Srivastava, and Iryna Gurevych. 2021. https://openreview.net/forum?id=wCu6T5xFjeJ BEIR : A heterogeneous benchmark for zero-shot evaluation of information retrieval models . In Thirty-fifth Conference on Neural Info...

    2021

  39. [47]

    James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. https://doi.org/10.18653/v1/N18-1074 FEVER : a large-scale dataset for fact extraction and VER ification . In Proceedings of the 2018 Conference of the North A merican Chapter of the Associatio...

    2018 doi

  40. [48]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  41. [49]

    Christophe Van Gysel and Maarten de Rijke. 2018. Pytrec\_eval: An extremely fast python interface to trec\_eval. In SIGIR. ACM

    2018

  42. [50]

    David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.609 Fact or fiction: Verifying scientific claims . In Proceedings of the 2020 Conference on Empirical Methods in Na...

    2020 doi

  43. [51]

    Liang Wang, Nan Yang, and Furu Wei. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.585 Query2doc: Query expansion with large language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9414--9423, Singapore. Association...

    2023 doi

  44. [52]

    Xiaohua Wang, Zhenghua Wang, Xuan Gao, Feiran Zhang, Yixin Wu, Zhibo Xu, Tianyuan Shi, Zhengyuan Wang, Shizheng Li, Qi Qian, Ruicheng Yin, Changze Lv, Xiaoqing Zheng, and Xuanjing Huang. 2024 a . https://doi.org/10.18653/v1/2024.emnlp-main.981 Searching for best practices in r...

    2024 doi

  45. [53]

    Yuxia Wang, Minghan Wang, and Preslav Nakov. 2024 b . https://aclanthology.org/2024.findings-eacl.65/ Rethinking STS and NLI in large language models . In Findings of the Association for Computational Linguistics: EACL 2024, pages 965--982, St. Julian ' s, Malta. Association f...

    2024

  46. [54]

    Cheng Xu, Shuhao Guan, Derek Greene, M Kechadi, et al. 2024 a . Benchmark data contamination of large language models: A survey. arXiv preprint arXiv:2406.04244

    2024 arXiv

  47. [55]

    Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. 2024 b . Benchmarking benchmark leakage in large language models. arXiv preprint arXiv:2404.18824

    2024 arXiv

  48. [56]

    Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. 2023. https://doi.org/10.18653/v1/2023.findings-acl.551 Do large language models know what they don`t know? In Findings of the Association for Computational Linguistics: ACL 2023, pages 8653--8665...

    2023 doi

  49. [57]

    Yejun Yoon, Jaeyoon Jung, Seunghyun Yoon, and Kunwoo Park. 2024. https://doi.org/10.18653/v1/2024.fever-1.15 H er O at AV eri T e C : The herd of open large language models for verifying real-world claims . In Proceedings of the Seventh Fact Extraction and VERification Worksho...

    2024 doi

  50. [58]

    Hanning Zhang, Shizhe Diao, Yong Lin, Yi Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. 2024. https://doi.org/10.18653/v1/2024.naacl-long.394 R -tuning: Instructing large language models to say I don`t know' . In Proceedings of the 2024 Conference of the ...

    2024 doi

  51. [59]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://openreview.net/forum?id=SkeHuCVFDr Bertscore: Evaluating text generation with bert . In International Conference on Learning Representations

    2020

  52. [60]

    Wayne Xin Zhao, Jing Liu, Ruiyang Ren, and Ji-Rong Wen. 2024. https://doi.org/10.1145/3637870 Dense text retrieval based on pretrained language models: A survey . ACM Trans. Inf. Syst., 42(4)

    2024 doi

  53. [61]

    Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, and Tat-Seng Chua. 2021. Retrieving and reading: A comprehensive survey on open-domain question answering. arXiv preprint arXiv:2101.00774

    2021 arXiv

  54. [62]

    Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ji-Rong Wen. 2023. Large language models for information retrieval: A survey. arXiv preprint arXiv:2308.07107

    2023

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.