{"id":"6097390c-31f8-42ff-af02-9ab50f2ae98e","arxiv_id":"2504.14175","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Query expansion helps fact-verification retrieval primarily when LLM-generated documents reproduce content entailed by gold evidence, indicating possible knowledge leakage in three common benchmarks.","lead":"What if LLM-based search tricks only work because the model has already memorized the answer? This paper finds that query expansion improves fact-checking retrieval mainly on claims where the model's generated text overlaps the gold evidence, suggesting benchmark leakage may be inflating reported gains.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Matched/unmatched performance gap is confounded by query expansion's intended mechanism; no control rules out general-knowledge generation, so leakage conclusion is unsupported without a post-cutoff/synthetic evidence test.","rationale":"The empirical work is careful: multiple datasets, models, repetitions, statistical tests, and manual validation of the NLI matcher (F1 0.8, kappa 0.837) are real strengths. The concern is not the measurements but the inference from the matched/unmatched split to knowledge leakage. The split is not exogenous: matching is defined by the generated document's semantic proximity to gold evidence, which is the very mechanism through which hypothetical-document QE is supposed to help. Thus the central correlation is fully consistent with QE functioning as intended, and it cannot discriminate between 'the LLM memorized the gold evidence' and 'the LLM generated a good hypothetical document from general knowledge.' The paper's own Limitation (1) concedes that no causal link is established, yet the Discussion and Abstract interpret the pattern as potential leakage. The proposed control—using post-cutoff or synthetic evidence—would settle whether the correlation depends on possible pretraining exposure. This is a stronger requirement than merely adding covariates for claim difficulty, because even after controlling for difficulty the pattern would not identify leakage. The reader's weakest assumption captures the same gap; I agree with it. The conditional verdict stands: the paper is a useful observation, but its headline leakage claim is not yet established.","tokens_in":25027,"tokens_out":5601,"duration_ms":49510,"concrete_test":"Construct a leakage-free control set from AVeriTeC or SciFact by keeping only claims whose gold evidence was published or uploaded after the QE model's training cutoff (e.g., AVeriTeC claims from 2023 for models with earlier cutoffs), or by generating synthetic claims from a private corpus never released to the LLM. Run the same Query2doc/HyDE pipelines and the Section 3.3 matching algorithm on this control set. If the matched-condition QE advantage persists (matched claims still show large gains over the no-expansion baseline), the central leakage interpretation fails, because no pretraining exposure is possible; if the advantage disappears, the leakage hypothesis is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central inference—that QE gains concentrate on 'matched' claims and therefore indicate knowledge leakage—rests on treating the matched condition as a proxy for pretraining exposure. But the matched condition is defined as the LLM-generated document containing a sentence entailed by gold evidence (Section 3.3). Under the standard hypothesis-document account of QE, this is precisely the condition under which expansion should help: a generated document that is semantically aligned with the gold evidence will, when concatenated or averaged into the query, retrieve that evidence more easily. Conversely, when the generation is inaccurate, expansion should hurt. The observed Recall/NDCG/F1 gap between M and ¬M is therefore exactly what QE is designed to produce, independent of any leakage. Additionally, claims that are easy for the LLM to generate gold-entailed sentences about are likely easier retrieval targets generally (well-covered topics, claim contains the answer, etc.), so the unmatched condition may be a proxy for claim difficulty rather than absence of leakage. The paper provides no control (e.g., claims with evidence published after the LLM's cutoff, or synthetic evidence the LLM cannot have memorized) to show the pattern changes when leakage is impossible. The Limitations section acknowledges this and disclaims a causal link, but the abstract and Discussion nevertheless present the correlation as suggesting leakage, which overstates the support.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript investigates whether the effectiveness of LLM-based query expansion (QE) in zero-shot evidence retrieval is partly an artifact of knowledge leakage. Using fact verification as a testbed, it applies Query2doc and HyDE over FEVER, SciFact, and AVeriTeC with seven backbone LLMs, and labels each claim as matched if the generated pseudo-document contains a sentence entailed by the gold evidence under a GPT-4o-mini NLI judge. The paper reports that both QE methods improve retrieval and verdict prediction on average, but that the gains concentrate on matched claims: performance on unmatched claims is often below the non-expanded baseline. From this correlation the paper concludes that knowledge leakage may be present in these benchmarks and may inflate the perceived performance of LLM-based query expansion. The Limitations section explicitly disclaims a causal link between leakage and generation, but the abstract, Discussion, and Conclusion present the correlation as suggestive evidence of leakage.","tokens_in":25210,"tokens_out":4010,"duration_ms":39131,"significance":"If the claimed leakage effect were established, the result would be important for interpreting a large body of zero-shot retrieval and retrieval-augmented generation evaluations, and it would motivate new benchmark designs and QE methods robust to unseen knowledge. The paper's strengths are its breadth and reproducibility: the matched-versus-unmatched pattern is consistent across three datasets, seven LLMs, two QE methods, and two values of k, with eight repetitions and standard errors, and the automatic NLI-based matching is validated against manual annotation on a 500-sample subset. However, the central inferential step from the matched/unmatched performance gap to knowledge leakage is not currently supported, because the matched condition is defined using the same gold evidence against which retrieval is scored. The empirical pattern is valuable as a benchmark finding, but the leakage interpretation requires additional controls that the manuscript does not provide.","major_comments":[{"comment":"The matched condition is defined by NLI entailment between an LLM-generated sentence and gold evidence, while the retrieval metrics in Table 4 are evaluated against that same gold evidence. Under the standard hypothesis-document account of QE, this is exactly the condition under which expansion should help: a generated document semantically aligned with the gold evidence should make that evidence easier to retrieve, whether or not the LLM ever saw the benchmark during pretraining. Conversely, an inaccurate generated document should hurt retrieval. The M-versus-unmatched gap in Table 4 is therefore predicted by the intended QE mechanism even with zero leakage, and the gap cannot by itself indicate knowledge leakage. The paper needs a control in which leakage is impossible, such as claims whose gold evidence was published after the model's training cutoff or evidence constructed synthetically, and should show that the matched/unmatched gap disappears or changes in that setting. This control should be added or the leakage claim should be substantially softened.","section":"Section 3.3, Table 4"},{"comment":"The matched/unmatched distinction is also confounded with claim difficulty. Claims for which the LLM can produce gold-entailed sentences are likely to be claims that are well covered in general knowledge, that contain the answer lexically, or that are otherwise easier retrieval targets; the unmatched set may therefore be proxying for hard claims rather than for absence of leakage. Since the manuscript does not control for baseline retrievability or claim difficulty, the claim that 'performance improvements from query expansion were consistent only when LLM-generated documents contained sentences entailed by gold evidence' (Section 5) conflates the leakage hypothesis with a difficulty confound. A concrete remedy would be to compare matched and unmatched claims with matched baseline retrieval difficulty, or to use a post-hoc matched-pair analysis where gold evidence difficulty is held fixed.","section":"Section 5, Table 4"},{"comment":"An NLI entailment match between a generated sentence and gold evidence establishes semantic equivalence, not pretraining exposure. The phrasing in Section 3.3 ('it may indicate that the backbone LLM was exposed to knowledge related to the corresponding evidence during training') is appropriately hedged, but Section 5 states that the seven LLMs 'were likely exposed to knowledge sources from the three benchmarks during training.' That stronger statement does not follow from the matched proportions in Table 3, because the same outputs could arise from general world knowledge, from the claim containing most of the evidence content, or from the model reconstructing a well-known fact without having memorized the specific benchmark document. The Discussion should explicitly acknowledge this alternative explanation or add a control that distinguishes memorization from reasoning over general knowledge.","section":"Section 3.3, Table 3, Section 5"}],"minor_comments":[{"comment":"In the sentence 'These methods assume that LLMs can generate hypothetical documents,' the phrase 'hypotheticaldocuments' appears without a space in the manuscript text; this should be corrected.","section":"Section 1"},{"comment":"The dataset name is inconsistently typeset as 'A VeriTeC', 'AVeriTeC', and 'AVeriTeC' across the body, tables, and appendix; the spelling should be standardized.","section":"Throughout"},{"comment":"The Mann-Whitney U test is mentioned without specifying whether it is applied to per-claim scores, per-repetition aggregate scores, or to the retrieval metrics directly; since only eight repetitions were run, the level of analysis affects interpretation of the reported p<0.001 values.","section":"Appendix C"},{"comment":"The ROUGE-2 threshold of 0.95 for removing claim reproductions is described as based on manual inspection but no sensitivity analysis is reported; a short description of how the threshold was chosen would improve reproducibility.","section":"Section 3.3"},{"comment":"Equation (3) uses the notation max_X with a boolean function X but does not define the constraint set for X; please specify that X is an assignment matrix and state the constraints used in the Hungarian algorithm.","section":"Appendix B, Equation (3)"},{"comment":"The text refers to 'Pyserini' in one place and 'PySerini' in another; the spelling should be made consistent.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The empirical pattern reported in Tables 4 and A2/A3 is consistent and well documented, and the manual validation of the NLI matching is a genuine strength. The main risk is that the paper's title and framing claim a leakage interpretation that the current experiments cannot support, because the matched condition is inherently tied to the gold evidence used for evaluation. The authors should be pushed to add a post-cutoff or synthetic-evidence control, or to reframe the paper as a study of the conditions under which QE helps rather than as evidence of benchmark contamination. If the control is infeasible, the paper's contribution is still publishable as a largely negative methodological finding, but the leakage framing would need to be removed from the title and abstract."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's real contribution is the conditional analysis: QE gains in fact verification concentrate on claims where the generated text overlaps gold evidence, and unmatched claims often lose. That pattern is measured carefully across three datasets, seven LLMs, two QE methods, and two k values, with repetitions and standard errors. The NLI-based matching is simple and the manual annotation check (kappa 0.837) gives it credibility. This is the first time someone has sliced QE performance by whether the generated document is entailed by gold evidence, and the result is genuinely useful for interpreting zero-shot retrieval results.\n\nBut the leap from that pattern to knowledge leakage is not supported. The matched condition is defined by NLI entailment with the gold evidence, and the retrieval metrics are scored against that same gold evidence. Under the standard hypothesis-document account of QE, a generated document that is semantically aligned with the gold evidence should retrieve that evidence more easily; when the generation is inaccurate, expansion should hurt. You do not need contamination to produce exactly the observed Recall/NDCG gap. Claim difficulty is another confounder: claims that are easy for the model to generate gold-entailed sentences about are probably easier retrieval targets regardless. The paper's own Limitations section says it does not claim a causal link between leakage and generation, and suggests a synthetic-evidence control as future work. That control is exactly what is needed to make the leakage interpretation stick. Without it, phrases like \"may be present\" in the abstract and discussion lean harder on the correlation than the evidence warrants.\n\nWhat survives the confound is the practical implication: QE only helps when the model can already produce text that looks like the evidence. For novel or niche claims, it can even hurt. That point stands whether the mechanism is benchmark contamination or general world knowledge, and it is worth stating clearly.\n\nThis paper deserves a serious referee. The empirics are solid, the writing is honest about limitations, and the diagnostic could influence how the IR/RAG community evaluates QE. I would ask the authors to either run a post-cutoff or synthetic-evidence control or reframe the claims away from leakage toward the dependence on parametric knowledge. As it stands, the leakage hypothesis is intriguing but not established.\n\nFor a reading group: worth discussing, especially the evaluation-design lesson. I would cite it as a cautionary example of how a condition defined by the metric can masquerade as an external cause.","headline":"A useful diagnostic for when QE helps, but the leakage conclusion is overclaimed; the matched/unmatched gap is the QE mechanism doing its job.","tokens_in":25810,"tokens_out":1765,"would_cite":true,"duration_ms":20156,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM query expansion helps only when models leak benchmark answers","keywords":["query expansion","knowledge leakage","fact verification","hypothetical documents","natural language inference","information retrieval","LLM memorization","zero-shot retrieval"],"falsifier":"Construct synthetic fact-verification claims from documents published after the target LLM's training cutoff, then run HyDE and Query2doc with that model. If query expansion still improves retrieval and verdict prediction on matched claims, or if the matched/unmatched performance gap persists without any possible pretraining exposure, the leakage interpretation is wrong.","tokens_in":24767,"feed_emoji":"🔍","tokens_out":4093,"duration_ms":30189,"temperature":0.7,"pith_summary":"This paper challenges the assumption behind LLM-based query expansion (QE) methods like HyDE and Query2doc: that a language model can generate a useful hypothetical document to help retrieve real evidence. Using fact verification as a testbed, the authors ask whether the apparent success of such methods is actually driven by knowledge leakage, where the LLM has seen the benchmark's gold evidence during pretraining and simply reproduces it. Across three benchmarks (FEVER, SciFact, AVeriTeC) and seven backbone LLMs, they find that QE improves retrieval and verdict prediction only when the generated documents contain sentences entailed by the gold evidence. When no such entailment is detected, QE often performs worse than the baseline retriever. The authors conclude that knowledge leakage may inflate the perceived effectiveness of LLM-based query expansion and that these methods may be of limited use in real-world fact-checking of novel or niche knowledge.","feed_headline":"LLM query expansion helps only when models leak benchmark answers","feed_subtitle":"Across three fact-checking benchmarks and seven LLMs, retrieval gains appear only when generated text matches gold evidence.","key_machinery":"The central mechanism is an NLI-based matching algorithm that classifies each claim into a matched or unmatched condition. The algorithm segments the LLM-generated document into sentences, removes sentences that merely reproduce the claim, and runs a natural language inference model (GPT-4o-mini) on each pair of gold evidence sentence and generated sentence; if any pair is labeled entailment, the claim is matched. This binary condition is then used to stratify retrieval and verification performance, and the paper's finding is the consistent gap between the two conditions.","core_discovery":"The paper establishes a conditional pattern: LLM-based query expansion is effective exactly when the LLM-generated document reproduces information entailed by the benchmark's gold evidence. Using an NLI-based matching algorithm to label each claim as matched (at least one generated sentence entailed by gold evidence) or unmatched, the authors show that across all tested LLMs and datasets, matched claims enjoy significant gains over both all claims and unmatched claims, while unmatched claims frequently fall below the baseline. The matched proportion is high, often between 40% and 80% depending on dataset, model, and method. The authors interpret this as evidence that the LLMs were exposed to benchmark knowledge sources during pretraining, and that the hypothetical documents are in large part recall of memorized content rather than genuine extrapolation.","pith_inferences":["A causal test of the leakage interpretation could be built by generating claims exclusively from documents published after the target LLM's training cutoff and checking whether QE gains disappear or the matched/unmatched gap persists.","The matched/unmatched split offers a practical sanity check for any retrieval benchmark: if a generation-based method only helps on instances where the generator already knows the answer, the benchmark's difficulty is not where it appears.","The same NLI matching could audit closed-book QA and retrieval-augmented generation pipelines to see whether performance gains are similarly concentrated in leaked instances.","The matched condition may be confounded with claim difficulty, since claims whose answers are easily inferred from the claim itself could be both more likely to be matched and easier to retrieve; a design controlling for claim-inherent answerability would sharpen the leakage interpretation."],"forward_implications":["If QE effectiveness depends on leakage, then reported gains on standard fact-verification benchmarks overestimate real-world value for novel or niche claims.","On unmatched claims, QE can actively hurt retrieval compared to using the claim alone, so practitioners should use generation-based expansion cautiously.","Benchmark creators should treat contamination as a confound and build evaluation sets with post-cutoff or synthetic knowledge.","The methodology can be adapted to other knowledge-intensive tasks as a leakage diagnostic, though the paper does not test those settings."],"supporting_citations":[{"why":"Introduces HyDE, the dense-retrieval query expansion method whose effectiveness is tested and re-interpreted.","marker":"(Gao et al., 2023)"},{"why":"Introduces Query2doc, the sparse-retrieval query expansion method used throughout the experiments.","marker":"(Wang et al., 2023)"},{"why":"Provides the FEVER benchmark, one of the three fact-verification datasets used.","marker":"(Thorne et al., 2018)"},{"why":"Provides the SciFact benchmark, one of the three fact-verification datasets used.","marker":"(Wadden et al., 2020)"},{"why":"Provides the AVeriTeC benchmark, one of the three fact-verification datasets used and the basis for manual annotation validation.","marker":"(Schlichtkrull et al., 2023)"},{"why":"Provides the Contriever dense retriever used as the baseline and encoder within HyDE.","marker":"(Izacard et al., 2021)"},{"why":"Supplies the BEIR harness from which the FEVER and SciFact test sets are drawn.","marker":"(Thakur et al., 2021)"},{"why":"Supplies the NLI prompting approach used by the matching algorithm.","marker":"(Wang et al., 2024b)"},{"why":"Provides the Pyserini BM25 implementation used as the Query2doc baseline and retriever.","marker":"(Lin et al., 2021)"}],"fun_headline_variants":["LLM query expansion gains only when models leak gold evidence","Query expansion helps only if LLM recalls benchmark answers","Retrieval gains tied to memorized answers, not genuine extrapolation","Fact-checking boosts vanish unless LLM leaks gold evidence","LLM expansion works only via benchmark knowledge leakage"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that an NLI entailment match between an LLM-generated sentence and gold evidence indicates the LLM was exposed to that specific evidence during pretraining, rather than that the model reconstructed the fact from general world knowledge or from the claim itself.","fun_headline_variants_meta":{"raw":{"variants":["LLM query expansion gains only when models leak gold evidence","Query expansion helps only if LLM recalls benchmark answers","Retrieval gains tied to memorized answers, not genuine extrapolation","Fact-checking boosts vanish unless LLM leaks gold evidence","LLM expansion works only via benchmark knowledge leakage"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000153,"raw_usage":{"total_tokens":1147,"prompt_tokens":824,"completion_tokens":323,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":440,"completion_tokens_details":{"reasoning_tokens":242}},"tokens_in":440,"tokens_out":323,"duration_ms":3338,"temperature":1.0,"reasoning_tokens":242,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:54:11.146379+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct synthetic fact-verification claims from documents published after the target LLM's training cutoff, then run HyDE and Query2doc with that model. If query expansion still improves retrieval and verdict prediction on matched claims, or if the matched/unmatched performance gap persists without any possible pretraining exposure, the leakage interpretation is wrong.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the AVeriTeC benchmark, one of the three fact-verification datasets used and the basis for manual annotation validation."}],"review_version":1}