Pith. sign in

REVIEW 4 major objections 2 minor 7 cited by

Search-Time Data Contamination

T0 review · 4 major / 2 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a measurable share of web-enabled agent scores comes from retrieving leaked answer keys rather than genuine reasoning.

desk verdict A legitimate measurement paper quantifying search-time data contamination; the 3%/15% figures are plausible but hinge on labeling and ablation details the abstract doesn't show. read the letter →

arxiv 2508.13180 v1 pith:VUH64C5C submitted 2025-08-12 cs.AI cs.LG

Ziwen Han , Meher Mankikar , Julian Michael , Zifan Wang This is my paper
classification cs.AIcs.LG
keywords search-timecontaminationdataleakageLLMagentsbenchmarkevaluationHuggingFaceanswercopyingretrievalintegrity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper identifies a new form of data leakage, search-time contamination, in which search-based LLM agents retrieve the actual evaluation dataset—questions and ground-truth answers—from a public platform during the test itself. Using logs from three standard benchmarks (Humanity's Last Exam, SimpleQA, and GPQA), the authors find that for about 3% of questions the agent's search results include the benchmark dataset hosted on HuggingFace, and the agent often explicitly acknowledges finding the question-answer pair. When HuggingFace is blocked, accuracy on this contaminated subset drops by roughly 15%, showing that the inflated scores on those questions came from copying rather than capability. The paper argues this form of leakage is distinct from training-data contamination and proposes reporting and benchmark-design practices to mitigate it.

What carries the argument

Search-time contamination (STC) is the central mechanism: a tool-using agent's retrieval step surfaces a source containing the test question, or a near-duplicate, together with its answer, letting the agent copy instead of infer. The paper's detection method relies on matching retrieved URLs and near-duplicate questions, plus agents' own reasoning-chain acknowledgments of discovering question-answer pairs on HuggingFace. The controlled intervention—blocking HuggingFace and observing a 15% accuracy drop on the flagged subset—is what turns the anecdotal retrieval events into a quantitative contamination estimate.

What would settle it

Run a fresh evaluation on a random sample of HLE, SimpleQA, and GPQA questions with HuggingFace blocked from the outset, and compare accuracy on the questions this paper would flag as contaminated; if the accuracy does not drop by roughly 15% relative to an unblocked run on the same questions, the contamination estimate fails to replicate.

Watch

Extended reading notes

Core claim

The central claim is that search-time contamination exists and is measurable: during evaluation, search-based agents retrieve the very benchmark datasets that contain their test questions alongside ground-truth labels, allowing them to copy answers. Analyzing logs from HLE, SimpleQA, and GPQA, the authors find that approximately 3% of questions trigger retrieval of the evaluation dataset from HuggingFace, and agents frequently acknowledge in their reasoning chains that they discovered question-answer pairs there. Blocking HuggingFace causes accuracy on that contaminated subset to fall by approximately 15%, a drop attributed to removing access to the leaked answers. The authors conclude that

Load-bearing premise

The 3% contamination estimate depends on reliably flagging every leaked question from agent logs and near-duplicate matching, with no undetected copying among the other 97% and no false positives, so any error in that classification directly changes both headline numbers.

Editorial extensions

If this is right

  • Benchmark scores for search-based agents may overstate true capability by the margin contributed by leaked-answer copying on the contaminated subset.
  • Evaluation of web-enabled agents should report whether known dataset hosts were reachable, and ideally run with such hosts blocked as a control.
  • Publicly hosted evaluation datasets create a repeatable leakage channel: millions of queries against the same benchmark can compound a small per-question leak into a serious validity threat.
  • Releasing full retrieval logs, as this paper does, enables independent auditing of whether reported scores are contaminated.
  • The proposed best practices could generalize to other public dataset repositories beyond HuggingFace, making them a template for trustworthy agent evaluation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 3% figure is likely a lower bound: reasoning-chain acknowledgments will miss cases where an agent copies answers without explicitly naming HuggingFace, so the true contamination rate on these benchmarks could be higher.
  • The same mechanism probably affects other agent benchmarks whose evaluation sets are public on HuggingFace or similar platforms, not just the three studied here.
  • Blocking HuggingFace at evaluation time is a stopgap; a more durable fix is to use evaluation items that are non-public, dynamically generated, or otherwise not retrievable by the agent's own search tools.
  • A natural next experiment would test whether alternative hosts—GitHub repositories, personal websites, or cached search results—produce the same accuracy drop when blocked, which would show whether STC extends beyond HuggingFace.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 2 minor

Summary. The paper identifies 'search-time contamination' (STC): when search-based LLM agents retrieve evaluation datasets (e.g., on HuggingFace) containing test questions and ground-truth answers, they can copy answers rather than reason, undermining benchmark validity. Based on agent logs, the authors report that on HLE, SimpleQA, and GPQA, approximately 3% of questions trigger retrieval of the answer-bearing datasets, and that blocking HuggingFace reduces accuracy on this contaminated subset by approximately 15%. They also report ablations suggesting that HuggingFace may not be the only source of STC, propose best practices for benchmark design and reporting, and release the full experimental logs.

Significance. If the quantitative claims hold, the paper makes a valuable and timely contribution to LLM-agent evaluation. Measuring a concrete, repeated leakage channel (public evaluation datasets retrievable at inference time) is an important complement to training-data contamination studies. The proposed intervention (blocking the source and measuring subset accuracy) is a clean comparison in principle, and the public release of logs is a strong reproducibility practice. The paper's significance is therefore potentially high, but it rests entirely on the reliability of its contamination labels and on the causal isolation of the HuggingFace-blocking intervention.

major comments (4)
  1. [Abstract (contamination labeling)] Both headline numbers depend on how the contaminated subset is identified. The abstract mentions two signals—matching retrieved HuggingFace URLs (presumably with near-duplicate question matching) and agents' in-chain acknowledgments—but gives no validation of either. If the near-duplicate threshold is arbitrary or the self-reports are unreliable, the 3% rate and the 15% drop estimate are biased in unknown directions. The revised manuscript must describe the labeling protocol, report precision/recall against manual annotation, give inter-annotator agreement or audit results, and show sensitivity to the matching threshold.
  2. [Abstract (blocking intervention)] The causal interpretation of the 15% drop assumes that blocking HuggingFace affects only the ability to retrieve the leaked answer key. Blocking a major domain can change retrieval order, cause timeouts, or alter the entire search trajectory, potentially affecting accuracy even on uncontaminated questions. The paper needs control analyses: for example, accuracy on the uncontaminated subset before/after blocking, per-benchmark breakdowns, and a discussion of other domains that become more prominent after the block. Without these, the 15% figure cannot be attributed specifically to loss of the answer key.
  3. [Abstract (per-benchmark and aggregate stability)] The abstract reports aggregate figures across HLE, SimpleQA, and GPQA. It is not clear whether the approximately 3% contamination rate and approximately 15% drop are consistent across benchmarks or driven by one dataset. A per-benchmark table with confidence intervals is necessary to assess whether the conclusions are robust or an artifact of a single benchmark's formatting or public accessibility.
  4. [Abstract (ablation for other STC sources)] The authors state that ablation experiments show HuggingFace may not be the sole source of STC, but the abstract does not explain how these ablations are designed or what other sources are identified. Since the paper's proposed best practices depend on the breadth of STC, the revised manuscript should specify the alternative sources, the method for detecting them, and the estimated fraction of contamination they explain.
minor comments (2)
  1. [Abstract (language)] Minor grammatical and hyphenation issues: 'search based agent logs' should be 'search-based agent logs'; 'HuggingFace, an online platform hosting evaluation datasets, appears among retrieved sources' could be rephrased for clarity. Also consider defining 'approximately 3%' and 'approximately 15%' with explicit counts or confidence intervals in the abstract.
  2. [Abstract (terminology)] The phrase 'data contamination' is used for training-data leakage while 'search-time contamination' is introduced for retrieval-time leakage. The relationship between the two—and whether STC can also be viewed as test-time leakage—should be clarified in the introduction so readers do not conflate them.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reported quantities are direct empirical measurements, not derivations that reduce to their own inputs.

full rationale

This is an empirical measurement study rather than a formal derivation. The two headline numbers—approximately 3% of questions contaminated and approximately 15% accuracy drop after blocking HuggingFace—are direct observational or interventional statistics, not quantities fitted to or implied by the same data. The contaminated subset is operationally defined as questions whose retrieval logs show HuggingFace datasets containing ground-truth labels, or whose reasoning chains acknowledge finding question-answer pairs; measuring the frequency and downstream accuracy of that subset is consistent with the definition, but it is not a circular derivation. The ablation compares that subset with HuggingFace blocked, and the observed drop is a contingent empirical outcome, not a mathematically forced consequence of the labeling rule. No self-citations, uniqueness theorems, or imported ansatze appear in the abstract. Concerns about imperfect labels (self-report honesty, near-duplicate matching thresholds) or about blocking HuggingFace perturbing search trajectories independently of answer-key access are validity and causal-identification concerns, not circularity; under the reviewing rules they belong in a correctness-risk assessment rather than raising the circularity score. Accordingly, no circular step can be exhibited, and the score is 0.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

This is a measurement paper, so the ledger holds no fitted constants in the physics sense and no invented objects. STC is a named failure mode, not a postulated entity. The load-bearing assumptions are operational: log fidelity, contaminated-subset labeling accuracy, and causal isolation of the platform block. Each is checkable against the released logs, which is why the paper earns a middle reproducibility score despite this.

free parameters (1)
  • Near-duplicate matching threshold for contamination detection
    STC is defined to include the test question 'or a near-duplicate' surfacing with its answer; the rule that decides which retrieved sources count as the leaked dataset is a decision parameter not reported in the abstract, and it determines membership in the contaminated subset.
assumptions (3)
  • domain assumption The retrieval logs and reasoning traces faithfully record every source the search-based agent actually saw and used.
    STC detection consists of finding HuggingFace among retrieved sources and reading agent acknowledgments; if the search tool truncates, deduplicates, or omits sources, the 3% prevalence estimate is biased. Invoked implicitly throughout the measurement.
  • domain assumption The contaminated-subset labeling (near-duplicate question matching and/or in-chain acknowledgment) has negligible false negatives and false positives.
    Both headline numbers, 3% prevalence and the 15% post-block drop, are computed on this labeled subset; labeling error directly rescales both figures. The specific operationalization is not described in the abstract.
  • domain assumption Blocking HuggingFace removes only the leaked-answer channel and otherwise leaves agent behavior unchanged.
    The ~15% drop is interpreted causally as the effect of removing STC. If HuggingFace also supplied legitimate neutral facts, or if agents partially compensate for the block with other sources, the drop misestimates contamination's contribution. The abstract's own ablation concedes other sources exist.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Search-Time Data Contamination." pith.science (2026). https://pith.science/paper/VUH64C5C

@misc{pith2026250813180,
  author       = {Pith},
  title        = {Pith review of: Search-Time Data Contamination},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VUH64C5C}},
  note         = {Machine review of arXiv:2508.13180}
}
read the original abstract

Data contamination refers to the leakage of evaluation data into model training data, resulting in overfitting to supposedly held-out test sets and compromising test validity. We identify an analogous issue, search-time contamination (STC), in evaluating search-based LLM agents which use tools to gather information from online sources when answering user queries. STC occurs when the retrieval step surfaces a source containing the test question (or a near-duplicate) alongside its answer, enabling agents to copy rather than genuinely infer or reason, undermining benchmark integrity. We find that HuggingFace, an online platform hosting evaluation datasets, appears among retrieved sources in search based agent logs. Consequently, agents often explicitly acknowledge discovering question answer pairs from HuggingFace within their reasoning chains. On three commonly used capability benchmarks: Humanity's Last Exam (HLE), SimpleQA, and GPQA, we demonstrate that for approximately 3% of questions, search-based agents directly find the datasets with ground truth labels on HuggingFace. When millions of evaluation queries target the same benchmark, even small, repeated leaks can accelerate the benchmark's obsolescence, shortening its intended lifecycle. After HuggingFace is blocked, we observe a drop in accuracy on the contaminated subset of approximately 15%. We further show through ablation experiments that publicly accessible evaluation datasets on HuggingFace may not be the sole source of STC. To this end, we conclude by proposing best practices for benchmark design and result reporting to address this novel form of leakage and ensure trustworthy evaluation of search-based LLM agents. To facilitate the auditing of evaluation results, we also publicly release the complete logs from our experiments.

Discussion (0). Sign in to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Reasoning Gets Harder for LLMs Inside A Dialogue

    cs.CL 2026-03 unverdicted novelty 7.0 of 10

    LLMs show a consistent performance drop on arithmetic, spatial, and temporal reasoning tasks when framed in multi-turn dialogues versus isolated settings, demonstrated by the new BOULDER benchmark across eight travel-...

  2. ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to Answer and Judge

    cs.CL 2025-10 unverdicted novelty 7.0 of 10

    ProfBench is a new multi-domain benchmark with human-expert rubrics for judging LLM responses on professional tasks, showing top models reach only 65.9% performance while providing cheap LLM judges that reduce evaluat...

  3. MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than Outcomes

    cs.CL 2025-10 conditional novelty 7.0 of 10

    MoReBench evaluates LLM moral reasoning processes with 1,000 expert-rubric scenarios and finds current models are partial toward Utilitarian and Deontological frameworks.

  4. How Benchmarks Mis-Score Computer-Use Agents

    cs.AI 2026-07 conditional novelty 6.0 of 10

    About 15.3% of FAIL verdicts across five CUA benchmarks are wrong, and genuine failures are mostly verification and planning errors, not clicks.

  5. Every Model Cheats: Prompt-Level Mitigation of Cheating on Offensive Cyber Tasks

    cs.CR 2026-07 conditional novelty 6.0 of 10

    On Cybench CTF tasks, 21 of 22 models cheated under baseline, 37% of baseline passes were cheated, and stricter anti-cheat prompts reduced but did not remove cheating.

  6. ToolAnchor: Anchoring Counterfactual Context to Boost Agentic Tool-use Capability

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A three-stage teacher-hypothesize, student-verify, then-train pipeline lets a post-trained tool-using agent adopt new visual tools without retraining from scratch.

  7. SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agents

    cs.AI 2025-09 conditional novelty 6.0 of 10

    A 20B autonomously reasoning deep-research agent trained with synthetic-data RL reaches 28.7% on Humanity's Last Exam, exceeding several larger and proprietary baselines.

Reference graph

Works this paper leans on

20 extracted references · 10 canonical work pages · cited by 7 Pith papers

  1. [1]

    Chandrahasan, J

    P. Chandrahasan, J. Jin, Z. Zhang, T. Wang, A. Tang, L. Mo, M. Ziyadi, L. F. R. Ribeiro, Z. Qiu, M. Dreyer, A. Asai, and C. Xiong. Deep research comparator: A platform for fine-grained human annotations of deep research agents. 2025. URL https://api.semanticscholar.org/CorpusID:280048405

    work page 2025

  2. [2]

    DeepMind

    G. DeepMind. Gemini deep research - your personal research assistant. URL https://gemini.google/overview/deep-research/

    work page

  3. [3]

    C. Deng, Y. Zhao, Y. Heng, Y. Li, J. Cao, X. Tang, and A. Cohan. Unveiling the spectrum of data contamination in language models: A survey from detection to remediation, 2024. URL https://arxiv.org/abs/2406.14644

    arXiv 2024

  4. [4]

    Y. Dong, X. Jiang, H. Liu, Z. Jin, B. Gu, M. Yang, and G. Li. Generalization or memorization: Data contamination and trustworthy evaluation for large language models, 2024. URL https://arxiv.org/abs/2402.15938

    arXiv 2024

  5. [5]

    M. Du, B. Xu, C. Zhu, X. Wang, and Z. Mao. Deepresearch bench: A comprehensive benchmark for deep research agents. ArXiv, abs/2506.11763, 2025. URL https://api.semanticscholar.org/CorpusID:279391682

    arXiv 2025

  6. [6]

    B. Gou, Z. Huang, Y. Ning, Y. Gu, M. Lin, W. Qi, A. Kopanev, B. Yu, B. J. Gutiérrez, Y. Shu, C. H. Song, J. Wu, S. Chen, H. N. Moussa, T. Zhang, J. Xie, Y. Li, T. Xue, Z. Liao, K. Zhang, B. Zheng, Z. Cai, V. Rozgic, M. Ziyadi, H. Sun, and Y. Su. Mind2web 2: Evaluating agentic search with agent-as-a-judge, 2025

    work page 2025

  7. [7]

    Laine, B

    R. Laine, B. Chughtai, J. Betley, K. Hariharan, M. Balesni, J. Scheurer, M. Hobbhahn, A. Meinke, and O. Evans. Me, myself, and ai: The situational awareness dataset (sad) for llms. Advances in Neural Information Processing Systems, 37: 0 64010--64118, 2024

    work page 2024

  8. [8]

    Miroyan, T.-H

    M. Miroyan, T.-H. Wu, L. King, T. Li, J. Pan, X. Hu, W.-L. Chiang, A. N. Angelopoulos, T. Darrell, N. Norouzi, and J. Gonzalez. Search arena: Analyzing search-augmented llms. ArXiv, abs/2506.05334, 2025. URL https://api.semanticscholar.org/CorpusID:279243096

    arXiv 2025

Show all 20 references
  1. [9]

    URL https://openai.com/index/introducing-deep-research/

    OpenAI, a . URL https://openai.com/index/introducing-deep-research/

  2. [10]

    URL https://cdn.openai.com/pdf/8124a3ce-ab78-4f06-96eb-49ea29ffb52f/gpt5-system-card-aug7.pdf

    OpenAI, b . URL https://cdn.openai.com/pdf/8124a3ce-ab78-4f06-96eb-49ea29ffb52f/gpt5-system-card-aug7.pdf

  3. [11]

    URL https://www.perplexity.ai/hub/blog/introducing-perplexity-deep-research

    Perplexity. URL https://www.perplexity.ai/hub/blog/introducing-perplexity-deep-research

  4. [12]

    L. Phan, A. Gatti, Z. Han, N. Li, J. Hu, H. Zhang, C. B. C. Zhang, M. Shaaban, J. Ling, S. Shi, et al. Humanity's last exam. arXiv preprint arXiv:2501.14249, 2025

    2025 arXiv

  5. [13]

    D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024

    2024

  6. [14]

    Y. Song, K. Thai, C. M. Pham, Y. Chang, M. Nadaf, and M. Iyyer. Bearcubs: A benchmark for computer-using web agents. ArXiv, abs/2503.07919, 2025. URL https://api.semanticscholar.org/CorpusID:276929324

    2025 arXiv

  7. [15]

    Srivastava, A

    A. Srivastava, A. Rastogi, A. Rao, A. A. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on machine learning research, 2023

    2023

  8. [16]

    J. Wei, N. Karina, H. W. Chung, Y. J. Jiao, S. Papay, A. Glaese, J. Schulman, and W. Fedus. Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368, 2024

    2024 arXiv

  9. [17]

    J. Wei, Z. Sun, S. Papay, S. McKinney, J. Han, I. Fulford, H. W. Chung, A. T. Passos, W. Fedus, and A. Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12516, 2025

    2025 arXiv

  10. [18]

    URL https://x.ai/news/grok-4

    XAI. URL https://x.ai/news/grok-4

  11. [19]

    Y. Xi, J. Lin, M. Zhu, Y. Xiao, Z. Ou, J. Liu, T. Wan, B. Chen, W. Liu, Y. Wang, R. Tang, W. Zhang, and Y. Yu. Infodeepseek: Benchmarking agentic information seeking for retrieval-augmented generation. ArXiv, abs/2505.15872, 2025. URL https://api.semanticscholar.org/CorpusID:278788862

    2025 arXiv

  12. [20]

    Zhang, J

    H. Zhang, J. Da, D. Lee, V. Robinson, C. Wu, W. Song, T. Zhao, P. Raja, C. Zhuang, D. Slack, et al. A careful examination of large language model performance on grade school arithmetic. Advances in Neural Information Processing Systems, 37: 0 46819--46836, 2024

    2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.