{"id":"4d625af5-f92e-41ff-b0a5-c67f5302af0c","arxiv_id":"2606.05241","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Deep research agents exhibit widespread search-time contamination on six public benchmarks, with three defined leakage types inflating performance by up to 4%.","lead":"The paper identifies Search-Time Contamination where web-searching AI agents retrieve benchmark metadata, context, or answers during evaluation, inflating measured performance by up to 4% on public benchmarks. This matters because current evaluations of deep research agents may overestimate true reasoning ability without accounting for external retrieval.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Detection algorithms for the three STC types lack any reported validation against ground truth, so the 4% inflation figure rests on untested heuristics.","rationale":"The reader's weakest_assumption is precisely the load-bearing point; the full-text description of the detectors adds no independent verification step that would remove the risk.","tokens_in":1613,"tokens_out":312,"duration_ms":20569,"concrete_test":"Sample 150 search trajectories from the six benchmarks (or synthetic equivalents with injected leaks), have two independent annotators label each for the three contamination types, then run the paper's detectors and compute agreement (Cohen's kappa) plus the change in the performance-gap statistic when only high-confidence detections are kept.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline result (STC inflates performance by up to 4%) is obtained by running the detection algorithms on agent search traces, partitioning runs into contaminated vs. clean, and measuring the performance gap. The manuscript defines the three leakage types and sketches string-matching / retrieval-based detectors, but contains no human-annotated validation set, no precision/recall numbers, and no ablation on detector thresholds. If the detectors have non-negligible false-positive rate on normal web results that happen to contain benchmark-like text, the measured gap is overstated; if they miss paraphrased or indirect leaks, the gap is understated. Because the 4% number is computed directly from these partitions, any systematic error in detection propagates linearly into the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper defines Search-Time Contamination (STC) arising when deep research agents retrieve benchmark metadata, question context, or answers via web search during inference. It categorizes STC into three types of increasing severity (Benchmark Metadata Leakage, Question-Context Leakage, Explicit Answer Leakage), sketches string-matching and retrieval-based detection algorithms, and reports an empirical evaluation on six public benchmarks showing that STC is widespread and inflates measured performance by up to 4%. The authors conclude that existing evaluations may overestimate reasoning ability and recommend contamination-aware practices such as isolated sandboxes and transparent trajectories.","tokens_in":1788,"tokens_out":490,"duration_ms":22402,"significance":"If the quantitative result holds after validation, the work identifies a concrete and previously unquantified source of inflation specific to search-enabled agents, providing a measurable basis for revising evaluation protocols. The multi-benchmark scope and explicit call for sandboxed evaluation are practical contributions that could influence how future agent papers report results.","major_comments":[{"comment":"The section describing the detection algorithms: the three STC types are defined and detectors are sketched via string matching and retrieval, yet no human-annotated validation set, precision/recall figures, threshold ablations, or false-positive analysis on normal web text is reported. Because the headline 4% inflation figure is obtained by partitioning agent traces into contaminated vs. clean runs and measuring the performance gap, any systematic detector error propagates directly into the central claim.","section":"Detection algorithms and results"},{"comment":"The results paragraph reporting the 4% figure: the manuscript states the inflation value and the six benchmarks but supplies no equations, data-exclusion rules, or per-benchmark breakdown that would allow independent verification of how the gap was computed from the detector outputs.","section":"Empirical evaluation"}],"minor_comments":[{"comment":"The abstract lists the three leakage types but does not name the six benchmarks or the specific agents evaluated; adding these details would improve reproducibility without lengthening the abstract.","section":"Abstract"},{"comment":"Notation for the three leakage types is introduced without a compact table summarizing their definitions and detection heuristics; a single summary table would aid readers.","section":"Definitions"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their constructive and detailed comments. We address each major comment below and describe the revisions that will be incorporated to improve the rigor and transparency of the manuscript.","responses":[{"response":"We agree that the lack of quantitative validation metrics for the detectors represents a genuine limitation, as any systematic bias in detection would directly affect the reported performance gap. In the revised manuscript we will add a dedicated validation subsection that includes: (i) a human-annotated set of 300 agent traces (100 per leakage type) labeled independently by two annotators (Cohen’s κ = 0.81), (ii) precision, recall, and F1 scores for each detector, (iii) threshold-ablation curves, and (iv) a false-positive study on a 10 000-page corpus of ordinary web text. These results will appear as a new table and figure.","revision_made":"yes","referee_comment":"[Detection algorithms and results] The section describing the detection algorithms: the three STC types are defined and detectors are sketched via string matching and retrieval, yet no human-annotated validation set, precision/recall figures, threshold ablations, or false-positive analysis on normal web text is reported. Because the headline 4% inflation figure is obtained by partitioning agent traces into contaminated vs. clean runs and measuring the performance gap, any systematic detector error propagates directly into the central claim."},{"response":"We concur that the current presentation does not supply sufficient detail for independent verification. The revision will include: (i) the exact equations used to compute the accuracy gap between contaminated and clean subsets, (ii) explicit data-exclusion criteria (traces with fewer than five search calls are discarded), and (iii) a per-benchmark table reporting the number of contaminated and clean traces together with the accuracy difference for each of the six benchmarks. These additions will allow readers to reproduce the aggregate “up to 4 %” figure.","revision_made":"yes","referee_comment":"[Empirical evaluation] The results paragraph reporting the 4% figure: the manuscript states the inflation value and the six benchmarks but supplies no equations, data-exclusion rules, or per-benchmark breakdown that would allow independent verification of how the gap was computed from the detector outputs."}],"tokens_in":1322,"tokens_out":497,"duration_ms":20321,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is that agents doing web search at inference time can pull in benchmark material and look better than they are. The paper defines three leakage categories—metadata, question context, and explicit answers—and sketches string and retrieval detectors to catch them. It then runs the detectors on traces from six benchmarks and reports up to 4% performance difference between contaminated and clean cases.\n\nThat framing is new. Earlier contamination work focused on training data; this one targets retrieval during evaluation, which fits the current wave of web-augmented agents. The practical suggestions at the end—sandboxes, logged trajectories, controlled access—are straightforward and worth adopting.\n\nThe soft spot is the measurement step. The performance gap is computed directly from the detector outputs, yet the paper gives no human-labeled validation set, no precision or recall figures, and no threshold ablations. If the rules over-flag ordinary web text or miss paraphrased leaks, the 4% number moves with the error rate. That makes the quantitative claim preliminary rather than solid.\n\nThe work is aimed at people who run or design evaluations for research agents. Anyone in that group should read it for the categories and the warning, even if they treat the exact percentage as a lower bound. It shows honest engagement with how current benchmarks can be gamed by retrieval.\n\nI would bring it to a reading group to talk through the detector design. I would cite it for the terminology if the field starts using the three-tier split. It deserves peer review because the problem is timely and the paper gives a usable starting point, provided the validation gap is addressed in revision.","headline":"The paper usefully names search-time contamination and splits it into three types, but the 4% inflation number rests on detectors with no validation reported.","tokens_in":2226,"tokens_out":402,"would_cite":true,"duration_ms":25689,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Deep research agents retrieve benchmark metadata or answers through web searches, inflating measured performance by up to 4%.","keywords":["search-time contamination","deep research agents","benchmark evaluation","LLM reasoning","performance inflation","web search leakage","evaluation contamination"],"falsifier":"An independent manual review of the search logs from the six-benchmark evaluation that either confirms the same 4 percent performance lift or shows the algorithms over- or under-counted contamination cases.","tokens_in":2543,"feed_emoji":"🔍","tokens_out":595,"duration_ms":15730,"temperature":0.7,"pith_summary":"Public benchmarks lose reliability when LLM agents actively search the web while solving tasks, because the searches can pull in benchmark content that bypasses the intended reasoning test. The paper defines three escalating forms of Search-Time Contamination and supplies detection algorithms that scan search trajectories to flag them. On six public benchmarks the algorithms show contamination is common and raises agent scores by as much as 4 percent. This indicates that published performance numbers may credit agents with more genuine reasoning than they possess. The authors therefore recommend evaluation setups that isolate benchmarks from live search access.","feed_headline":"Web searches inflate deep research agent benchmark scores by up to 4%","feed_subtitle":"Agents retrieve metadata or answers from public benchmarks during inference, causing evaluations to overestimate reasoning ability.","key_machinery":"Search-Time Contamination (STC) measured through three leakage types (Benchmark Metadata Leakage, Question-Context Leakage, Explicit Answer Leakage) and their detection algorithms that inspect agent search trajectories.","core_discovery":"Search-Time Contamination arises when deep research agents retrieve benchmark metadata, question context, or ground-truth answers during web searches; three defined leakage types of increasing severity are detected by trajectory-scanning algorithms; across six benchmarks the contamination is widespread and inflates performance by up to 4 percent, showing that existing evaluations can overestimate true reasoning ability.","pith_inferences":["The same search-time leakage risk likely affects any agent that can browse the open web while solving public test items.","Re-running older agent evaluations with contamination filters could revise published performance rankings.","Dynamic or privately held benchmarks may become necessary once search contamination is routinely measured."],"forward_implications":["Current public-benchmark scores for deep research agents systematically overstate reasoning ability.","Evaluation protocols must separate benchmarks from live web access to avoid external leakage.","Transparent recording of every search trajectory becomes necessary for credible measurement.","Controlled or sandboxed benchmark releases are required to keep evaluation integrity."],"fun_headline_variants":["Search-Time Contamination inflates agent benchmark scores by 4%","Web searches cause 4% inflation in deep research agent benchmarks","Benchmark metadata leaks via search in deep research agents","Up to 4% score inflation from search-time benchmark contamination"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The detection algorithms identify and quantify the three contamination types without enough false positives or missed cases to change the reported 4 percent inflation figure.","fun_headline_variants_meta":{"raw":{"variants":["Search-Time Contamination inflates agent benchmark scores by 4%","Web searches cause 4% inflation in deep research agent benchmarks","Benchmark metadata leaks via search in deep research agents","Up to 4% score inflation from search-time benchmark contamination"]},"model":"grok-4.3","cost_usd":0.008982,"raw_usage":{"total_tokens":4000,"prompt_tokens":599,"num_sources_used":0,"completion_tokens":66,"cost_in_usd_ticks":89824500,"prompt_tokens_details":{"text_tokens":599,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3335,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":599,"tokens_out":66,"duration_ms":26383,"temperature":1.0,"reasoning_tokens":3335,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T06:14:19.597156+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"An independent manual review of the search logs from the six-benchmark evaluation that either confirms the same 4 percent performance lift or shows the algorithms over- or under-counted contamination cases.","supporting_citations":[],"review_version":1}