{"id":"f4d01402-035a-46f7-86fa-064fe19afcb3","arxiv_id":"2508.07999","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new bilingual benchmark shows current AI search agents complete fewer than 5% of wide-scale information collection tasks, while humans can reach near 100%.","lead":"This paper introduces WideSearch, a benchmark of 200 questions in English and Chinese that tests whether AI search agents can collect large volumes of verified information from the web. Across 10 leading agentic search systems, most scored near 0% success and the best only 5%, while human testers could reach near 100%.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Near-zero agent success may stem from an over-strict automatic answer checker that penalizes semantically correct paraphrases; without evidence of normalization, the headline deficiency claim is unsubstantiated.","rationale":"The reader's verdict (UNVERDICTED) is appropriate given the abstract-only review and lack of access to the dataset or evaluation code. My stress-test identifies a specific, load-bearing concern that goes beyond 'insufficient information': the validity of the automatic scoring mechanism. If the checker requires exact matches, the empirical results are meaningless. This concern is concrete and testable with a human-annotation study. It is not an ad hominem or a consensus disagreement; it is a standard risk in benchmark construction. The paper's own claim of 'objective verification' actually raises the suspicion that the checker is string-based. A rigorous five-stage QC pipeline cannot by itself rule out false negatives—it only ensures the ground-truth facts are correct and complete, not that all legitimate agent outputs are recognized. Therefore, the paper's central claim should be accepted only after the proposed test demonstrates that automatic and human scoring converge. I recommend CONDITIONAL rather than REJECT because the authors have released the dataset and code, so the test is straightforward to run, and the benchmark may still be valuable even if the scoring needs adjustment. The reader and I partially agree: they mention 'verifiable' and 'artifacts' generally; I sharpen this to the scoring rubric's semantic tolerance.","tokens_in":696,"tokens_out":4016,"duration_ms":54068,"concrete_test":"Sample 20–30 questions from the released WideSearch dataset and select 2–3 of the top-performing agent systems from the paper. Run these agents to produce outputs. Have two independent human annotators, blind to the official scores, judge each atomic fact in the agent outputs for semantic correctness against the ground-truth fact list, allowing paraphrases and equivalent expressions. Separately, apply the official automatic scorer to the same outputs. Compute the success rates (fraction of questions where all facts are correct) under human vs. automatic scoring. If human-assessed success is significantly higher (e.g., >20 percentage points), the automatic checker is too strict and the near-0% claim does not robustly measure agent capability.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim—that SOTA agentic search systems achieve near-0% success while humans reach ~100%—rests entirely on the automatic evaluation's ability to credit agents for correctly retrieved information. The abstract states that atomic facts are 'verified one by one objectively,' but objective verification is often implemented as exact or near-exact string matching. If the ground-truth facts are expressed in a canonical form and the checker does not accommodate synonyms, reordering, or other natural paraphrases, then any agent that returns the correct fact with slightly different wording is scored as wrong. This is a well-documented failure mode in QA benchmarks (e.g., SQuAD exact-match issues). Human testers, by contrast, can be instructed to match the expected format or are evaluated leniently, inflating human scores. The paper's five-stage QC pipeline may ensure facts are unambiguous and verifiable, but it does not guarantee the checker handles semantic equivalence. Without a description of the answer normalization step—or an empirical demonstration that human-judged and automatic scores agree—the near-0% numbers are uninterpretable: they could reflect agent deficiency or merely a rigid scoring rubric. This is the weakest assumption because every other conclusion in the abstract derives from these scores.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces WideSearch, a benchmark of 200 manually curated questions (100 English, 100 Chinese) from more than 15 domains, grounded in real user queries, for evaluating agentic broad information-seeking. It reports a five-stage quality-control pipeline intended to ensure difficulty, completeness, and verifiability, and evaluates 10+ state-of-the-art agentic search systems, claiming that most achieve near 0% overall success, the best reaches 5%, while human testers can reach near 100% given sufficient time. The dataset, evaluation pipeline, and benchmark results are stated to be publicly released.","tokens_in":1008,"tokens_out":3024,"duration_ms":38578,"significance":"If the reported numbers are robust, WideSearch would be a valuable and much-needed benchmark for an underevaluated capability of LLM-based search agents. The claimed strengths are real: manual curation grounded in real queries, bilingual coverage, comparison of multiple systems, and an explicit human cross-validation component. The public release of dataset and evaluation code is a concrete contribution. However, the central claim—that current agents are catastrophically deficient at wide information-seeking—depends entirely on the fairness and semantic sensitivity of the automatic scoring procedure. The abstract alone does not establish that near-zero scores reflect agent deficiency rather than a rigid or incomplete answer rubric, and the absence of error bars and human-evaluation protocol details makes the headline numbers difficult to interpret. These issues are load-bearing and need to be addressed in the full paper.","major_comments":[{"comment":"The claim that atomic facts are 'verified one by one objectively' is the linchpin of the near-0% vs. near-100% gap, but the abstract gives no information about answer normalization. If the checker uses exact or near-exact string matching, semantically correct paraphrases, synonyms, or reorderings may be marked wrong, making the agent scores uninterpretable. Please provide the evaluation code, a paraphrase-sensitivity analysis (e.g., scoring human-judged correct answers with the automatic checker), or agreement statistics between the automatic checker and human judges on the same agent outputs.","section":"Abstract (automatic verification)"},{"comment":"No confidence intervals or variance measures are reported for the success rates. With 200 questions, a 0/200 result has a one-sided 95% upper bound of about 1.5%, and the difference between 0% and 5% may not be statistically meaningful across systems. Report per-system success rates with binomial confidence intervals, per-question pass/fail counts, and an analysis of whether failures cluster on a few pathological questions or are spread across all questions. This would distinguish a genuinely hard benchmark from an evaluation artifact.","section":"Abstract (results reporting)"},{"comment":"The near-100% human success rate is reported without a protocol. If humans were allowed unlimited time, iterative searching, and lenient human judgment while agents were scored by an automatic checker, the human-agent gap is inflated by design. Specify the human evaluation setup: number of testers, time limits, interface, and, critically, whether human responses were scored by the same automatic checker or by human assessors. If the latter, the comparison is not apples-to-apples.","section":"Abstract (human evaluation)"},{"comment":"The five-stage quality-control pipeline is invoked as the primary evidence that questions are fair, complete, and verifiable, but no stage is described. To assess the benchmark's construct validity, the paper should specify the five stages, report inter-annotator agreement on answer completeness, and give examples of questions that were modified or discarded during curation. Without this, the near-zero agent scores could be an artifact of idiosyncratic question difficulty or ambiguous ground truth.","section":"Abstract (quality-control pipeline)"}],"minor_comments":[{"comment":"The abstract points to a project page, but the paper itself should include at least a few sample questions and their atomic ground-truth facts in an appendix so that readers can independently judge question quality and scoring.","section":"Abstract (accessibility)"},{"comment":"The term 'wide-context collection' could be defined more precisely; clarify the minimum number of atomic facts per question, how breadth is measured, and how WideSearch differs from existing multi-hop QA or web-agent benchmarks.","section":"Abstract (scope definition)"},{"comment":"The public release is welcome; for archival reproducibility, the full paper should include a stable version identifier or DOI for the dataset and evaluation code, not only a project URL.","section":"Abstract (reproducibility)"}],"recommendation":"uncertain","confidential_remarks":"This review is based solely on the abstract and the provided reader's analysis; the full manuscript text was not available. The central claim is plausible and important, but it is not verifiable from the abstract alone. I recommend that the editor obtain the full manuscript before making a decision. The stress-test concern about answer normalization lands directly on the abstract's unsupported 'objectively verified' claim. If the full paper contains a detailed evaluation protocol and human-machine agreement analysis, my recommendation could change to minor or major revision depending on the specifics; if those details are absent, the manuscript needs substantial revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Before anything else: this is an abstract-only review, so every conclusion below is provisional. The headline claim—most agentic search systems scoring near 0% while humans approach 100% on a 200-question bilingual benchmark—is exactly the kind of result that would matter if it holds. It would say the current agentic search stack is not just imperfect but basically non-functional at wide-context collection. That would be a real call to action.\n\nWhat the paper does well: it identifies a genuinely under-tested capability—collecting many atomic facts across a broad context, as opposed to answering a single query—and builds a benchmark specifically for that. 200 manually curated questions, English and Chinese, 15+ domains, a five-stage QC pipeline, and a public release of data and evaluation code are all the right moves. The human cross-validation is also a sensible external sanity check that the questions are answerable.\n\nThe soft spot is exactly where the stress-test note points: the abstract says facts are 'verified one by one objectively' but says nothing about how that verification handles paraphrase, synonymy, or reordering. If the checker is exact-match or near-exact-match, near-zero agent scores could be an artifact of rigid grading rather than true agent deficiency. Human testers, by contrast, might get informal credit for semantically correct answers, which would inflate the human–agent gap. This is not a claim that the paper is wrong—only that the abstract’s critical numbers are uninterpretable without this detail. There are also no error bars, no per-system breakdowns beyond the best score, and no description of how humans were instructed or scored.\n\nI would not set the stress-test concern as a proven flaw. It is a possible flaw, and a serious one. The public release means the community can check quickly, but a reader shouldn't have to. A referee should ask for the normalization method and an agreement analysis between automatic and human scoring.\n\nBottom line: the benchmark is plausibly useful and the result is striking enough to deserve referee time. It is not a desk-reject. But as an abstract, it overclaims by implying the scores are self-evidently rigorous. For my own work, I would not cite it until the evaluation methodology is visible.","headline":"Striking numbers, but the scoring rubric is doing a lot of unstated work; worth a look, not a verdict.","tokens_in":1393,"tokens_out":1077,"would_cite":false,"duration_ms":13519,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Most state-of-the-art agentic search systems score near zero on WideSearch, a new benchmark for broad information seeking, while humans can reach near 100 percent.","keywords":["agentic search","benchmark","LLM agents","wide-context information seeking","reliability evaluation","question answering","information retrieval"],"falsifier":"If a current agentic search system achieved substantially higher than 5 percent success on a random subset of WideSearch under the same evaluation protocol, or if independent human testers failed to reproduce near-100 percent on a sample of questions, the claim that agents are categorically deficient for wide-context search would be weakened.","tokens_in":671,"feed_emoji":"🔍","tokens_out":1414,"duration_ms":18274,"temperature":0.7,"pith_summary":"WideSearch is a benchmark designed to test how reliably large language model agents collect large amounts of verifiable information across many sources. The paper claims that over ten current agentic search systems, including single-agent, multi-agent, and commercial end-to-end products, almost all score near 0 percent overall success, with the best reaching only 5 percent. In contrast, human testers given enough time can reach near 100 percent. The authors argue this reveals a critical deficiency in today's search agents for large-scale, wide-context information collection, and they provide the benchmark, evaluation pipeline, and results publicly to spur improvement.","feed_headline":"Search agents score near zero on new wide-info benchmark","feed_subtitle":"Best system succeeds only 5% while humans can reach near 100%, exposing a major reliability gap in LLM-based search.","key_machinery":"The central object is the WideSearch benchmark itself: a set of 200 real-user-based questions with atomic, individually verifiable ground-truth facts, plus an objective evaluation pipeline. The benchmark's design is the mechanism that exposes the failure—by requiring complete collection of many small facts rather than a single answer, it transforms information seeking into a reliability test where partial answers cannot pass.","core_discovery":"The paper introduces WideSearch, a benchmark of 200 manually curated questions (100 English, 100 Chinese) drawn from real user queries across more than 15 domains. Each question requires collecting many atomic facts that can be verified one by one and arranged into a structured output. A five-stage quality control pipeline is claimed to ensure difficulty, completeness, and verifiability. Benchmarking over ten state-of-the-art agentic search systems, the paper finds that most achieve overall success rates near 0 percent and the best reaches only 5 percent, while multiple human testers can achieve near 100 percent given sufficient time. The core claim is that current automated agents are far f","pith_inferences":["A likely implication the paper leaves implicit is that the bottleneck is not language understanding or retrieval quality alone but sustained, exhaustive coverage—agents stop too early or miss scattered facts.","The benchmark could be extended to test whether techniques like structured planning, memory, or explicit fact-checking loops improve completeness, and such extensions would directly test the paper's diagnosis.","The large human-versus-agent gap suggests that future evaluation should also measure time-to-completion and cost, since humans reach near 100 percent only with sufficient time."],"forward_implications":["If the benchmark accurately reflects real broad-info-seeking tasks, current LLM-based agents cannot be trusted for large-scale research or planning without human oversight.","The near-zero success rates across diverse agent architectures suggest that scaling compute or adding more agents does not solve completeness and reliability in information collection.","WideSearch provides a concrete, verifiable target for future agent design: a system that can approach human-level completeness on these 200 questions would represent a major advance.","The public release of dataset and evaluation code allows direct comparison and iterative improvement, making agent reliability measurable rather than anecdotal."],"supporting_citations":[],"fun_headline_variants":["WideSearch benchmark: AI search agents fail, best hits 5%","Agentic search agents score near 0% on new WideSearch test","Wide-info seeking: agents top out at 5%, humans near 100%","New benchmark reveals search agents can't handle wide info","WideSearch: agents max 5% success, humans near perfect"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The central assumption is that the 200 questions and their atomic ground-truth facts are complete, fair, and verifiable, so that near-zero agent scores reflect genuine agent deficiency rather than artifacts of question difficulty, ambiguous queries, or incomplete answer keys.","fun_headline_variants_meta":{"raw":{"variants":["WideSearch benchmark: AI search agents fail, best hits 5%","Agentic search agents score near 0% on new WideSearch test","Wide-info seeking: agents top out at 5%, humans near 100%","New benchmark reveals search agents can't handle wide info","WideSearch: agents max 5% success, humans near perfect"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000425,"raw_usage":{"total_tokens":2054,"prompt_tokens":818,"completion_tokens":1236,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":1154}},"tokens_in":562,"tokens_out":1236,"duration_ms":12039,"temperature":1.0,"reasoning_tokens":1154,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T21:41:45.995324+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If a current agentic search system achieved substantially higher than 5 percent success on a random subset of WideSearch under the same evaluation protocol, or if independent human testers failed to reproduce near-100 percent on a sample of questions, the claim that agents are categorically deficient for wide-context search would be weakened.","supporting_citations":[],"review_version":1}