{"id":"6ed0873f-13a0-45cc-a038-5371468d4902","arxiv_id":"2506.21506","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Mind2Web 2 is a 130-task benchmark for long-horizon agentic search with a tree-structured automated judge, showing OpenAI Deep Research reaches about 50 to 70 percent of human performance.","lead":"Mind2Web 2 introduces 130 long-horizon, time-varying web search tasks and an automated Agent-as-a-Judge evaluator that checks both answer correctness and source citations. Tests of ten AI search systems against humans find the best system, OpenAI Deep Research, reaches roughly 50 to 70 percent of human performance in about half the time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Judge-reliability claim rests on a single-evaluator audit of 15/120 tasks; pooled node-level 99% accuracy can mask whole-task script failures, leaving downstream rankings not yet established.","rationale":"The paper's contribution is only as strong as its automatic judge. I read the benchmark construction, rubric formalism, and validation pipeline in good faith; the design is thoughtful and the two-stage script validation plus human audit is more than most benchmarks provide. My concern is not that the judge is demonstrably unreliable, but that the headline 99% figure is computed at the wrong granularity and on too narrow a sample to support generalization to all 120 private-test tasks. A single buggy script can produce many wrong leaf judgments; pooling 720 nodes across 15 tasks hides task-level failures, and the audit excludes trivial total-failure cases and uses one evaluator from the same group with developer-involved adjudication. The mixed-set human comparison is a real but secondary reporting issue: it can be fixed by recomputing agent scores on Subset-30. The judge-reliability issue is load-bearing because system rankings, error analysis, and the human-relative headline all inherit from it. The proposed external, blinded audit on 30+ tasks with per-task agreement would settle this directly. The reader's CONDITIONAL verdict already captures this concern, so my read does not change the verdict.","tokens_in":31208,"tokens_out":6593,"duration_ms":82368,"concrete_test":"Conduct a blinded external audit on a fresh sample of at least 30 private-test tasks not used in judge-agent development, with two independent evaluators who have no affiliation with the authors. For each task, compare judge-agent leaf-node decisions and final root scores against the evaluators on two non-trivial answers per task. Report per-task agreement and a confidence interval for task-level reliability, not pooled node accuracy, and explicitly list any final-score flips between success and failure. If the per-task agreement is materially below the claimed 99% or the confidence interval includes, say, 90%, the paper should soften the reliability claim and re-report rankings under a lower-bound correction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Agent-as-a-Judge makes evaluation rigorous and scalable across all 120 private-test tasks. The only direct evidence for this is the human audit in §4.4 and Appendix D.5: one evaluator, 15 sampled tasks, and 720 leaf-node verifications. The reported 99.03% correctness is computed at the node level, but node judgments are nested inside per-task judge-agent scripts. A single flawed script can corrupt many leaf judgments for one task, and those errors are diluted when pooled with hundreds of correct nodes from other tasks. With only 15 tasks, the task-level confidence interval is wide, and the sample explicitly excludes trivial total-failure answers, so it is not representative of the full answer distribution. The evaluator comes from the same research group, and discrepancy adjudication involves the judge-agent developers, limiting independence. The Limitations appendix acknowledges reliance on LLM judgments but does not address the narrowness of this reliability audit. If even a few of the 105 unaudited task scripts fail on non-trivial answers, system rankings, the error analysis, and the 50–70% human-relative headline all inherit that error. This is a load-bearing, testable assumption rather than a stylistic caveat.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Mind2Web 2, a benchmark of 130 long-horizon, time-varying web-search tasks with citation-backed answers, together with Agent-as-a-Judge, a rubric-tree evaluation framework implemented as task-specific judge agents. The authors report rubric complexity statistics, a human performance study on a randomly selected 30-task subset, evaluation of ten frontier agentic search systems and humans, an error analysis, and a human audit of judge agents on 15 tasks reporting 99.03% leaf-node correctness. The headline result is that OpenAI Deep Research achieves 50-70% of human performance while spending about half the time.","tokens_in":31362,"tokens_out":5895,"duration_ms":64070,"significance":"If the benchmark and judge pipeline hold up, this is a valuable community resource: it addresses a real evaluation gap for long-horizon agentic search, includes realistic live-web tasks, provides a public/private split, releases code and evaluation scripts, and offers a useful error-analysis taxonomy. The human-relative comparison is attention-worthy and timely. However, the evidence for the two strongest claims—the reliability of the judge agents and the 50-70% human-relative number—is currently thinner than the abstract suggests, so the paper needs strengthening before these claims can be taken at face value.","major_comments":[{"comment":"The abstract's '50-70% of human performance' is not directly supported by Table 3. Human Partial Completion (0.79) and Success Rate (0.54) are computed on Subset-30 only, whereas agent scores are reported for all 120 private-test tasks; the paper does not report agent scores restricted to Subset-30. Unless the authors show agent performance on the same 30 tasks or argue convincingly that Subset-30 is representative, the quoted ratio is an apples-to-oranges comparison. Please add a Subset-30 column/row for all systems or revise the claim.","section":"§4.2, Table 3"},{"comment":"The 99.03% verifier correctness is derived from 15 sampled tasks and 720 leaf-node checks scored by a single human evaluator, with trivial total-failure answers explicitly excluded. Because errors are nested within task-specific judge-agent scripts, pooled node-level accuracy can hide whole-task script failures: one flawed script can corrupt many leaf judgments for a task, and those errors are diluted when combined with hundreds of correct nodes from other tasks. The audit also relies on a human evaluator who participated in error analysis and on adjudication by a judge-agent developer, so it does not provide fully independent confirmation. Please report per-task error rates and a task-level confidence interval, justify the exclusion of total-failure answers, and clarify the independence of the evaluator and adjudication protocol.","section":"§4.4, Appendix D.5"}],"minor_comments":[{"comment":"The WebWalkerQA row cites [41], which is AssistantBench; the correct reference appears to be [37]. Please fix this citation.","section":"Table 1"},{"comment":"In the Extractor and Verifier prompts, 'http//' and 'https//' should be 'http://' and 'https://'.","section":"Appendix D.2"},{"comment":"Appendix E.1 states that Hugging Face Open Deep Research was run with OpenAI's o3 model as its base model; this important detail is absent from §4.1 and Table 3, and should be disclosed wherever the system is described as open-source.","section":"Appendix E.1"},{"comment":"The category 'Explicitly Time-varying Tasks' is defined only in the text (57 tasks); add the definition to the caption and consider showing error bars, since Figure 3 includes them.","section":"Figure 4"},{"comment":"The rubric formula's 'otherwise' branch (s(v)=1 when there are no non-critical children) could be misread as applying to leaf nodes; a one-sentence clarification that leaves are assigned by the Verifier would help.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and likely impactful, but the headline claims need stronger evidence. If the authors can supply agent scores on Subset-30 and a more robust judge audit, I would support acceptance. I would not reject on novelty grounds."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a genuine step forward: it is the first benchmark I know of that combines long-horizon, time-varying tasks with citation-backed agentic search evaluation, and the tree-structured rubric / judge-agent pipeline is a real methodological contribution, not a repackaging. The task collection effort (1000+ hours, multiple validation layers), the human performance study on Subset-30, and the detailed error analysis are all solid. The authors also release code and plan a leaderboard, which makes the work reproducible in the ways that matter.\n\nThe soft spots are real but not disqualifying. The judge reliability claim rests on a narrow audit: one evaluator, 15 of 120 private tasks, and a node-level 99% correctness that pools hundreds of leaf judgments. Node-level pooling can hide whole-task script failures, and the audit explicitly excluded trivial total-failure answers. A larger, more independent judge audit would substantially strengthen the paper. Similarly, the headline \"50–70% of human performance\" compares agents scored on all 120 tasks with humans scored on a 30-task subset; the paper does not report agent scores restricted to that same subset. That is easily fixed, but it should be fixed before the headline is taken at face value. The authors acknowledge some of these concerns in the Limitations appendix, but not the narrowness of the judge audit.\n\nI do not see a load-bearing flaw in the central methodology. The rubric design is coherent, the validation process is described in unusual detail, and the human study—while small—is a genuine attempt at grounding. The citation pattern looks appropriate; the comparisons to BrowseComp, GAIA, and PaperBench are accurate, and the paper positions itself fairly as a complement to these, not a replacement.\n\nWho this is for: anyone building or evaluating deep research systems, and anyone working on automated evaluation of long-form, attribution-dependent answers. It is a benchmark paper, so its value will be proven by adoption, but it is the most credible attempt I have seen in this space.\n\nMy recommendation: send it to peer review. A serious referee should ask for matched-subset agent scores and a more extensive judge audit, but these are revision-level issues, not grounds for rejection. I would cite this in my own work.","headline":"Genuinely new benchmark and judge framework with solid task quality; headline comparisons and judge-audit narrowness need revision before the numbers are taken at face value.","tokens_in":32048,"tokens_out":2362,"would_cite":true,"duration_ms":25965,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Mind2Web 2 claims to be the first benchmark for long-horizon, time-varying agentic search, and its Agent-as-a-Judge pipeline evaluates complex citation-backed answers with about 99 percent verifier correctness.","keywords":["agentic search","deep research systems","benchmark construction","Agent-as-a-Judge","tree-structured rubric","citation attribution","LLM-as-a-Judge","long-horizon web tasks"],"falsifier":"Have two independent evaluators from outside the benchmark team re-score leaf nodes on a stratified sample of all 120 private-test tasks, or swap the Verifier's underlying model and compare leaf-level agreement; if agreement falls materially below the reported 99 percent, or if agent scores restricted to Subset-30 differ from full-set scores, the central reliability and comparability claims are weakened.","tokens_in":30960,"feed_emoji":"🔍","tokens_out":7538,"duration_ms":71856,"temperature":0.7,"pith_summary":"Mind2Web 2 is a benchmark of 130 realistic, long-horizon web-search tasks whose answers are not fixed strings but change with the live web, and it argues that such tasks can be evaluated automatically and reliably. The paper introduces Agent-as-a-Judge: each task gets a tree-structured rubric whose leaf nodes are binary checks, implemented as a task-specific judge agent that extracts claims from an answer and verifies them against cited webpages. The authors report that this pipeline reaches about 99 percent agreement with a human evaluator on sampled leaf-node judgments. If the benchmark and judge hold up, the field gains a way to compare deep-research systems and humans on open-ended, citation-backed information work rather than on single-answer trivia.","feed_headline":"New benchmark scores agentic search on live, time-varying web tasks","feed_subtitle":"Judge agents verify answers against cited sources at 99 percent; top agent hits 50-70 percent of human scores.","key_machinery":"The load-bearing object is the rubric tree: a goal is decomposed into leaf-node binary verifications, and scores are aggregated bottom-up. Critical nodes act as gates (failure at any critical child zeroes the parent), non-critical children are averaged for partial credit, and sequential nodes short-circuit later checks after an earlier failure. This yields two metrics—Partial Completion (mean root score) and Success Rate (fraction of tasks scoring 1). Around this sits the judge agent, an agentic Python workflow per task with an Extractor that pulls structured claims out of the answer and a Verifier that checks each claim against cached webpage text and screenshots, using LLM calls for simple checking and URL-based verification.","core_discovery":"The paper claims that the central obstacle—evaluating complex, time-varying, citation-backed answers—can be removed by exploiting the generation-verification asymmetry: even if two agents return very different texts, the requirements of each task are known in advance, so a rubric can specify what counts as correct. Each rubric is a tree of dozens to hundreds of binary leaf checks that verify both correctness (does the answer satisfy every stated criterion?) and attribution (is each claim actually backed by its cited URL?). The judge agent runs this tree with LLM-based extractors and verifiers. On a human evaluation of 15 tasks, the verifier's leaf judgments matched human judgment at 99 percent after accounting for human annotation errors. The paper also reports that the best-performing Deep Research system reaches 50–70 percent of human Partial Completion while taking less than half the average human time.","pith_inferences":["A testable extension is to apply the rubric-tree judge design to other long-form, time-varying agent outputs, such as travel planning or procurement, whenever the criteria decompose into independent single-page verifications.","The reported 99 percent correctness may not transfer to the hardest private-test items, since collapsed webpage content already caused four of the eight remaining verifier discrepancies; measuring judge accuracy on those cases specifically would sharpen the claim.","The observed 50-70 percent human performance at half the time suggests that, as inference budgets grow, deep-research systems could overtake humans on tedious but constrained information-gathering tasks—while criterion violations by human participants show the human reference itself is noisy.","If the private-test rubrics and the judge-script generation pipeline stay hidden, leaderboard integrity depends on that secrecy, since a system optimized against this exact judge could inflate scores without improving real search ability."],"forward_implications":["If the benchmark holds, agentic search systems can be compared on tasks requiring dozens to hundreds of web actions and multi-source synthesis, not just single-answer lookups.","Because answers are time-varying, future benchmarks no longer need to freeze answers or restrict queries to static factoids to stay automatically evaluable.","The two metrics give developers a partial-credit signal for incremental progress, not only an all-or-nothing success flag.","The reported 50-70 percent of human performance at half the time suggests deep-research systems may already be cost-effective for some tedious information-gathering work, while the error analysis shows hallucinated or invalid citations remain the main barrier."],"supporting_citations":[{"why":"The nearest alternative evaluation methodology it contrasts with: same generation-verification asymmetry but restricted to static single-answer tasks.","marker":"[36]"},{"why":"Concurrent work that also uses tree-structured rubrics; Mind2Web 2 extends this with largely automated rubric generation and richer score aggregation.","marker":"[35]"},{"why":"The LLM-as-a-Judge baseline whose single-pass judgments the tree-structured Agent-as-a-Judge design is meant to surpass.","marker":"[45]"},{"why":"Source of the comparison point that automated evaluation of simpler web tasks typically falls below 90 percent correctness.","marker":"[39]"},{"why":"Live-web benchmark whose maintenance concerns motivate the answer-caching strategy and time-varying design choices.","marker":"[29]"},{"why":"Prior web-agent benchmark whose task domain guidance and collection pipeline inform the task-design principles.","marker":"[8]"},{"why":"System card for the best-performing Deep Research system evaluated, grounding the headline performance comparison.","marker":"[26]"}],"fun_headline_variants":["Agent-as-a-Judge: 99% human agreement on web answers","Benchmark with tree-based judges matches humans 99%","Long-horizon web tasks: agent judges score 50-70% of human","Best agent: 70% of human score in half the time","Mind2Web 2: benchmark for time-varying, citation-backed answers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire comparison rests on the judge agents' 99 percent correctness generalizing from 15 sampled tasks with one human evaluator from the same research group to all 120 private-test tasks, and on the 30-task human-tested subset representing the full test set.","fun_headline_variants_meta":{"raw":{"variants":["Agent-as-a-Judge: 99% human agreement on web answers","Benchmark with tree-based judges matches humans 99%","Long-horizon web tasks: agent judges score 50-70% of human","Best agent: 70% of human score in half the time","Mind2Web 2: benchmark for time-varying, citation-backed answers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000857,"raw_usage":{"total_tokens":3726,"prompt_tokens":955,"completion_tokens":2771,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":2676}},"tokens_in":571,"tokens_out":2771,"duration_ms":22963,"temperature":1.0,"reasoning_tokens":2676,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:23:02.419050+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two independent evaluators from outside the benchmark team re-score leaf nodes on a stratified sample of all 120 private-test tasks, or swap the Verifier's underlying model and compare leaf-level agreement; if agreement falls materially below the reported 99 percent, or if agent scores restricted to Subset-30 differ from full-set scores, the central reliability and comparability claims are weakened.","supporting_citations":[{"cited_title":"Deep research system card","cited_arxiv_id":null,"evidence_quote":"System card for the best-performing Deep Research system evaluated, grounding the headline performance comparison."}],"review_version":1}