REVIEW 13 cited by
Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
read the original abstract
Agentic search such as Deep Research systems-where agents autonomously browse the web, synthesize information, and return comprehensive citation-backed answers-represents a major shift in how users interact with web-scale information. While promising greater efficiency and cognitive offloading, the growing complexity and open-endedness of agentic search have outpaced existing evaluation benchmarks and methodologies, which largely assume short search horizons and static answers. In this paper, we introduce Mind2Web 2, a benchmark of 130 realistic, high-quality, and long-horizon tasks that require real-time web browsing and extensive information synthesis, constructed with over 1000 hours of human labor. To address the challenge of evaluating time-varying and complex answers, we propose a novel Agent-as-a-Judge framework. Our method constructs task-specific judge agents based on a tree-structured rubric design to automatically assess both answer correctness and source attribution. We conduct a comprehensive evaluation of ten frontier agentic search systems and human performance, along with a detailed error analysis to draw insights for future development. The best-performing system, OpenAI Deep Research, can already achieve 50-70% of human performance while spending half the time, highlighting its great potential. Altogether, Mind2Web 2 provides a rigorous foundation for developing and benchmarking the next generation of agentic search systems.
Forward citations
Cited by 13 Pith papers
-
SciHazard: A Benchmark for Measuring Scientific Safety Risks with Decomposed Harm Scoring
A new scientific-safety benchmark and a decomposed, retrieval-grounded metric that aligns with expert harm judgments substantially better than existing LLM-as-judge baselines.
-
Sequential Behavioral Watermarking for LLM Agents
SeqWM embeds watermarks into history-conditioned action transitions in LLM agent trajectories and verifies them position-agnostically, achieving robust detection under perturbations where prior per-step methods fail.
-
OS-SPEAR: A Toolkit for the Safety, Performance,Efficiency, and Robustness Analysis of OS Agents
OS-SPEAR is a new evaluation toolkit that tests 22 OS agents and identifies trade-offs between efficiency and safety or robustness.
-
DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Histories
The paper reframes image retrieval as agentic exploration over personal visual histories and shows the best tested multimodal agent scores only 28.7 exact match on its new DISBench benchmark.
-
From Simple QA to Deep Research: A Verifiable Benchmark Constructed through Iterative Task Evolution
An automatic pipeline evolves simple QA questions into 500 deep-research tasks with DAG-structured, fact-grounded rubrics that discriminate between models.
-
Persona2Web: Benchmarking Personalized Web Agents for Contextual Reasoning with User History
Persona2Web is a new open-web benchmark where agents must infer a user's preferences from synthetic browsing history to solve intentionally ambiguous queries; current best agents score 13% success.
-
DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generation
DEER uses 7 evaluation dimensions, 101 rubric items, task-specific expert guidance, and unsupported-claim backtracking to score deep-research reports; current systems score lowest on fulfilling expert requests and ana...
-
LocalSearchBench: Benchmarking Agentic Search in Real-World Local Life Services
LocalSearchBench—1.3M merchant records and 900 multi-hop local-life QA tasks across 9 Chinese cities—shows the best reasoning agent reaches only 35.6% correctness.
-
Sidekick: Designing Communication for Effective Multitasking with Computer Use Agents
Sidekick's multimodal feedback significantly improves multitasking performance with computer-use agents compared to chat-only or peripheral text feedback.
-
Agentic Reasoning for Large Language Models
The survey structures agentic reasoning for LLMs into foundational, self-evolving, and collective multi-agent layers while distinguishing in-context orchestration from post-training optimization and reviewing applicat...
-
Rethinking Agentic Reinforcement Learning In Large Language Models
The paper reviews conceptual foundations, methodological innovations, effective designs, critical challenges, and future directions for LLM-based Agentic Reinforcement Learning.
-
Rethinking Agentic Reinforcement Learning In Large Language Models
The paper surveys the conceptual foundations, methodological innovations, challenges, and future directions of agentic reinforcement learning frameworks that embed cognitive capabilities like meta-reasoning and self-r...
-
Rethinking Agentic Reinforcement Learning In Large Language Models
This review synthesizes conceptual foundations, methods, challenges, and future directions for agentic reinforcement learning in large language models.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.