Pith. sign in

REVIEW 3 major objections 5 minor 52 references

A fully automatic pipeline turns simple questions into 500 deep-research tasks with rubrics grounded in web evidence.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

An automatic pipeline evolves simple QA questions into 500 deep-research tasks with DAG-structured, fact-grounded rubrics that discriminate between models.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection The DAG-evolving task pipeline is a real step forward for automatic deep-research benchmarks, but the gold content derives from a single model's exploration, so the pointwise scores measure coverage of that trajectory, not absolute research quality. the 3 major comments →

arxiv 2608.02163 v1 pith:KOKWEBBJ submitted 2026-08-03 cs.AI

From Simple QA to Deep Research: A Verifiable Benchmark Constructed through Iterative Task Evolution

Can Wang , Haoran Chen , Haowen Gao , Hao Ding , Zhaoyang Liu , Zhiying Tu This is my paper
classification cs.AI
keywords deep researchbenchmark constructionautomatic evaluationrubricsDAGtask evolutioncheckpointsLLM agents
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that deep-research benchmarks — tasks that require multi-source search, integration, and open-ended answers — can be built fully automatically, without expert-authored questions or rubrics. It introduces a pipeline that starts from simple factual queries and iteratively evolves them into hard research tasks, representing each task as a directed acyclic graph of atomic steps. Every step carries checkpoints grounded in evidence gathered during exploration, and these checkpoints become the scoring rubric. The claim is that this produces 500 tasks with rubrics that give fine-grained, absolute, human-aligned scores, not just relative rankings. If right, it removes the main bottleneck to scalable deep-research evaluation: dependable pointwise scoring without human authoring.

Core claim

On the paper's own terms, the central discovery is that the task-specific knowledge needed to evaluate a deep-research answer — what evidence and reasoning a good response must include — can be revealed incrementally by an agent exploring the web, and that this same exploration can be converted into a verifiable scoring rubric. An Explorer solves a simple query and records a trajectory; a Formalizer distills the trajectory into a minimal DAG of evidential and analytical steps with independently checkable checkpoints; a Challenger uses explored-but-unused clues to pose a harder next query. Repeating this loop grows a simple question into a professional-level research task while the query, DAG

What carries the argument

The load-bearing mechanism is the task graph: a DAG whose nodes are atomic research steps (evidential nodes retrieve traceable facts; analytical nodes reason over them), and whose checkpoints are concrete, independently verifiable claims derived from the Explorer's trajectory. The Formalizer builds and maintains this DAG, and the rubric set is exactly one rubric per node, with weights propagated from leaves to parents so that steps supporting more downstream conclusions count more. Evaluation scores a response by whether its content satisfies the checkpoints under these weights — without requiring the response to reproduce the graph's solving path. The iterative loop (Explorer → Formalizer →

Load-bearing premise

The load-bearing premise is that one language model's web exploration is a sufficient and reliable oracle for the task-specific knowledge a deep-research answer must cover, so checkpoints derived from that single trajectory are treated as gold.

What would settle it

Have independent domain experts write gold answers for a random subset of the 500 tasks, then check whether every expert-identified essential point corresponds to a checkpoint in the automatic rubric and whether the checkpoints themselves are factually accurate against authoritative sources. If sizable gaps or errors appear in that sample, the rubric-based scores do not measure deep-research quality.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Benchmark construction in new domains can be automated: given any seed corpus of simple questions, the pipeline yields research tasks with traceable rubrics in roughly 14 rounds and about 265 tool calls per task.
  • Scores from this benchmark are absolute pointwise measures, not relative rankings; a model's score does not depend on which other models are in the comparison.
  • Distinguishing query forms (with hints, without hints, assigned-topic) isolates separate deep-research abilities, so failures can be attributed to retrieval, planning, or argument formation.
  • Because checkpoints are fact-grounded, the same rubric can be re-applied by different judges (LLM or human) with high agreement, enabling stable longitudinal comparisons.
  • If the pipeline works, it could be extended to professional domains, turning sparse long-tail knowledge into verifiable research tasks without expert labor.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's own limitation about construction cost (about 168 hours and $10,000) suggests a practical ceiling: scaling to thousands of tasks is expensive, although the cost is paid per benchmark, not per evaluation.
  • Checkpoint-based scoring likely rewards coverage of the constructor's chosen evidence and may undervalue a response that reaches correct conclusions through novel sources the Explorer never visited; a test would be whether accepted alternative answers score well.
  • The stability of scores across judges is reported for 100 tasks; checking all 500 and testing against independent expert rubrics would reveal whether the automatic rubrics generalize beyond their origin trajectory.
  • Because the Challenger can only use clues already in the trajectory, the benchmark's difficulty is bounded by the Explorer's search behavior; a stronger Explorer would produce harder tasks, so the method couples benchmark quality to the constructor model's competence.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces a fully automatic pipeline (Explorer–Formalizer–Challenger) that evolves simple seed questions into 500 deep-research tasks, each represented as a DAG of atomic evidential/analytical steps with associated checkpoints and pointwise rubrics. The authors report three query forms (hinted, without hints, assigned topic), human review of 100 tasks, evaluations of ten models, and comparisons with existing benchmarks. The central claim is that automatically constructed, source-grounded, pointwise rubrics can substitute for expert-authored rubrics and provide absolute, human-aligned, stable evaluation of open-ended deep-research responses.

Significance. If the construction method is valid, this is a substantial contribution: it removes the expert-authoring bottleneck for deep-research benchmarks, provides traceable DAG-structured rubrics, and ships a public 500-task benchmark with code and data. The human-review evidence (Fleiss κ=0.81, no poor ratings on 100 tasks) and judge–human correlations (0.88–0.92 across judges) are meaningful external validity checks. The three-query-form design is also a useful diagnostic contribution. The main risk is that the gold content is generated from a single LLM's exploration trajectory, which may encode that model's search blind spots; the paper's claims of 'absolute' pointwise quality scores depend on resolving this concern.

major comments (3)
  1. [Construction Pipeline, Formalizer and Stopping Criterion (Eq. 1)] The central validity claim rests on the rubric covering what a deep-research answer must include. The paper's current evidence addresses verifiability of checkpoints, not completeness of coverage.
  2. [Benchmark Evaluation, Table 4 and Table 6] Load-bearing? It affects the evaluation-stability claim but the human correlations are strong. Still, the overlap between judge and tested models is a concrete, addressable confound.
  3. [Construction Analysis, Figure 5] This is a secondary but still load-bearing point for the paper's claim that the pipeline produces controlled difficulty escalation.
minor comments (5)
  1. [Table 5] The table formatting is garbled in the arXiv text (e.g., 'Pearsonr–0.73 0.6024–0.79/0.76 0.899'). Please replace with a clean table with separate rows for each benchmark and metric, and define all abbreviations in the caption.
  2. [Human Review (Table 3)] The text says 'all 10,695 checkpoints that encode task knowledge were manually checked as reliable,' but Table 2 reports a mean of 104.85 checkpoints per task; 100 tasks would yield about 10,485 checkpoints. Please reconcile the number or clarify whether the 10,695 figure includes additional per-task metadata.
  3. [Hyperparameters (Appendix)] The reviser is described as using 'at most 100,000 output tokens' per construction step. This seems excessively high for the prompts shown and likely a typo; please confirm the intended token limit.
  4. [Benchmark Overview] Top-500 selection by DAG node count is a design choice that biases the benchmark toward broader tasks; please state explicitly in the main text that the benchmark is not a representative sample of seed queries, but a difficulty-filtered subset.
  5. [Notation] In Eq. (1), the notation G_t = G_{t-1} = G_{t-2} is used; please clarify that equality is on both the node set and edge set, as stated in the prose, and note whether node IDs must match exactly across rounds.

Circularity Check

0 steps flagged

No significant circularity: the benchmark construction is an empirical data-generation pipeline with external human validation, not a self-referential derivation.

full rationale

The paper's derivation chain is a construction pipeline, not a fitted model whose outputs are relabeled as predictions. The Explorer produces web trajectories, the Formalizer converts them into DAGs and rubrics, and the Challenger evolves the query; the stopping rule in Eq. (1) is a graph-stability convergence criterion, not an identity that forces the claimed result. The rubrics are not fitted to the test-model scores they later measure, and the benchmark's validity rests partly on independent external evidence: 182 hours of human review of 100 sampled tasks (Table 3), human-machine consistency comparisons against other benchmarks (Table 5), and senior-researcher judgments of query quality, DAG correctness, rubric coverage, checkpoint verifiability, and weight reasonableness. The acknowledged limitation in the Appendix ("The current benchmark score measures coverage and correctness of task-specific evidential and analytical checkpoints. It does not separately measure citation count, citation rate, citation format...") is a scope restriction, not a circular step. The deeper epistemic concern that gold checkpoints originate from a single GPT-5.5 trajectory is a real coverage-completeness and validity risk, but it is not a circularity in the derivation: the paper does not define task quality as "whatever the constructor model found" and then derive that the constructor model finds it. No load-bearing step reduces to a self-citation, uniqueness theorem, or fitted parameter. The relative difficulty evidence in Figure 5 is an empirical measurement using the constructed rubrics, not a consequence of an equation equating the predicted quantity with its input. The paper is therefore best assessed as having no significant circularity, with its main risks lying in benchmark-validity assumptions rather than in circular reasoning.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

There are no fitted physical parameters; the hand-chosen construction hyperparameters above shape the benchmark and should be reported as design choices. The main unexamined postulate is that LLM-produced trajectories are a reliable gold source: every checkpoint, and therefore every rubric, traces back to one model's exploration.

free parameters (3)
  • Stopping criterion stability window = 2 unchanged rounds; T = min{t>=3 : G_t=G_{t-1}=G_{t-2}}
    Hand-chosen convergence rule; directly controls task depth and breadth at convergence.
  • Top-500 DAG-node retention count = 500 of 963 converged seeds
    Tasks with the most DAG nodes are kept 'to ensure sufficient difficulty'; this selection shapes the final benchmark composition.
  • Maximum analytical chain length = 5 consecutive analytical nodes
    Failure criterion in the appendix; prunes tasks with overly long single reasoning chains, influencing the style of retained tasks.
axioms (5)
  • domain assumption Deep research solving processes can be represented as a DAG of atomic evidential/analytical steps.
    Task Representation section states this as the foundation; if some valid solution paths cannot be decomposed into such steps, rubrics will miss or misweigh them.
  • domain assumption The knowledge needed for evaluation is progressively revealed as an agent expands its exploration scope.
    Introduction's key insight; the entire Explorer-Formalizer-Challenger loop assumes exploration reveals all relevant evidence.
  • ad hoc to paper GPT-5.5 trajectories are factually reliable enough to serve as gold checkpoints.
    Explorer trajectories are the sole source of node content and checkpoints; no independent verification except the 100-task human review.
  • domain assumption Challenger expansion using unused trajectory information keeps queries coherent and solvable.
    Asserted in Construction Pipeline; Figure 5 and Table 3 give indirect empirical support but no proof.
  • domain assumption Human review of 100 random tasks generalizes to the full 500-task benchmark.
    Quality claims rest on a 20% sample; no audit is reported for the remaining 400 tasks.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of From Simple QA to Deep Research: A Verifiable Benchmark Constructed through Iterative Task Evolution." pith.science (2026). https://pith.science/paper/KOKWEBBJ

@misc{pith2026260802163,
  author       = {Pith},
  title        = {Pith review of: From Simple QA to Deep Research: A Verifiable Benchmark Constructed through Iterative Task Evolution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KOKWEBBJ}},
  note         = {Machine review of arXiv:2608.02163}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Deep research benchmarks require expert-level tasks and reliable evaluation grounded in task-specific knowledge. Existing benchmarks rely heavily on expert authoring or pre-existing human-authored materials, while fully automatic construction struggles to ensure consistent and traceable verification. To address this gap, we introduce a verifiable benchmark of 500 deep research tasks spanning 31 topics and 10 major categories, with three query forms designed to probe complementary capabilities required for deep research. The benchmark is constructed automatically using an iterative Explorer-Formalizer-Challenger pipeline that progressively transforms simple questions into deep research tasks. Each task is represented as a directed acyclic graph (DAG) of atomic steps and associated checkpoints, enabling the query, DAG, and rubrics to evolve together in a controlled manner. Experiments demonstrate that the benchmark clearly discriminates among models and query types, while its fact-grounded pointwise rubrics enable fine-grained, human-aligned, and stable evaluation. Our data, implementation, and results are publicly available.

Figures

Figures reproduced from arXiv: 2608.02163 by Can Wang, Hao Ding, Haoran Chen, Haowen Gao, Zhaoyang Liu, Zhiying Tu.

Figure 1
Figure 1. Figure 1: Round 1 turns the simple query q1 into a problem that requires one more step to solve. Round 2 further evolves it into a query with multiple answering aspects. Through further rounds up to Round T, the process yields a substan￾tially harder deep research task qT that requires multi-source search and integrative analysis. Yet the reliability and professional quality of the generated rubrics are not guarante… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of our benchmark construction pipeline. The Explorer, Formalizer, and Challenger loop evolves a simple [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Topic distribution of tasks in the benchmark. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Score distributions of the strongest and weakest models on [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Performance across evolution rounds. shows a clear cliff on q o T , whereas q a T does not produce a comparable drop, suggesting that its weakness lies in uncov￾ering the hidden goal of a query without hints, rather than in conducting research once the topic is clear. Notably, removing tools causes large drops for all ten models on all query types: overall scores fall by 0.14 on av￾erage, with mean drops o… view at source ↗
Figure 6
Figure 6. Figure 6: Final task DAG for nq open seed 0395. Round t Depth Nodes Main evolution focus 1 2 4 Identify the Twenty-Sixth Amendment, its text, and basic passage and ratification dates. 2 3 5 Reconstruct the formal proposing measure and distinguish ratification from certification. 3 7 10 Add S.J. Res. 7, the statutory background, Oregon v. Mitchell, and the need for an amendment. 4 7 13 Add House and Senate paths, sta… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

52 extracted references · 25 linked inside Pith

  1. [1]

    Alibaba Cloud Community. 2026. Qwen3.7: The Agent Frontier

    2026

  2. [2]

    Anthropic. 2026. Introducing Claude Sonnet 5

    2026

  3. [3]

    DeepSeek-AI. 2026. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv:2606.19348

    arXiv 2026

  4. [4]

    Du, M.; Xu, B.; Zhu, C.; Wang, X.; and Mao, Z. 2025. DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents. arXiv:2506.11763

    Pith/arXiv arXiv 2025

  5. [5]

    Gao, Y .; Zhao, R.; Deng, Y .; and Zhang, W. 2026. DR- Arena: an Automated Evaluation Framework for Deep Research Agents. arXiv:2601.10504

    Pith/arXiv arXiv 2026

  6. [6]

    Google DeepMind. 2026. Gemini 3.5 Flash Model Card

    2026

  7. [7]

    J.; Shu, Y .; Song, C

    Gou, B.; Huang, Z.; Ning, Y .; Gu, Y .; Lin, M.; Qi, W.; Kopanev, A.; Yu, B.; Guti ´errez, B. J.; Shu, Y .; Song, C. H.; Wu, J.; Chen, S.; Moussa, H. N.; Zhang, T.; Xie, J.; Li, Y .; Xue, T.; Liao, Z.; Zhang, K.; Zheng, B.; Cai, Z.; Rozgic, V .; Ziyadi, M.; Sun, H.; and Su, Y . 2025. Mind2Web 2: Evaluating Agentic Search with Agent- as-a-Judge. arXiv:2506.21506

    Pith/arXiv arXiv 2025

  8. [8]

    Guo, Z.; Xu, F.; Li, Y .; Li, M.; Zou, S.; Wu, J.; Shi, H.; Bai, H.; Leung, H.-f.; and King, I. 2025. ADRA-Bank: A Modular Benchmark for Academic Deep Research Agents. arXiv:2512.00986

    Pith/arXiv arXiv 2025

  9. [9]

    H.; Song, H.; Choi, S

    Han, J.; Kim, H.; Lee, C.; Lee, D.; Park, M. H.; Song, H.; Choi, S. J.; Lee, M.; and Lee, H. 2025. DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generation. arXiv:2512.17776

    Pith/arXiv arXiv 2025

  10. [10]

    Hu, C.; Du, H.; Wang, H.; Lin, L.; Chen, M.; Liu, P.; Miao, R.; Yue, T.; You, W.; Ji, W.; Yuan, W.; Deng, W.; Yuan, X.; Zhang, X.; Liu, X.; Liu, X.; Xu, Y .; Cao, Y .; Zhang, Y .; Wang, Y .; Shu, Y .; Zhang, Y .; Zhang, Y .; Gong, Z.; Chang, Z.; Li, B.; Ma, D.; Jia, F.; Wang, H.; Liu, J.; Bai, J.; Liu, J.; Liu, M.; Wang, N.; Wu, Q.; Du, Q.; Li, S.; Sun, W...

    arXiv 2025

  11. [11]

    Huang, P.; Zhong, Z.; Wan, Z.; Zhou, D.; Alam, S.; Wang, X.; Li, Z.; Dou, Z.; Zhu, L.; Xiong, J.; Tao, C.; Xu, Y .; Dimitriadis, D.; Zhang, T.; and Zhang, M. 2026. MMDeepResearch-Bench: A Benchmark for Multimodal Deep Research Agents. arXiv:2601.12346

    arXiv 2026

  12. [12]

    Java, A.; Khandelwal, A.; Midigeshi, S.; Halfaker, A.; Deshpande, A.; Goyal, N.; Gupta, A.; Natara- jan, N.; and Sharma, A. 2025. Characterizing Deep Research: A Benchmark and Formal Definition. arXiv:2508.04183

    Pith/arXiv arXiv 2025

  13. [13]

    Jin, J.; Yan, Z.; Wang, S.; Jiao, W.; and Lu, Y

  14. [14]

    Kausik, C.; Swaminathan, A.; and Kallus, N. 2026. The Context Gathering Decision Process: A POMDP Framework for Agentic Search. arXiv:2605.07042

    Pith/arXiv arXiv 2026

  15. [15]

    Lee, K.; Chang, M.-W.; and Toutanova, K. 2019. La- tent Retrieval for Weakly Supervised Open Domain Question Answering

    2019

  16. [16]

    Li, K.; Zhang, Z.; Yin, H.; Zhang, L.; Ou, L.; Wu, J.; Yin, W.; Li, B.; Tao, Z.; Wang, X.; Shen, W.; Zhang, J.; Zhang, D.; Wu, X.; Jiang, Y .; Yan, M.; Xie, P.; Huang, F.; and Zhou, J. 2025. WebSailor: Navigating Super- human Reasoning for Web Agent. arXiv:2507.02592

    Pith/arXiv arXiv 2025

  17. [17]

    Li, M.; Zeng, Y .; Cheng, Z.; Ma, C.; and Jia, K. 2025. ReportBench: Evaluating Deep Research Agents via Academic Survey Tasks. arXiv:2508.15804

    Pith/arXiv arXiv 2025

  18. [18]

    Li, R.; Du, M.; Xu, B.; Zhu, C.; Wang, X.; and Mao, Z. 2026. DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Re- port. arXiv:2601.08536

    arXiv 2026

  19. [19]

    Mialon, G.; Fourrier, C.; Swift, C.; Wolf, T.; LeCun, Y .; and Scialom, T. 2023. GAIA: A Benchmark for General AI Assistants. arXiv:2311.12983

    Pith/arXiv arXiv 2023

  20. [20]

    MindDR Team; and Li Auto Inc. 2026. Mind DeepRe- search Technical Report. arXiv:2604.14518

    Pith/arXiv arXiv 2026

  21. [21]

    MiniMax. 2026. MiniMax M2.5: Built for Real-World Productivity

    2026

  22. [22]

    OpenAI. 2026. Previewing GPT-5.6 Sol: a next- generation model. Announces the GPT-5.6 family in- cluding Sol, Terra, and Luna

    2026

  23. [23]

    Qwen Team. 2025. Qwen3 Technical Report. arXiv:2505.09388

    Pith/arXiv arXiv 2025

  24. [24]

    QwenCloud. 2026. qwen3.6-flash. Released 2026-04- 16 on the QwenCloud model changelog

    2026

  25. [25]

    Sharma, M.; Zhang, C. B. C.; Bandi, C.; Wang, C.; Aich, A.; Nghiem, H.; Rabbani, T.; Htet, Y .; Jang, B.; Basu, S.; Balwani, A.; Peskoff, D.; Ayestaran, M.; Hendryx, S. M.; Kenstler, B.; and Liu, B

  26. [26]

    Sun, R.; Bai, Z.; Zhang, W.; Zhang, Y .; Zhao, L.; Sun, S.; and Qiu, Z. 2025. FinResearchBench: A Logic Tree based Agent-as-a-Judge Evaluation Framework for Fi- nancial Research Agents. arXiv:2507.16248

    arXiv 2025

  27. [27]

    Tao, Z.; Wu, J.; Yin, W.; Zhang, J.; Li, B.; Shen, H.; Li, K.; Zhang, L.; Wang, X.; Jiang, Y .; Xie, P.; Huang, F.; and Zhou, J. 2025. WebShaper: Agentically Data Synthesizing via Information-Seeking Formalization. arXiv:2507.15061

    Pith/arXiv arXiv 2025

  28. [28]

    Tian, S.; Zhang, Z.; Chen, L.; and Liu, Z. 2024. MMInA: Benchmarking Multihop Multimodal Inter- net Agents. arXiv:2404.09992

    Pith/arXiv arXiv 2024

  29. [29]

    Wan, H.; Yang, C.; Yu, J.; Tu, M.; Lu, J.; Yu, D.; Cao, J.; Gao, B.; Xie, J.; Wang, A.; Zhang, W.; Torr, P.; and Zhou, D. 2025. DeepResearch Arena: The First Exam of LLMs’ Research Abilities via Seminar-Grounded Tasks. arXiv:2509.01396

    arXiv 2025

  30. [30]

    Wang, Y .; Wang, L.; Deng, Y .; Wu, K.; Xiao, Y .; Yao, H.; Kang, L.; Ye, H.; Jing, Y .; and Bing, L. 2026. Deep- ResearchEval: An Automated Framework for Deep Research Task Construction and Agentic Evaluation. arXiv:2601.09688

    arXiv 2026

  31. [31]

    W.; Passos, A

    Wei, J.; Sun, Z.; Papay, S.; McKinney, S.; Han, J.; Fulford, I.; Chung, H. W.; Passos, A. T.; Fedus, W.; and Glaese, A. 2025. BrowseComp: A Sim- ple Yet Challenging Benchmark for Browsing Agents. arXiv:2504.12516

    Pith/arXiv arXiv 2025

  32. [32]

    Wong, R.; Wang, J.; Zhao, J.; Chen, L.; Gao, Y .; Zhang, L.; Zhou, X.; Wang, Z.; Xiang, K.; Zhang, G.; Huang, W.; Wang, Y .; and Wang, K

  33. [33]

    Wu, J.; Li, B.; Fang, R.; Yin, W.; Zhang, L.; Tao, Z.; Zhang, D.; Xi, Z.; Fu, G.; Jiang, Y .; Xie, P.; Huang, F.; and Zhou, J. 2025. WebDancer: Towards Autonomous Information Seeking Agency. arXiv:2505.22648

    Pith/arXiv arXiv 2025

  34. [34]

    Wu, J.; Yin, W.; Jiang, Y .; Wang, Z.; Xi, Z.; Fang, R.; Zhang, L.; He, Y .; Zhou, D.; Xie, P.; and Huang, F. 2025. WebWalker: Benchmarking LLMs in Web Traversal. arXiv:2501.07572

    Pith/arXiv arXiv 2025

  35. [35]

    arXiv:2508.07999

    WideSearch: Benchmarking Agentic Broad Info-Seeking. arXiv:2508.07999

    Pith/arXiv arXiv

  36. [36]

    Xie, J.; Lin, T.; Wang, Z.; Ning, Y .; Yao, Y .; Xue, T.; Zhang, Z.; Li, Z.; Zhang, K.; Wu, Y .; Chen, S.; Gou, B.; Han, M.; Wang, Y .; Lee, V .; Wei, X.; Wang, X.; Su, Y .; and Sun, H. 2026. QUEST: Training Fron- tier Deep Research Agents with Fully Synthetic Tasks. arXiv:2605.24218

    Pith/arXiv arXiv 2026

  37. [37]

    Xu, R.; and Peng, J. 2025. A Comprehensive Survey of Deep Research: Systems, Methodologies, and Ap- plications. arXiv:2506.12594

    Pith/arXiv arXiv 2025

  38. [38]

    Wu, T.; Wang, Y .; Ma, X.; He, X.; Wang, S.; Yin, D.; and Zhao, X. 2026. DeepResearch-9K: A Chal- lenging Benchmark Dataset of Deep-Research Agent. arXiv:2603.01152

    Pith/arXiv arXiv 2026

  39. [39]

    Z.ai. 2026. GLM-5.2: Built for Long-Horizon Tasks

    2026

  40. [40]

    Zhang, G.; Geng, H.; Yu, X.; Yin, Z.; Zhang, Z.; Tan, Z.; Zhou, H.; Li, Z.; Xue, X.; Li, Y .; Zhou, Y .; Chen, Y .; Zhang, C.; Fan, Y .; Wang, Z.; Huang, S.; Piedrahita- Velez, F.; Liao, Y .; Wang, H.; Yang, M.; Ji, H.; Wang, J.; Yan, S.; Torr, P.; and Bai, L. 2025. The Landscape of Agentic Reinforcement Learning for LLMs: A Survey. arXiv:2509.02547

    Pith/arXiv arXiv 2025

  41. [41]

    Ye, J.; Du, Z.; Yao, X.; Lin, W.; Xu, Y .; Chen, Z.; Wang, Z.; Zhu, S.; Xi, Z.; Yuan, S.; Gui, T.; Zhang, Q.; Huang, X.; and Chen, J. 2025. ToolHop: A Query- Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use. arXiv:2501.02506

    Pith/arXiv arXiv 2025

  42. [44]

    latest / current / recent

    Zhou, S.; Xu, F. F.; Zhu, H.; Zhou, X.; Lo, R.; Srid- har, A.; Cheng, X.; Ou, T.; Bisk, Y .; Fried, D.; Alon, U.; and Neubig, G. 2023. WebArena: A Realistic Web Environment for Building Autonomous Agents. arXiv:2307.13854. Appendix Implementation Details and Experimental Settings Prompt Templates The following are the full construction and query-transform...

    Pith/arXiv arXiv 2023

  43. [45]

    **Main object **: any confusion about the query’s main object? any dropped subquestion?

  44. [46]

    **Comma segments **: split on ‘,‘; count **<= 3 ** (strongly prefer **<= 2 **)? If it still lists parallel items, use umbrella phrasing

  45. [47]

    levels‘ / ‘including‘ / ‘considering

    **Checklist smell **: still 2+ parallel facets (‘A and B‘, ‘A, B, or C‘)? ‘based on‘ / ‘in terms of‘ / ‘at ... levels‘ / ‘including‘ / ‘considering ... and ... broader ... history‘ / ‘roles behind‘? stacked style adjectives? Delete or merge all

  46. [48]

    **Sentences & question marks **: split on ‘.!?‘ + space; **<= 3 sentences **? **<= 2 ‘?‘**? Merge to one sentence when possible

  47. [49]

    **Length ratio **: ‘len(query_fuzzified) / len(query) <= 0.70‘? If not, keep deleting modifiers, lists, and method words

  48. [50]

    **Process leakage **: can a reader infer step order? Any First/Then/Investigate/Analyze by? Rewrite

  49. [51]

    first A then B

    **Read-aloud test **: if it sounds like a research rubric or outline rather than a real-user question, rewrite. Assigned-Topic Query Prompt The mermaid below describes my completed deep research content and conclusions (nodes capture the research process and information; check_points are the key findings/evidence). ## My research (mermaid) {mermaid} What ...

    1934

  50. [1971]

    0.09 E Verify completion of state ratifi- cation (1) States that the three-fourths threshold was reached on July 1, 1971

    (3) Accurately describes the governors’ and federal certifying official’s roles. 0.09 E Verify completion of state ratifi- cation (1) States that the three-fourths threshold was reached on July 1, 1971. (2) States that 38 of 50 states were required and identifies North Carolina as the 38th state when discussed. (3) Distinguishes completed ratification fro...

    1971

  51. [2025]

    arXiv:2511.07685

    ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents. arXiv:2511.07685

    arXiv

  52. [2026]

    arXiv:2605.11732

    AgentDisCo: Towards Disentanglement and Collaboration in Open-ended Deep Research Agents. arXiv:2605.11732

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.