Pith. sign in

REVIEW 3 major objections 6 minor 5 cited by

This paper introduces a 9,000-question deep-research benchmark with verifiable answers and an open training framework that lets a 3B model outscore DeepSeek V3.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 19:41 UTC pith:EQV4TVWK

load-bearing objection Useful released dataset and training framework, but the 'verifiable' label and SOTA results rest on a single unvalidated judge and a teacher-failed-only test set. the 3 major comments →

arxiv 2603.01152 v2 pith:EQV4TVWK submitted 2026-03-01 cs.AI

DeepResearch-9K: A Challenging Benchmark Dataset of Deep-Research Agent

Tongzhou Wu , Yuhao Wang , Xinyu Ma , Xiuqiang He , Shuaiqiang Wang , Dawei Yin , Xiangyu Zhao This is my paper
classification cs.AI
keywords deep-research agentsmulti-hop question answeringbenchmark datasetreinforcement learningLLM-as-judgesearch trajectoriesdifficulty scalingentity obfuscation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The authors aim to close two gaps they see in the field: few large, hard, real-world-style research tasks, and few open-source pipelines to train agents on them. They synthesize 9,000 multi-hop research questions from three existing QA datasets, write them as deliberately obfuscated narratives, and annotate each with a difficulty tier tied to the number of search-tool calls needed. They label the associated answer trajectories as correct or incorrect using a large language model as judge, release the dataset, and provide a training framework that supports reward-based tuning. Their headline finding is that a small open model (3B parameters) trained with reinforcement learning on this data reaches higher accuracy on the benchmark than the much larger judge model itself. If true, this suggests that careful benchmark construction plus reinforcement learning can transfer deep-research skills into much smaller models.

Core claim

The paper's central claim is that DeepResearch-9K provides a reusable, verifiable benchmark whose L1–L3 tiers reliably separate shallow retrieval from genuine long-horizon research, and that agents trained on its trajectories with the DeepResearch-R1 framework achieve state-of-the-art results on that benchmark. The empirical evidence is that the teacher agent's accuracy falls sharply from about 72% at L1 to about 24% at L3, matching its score on the existing BrowseComp-Plus benchmark, and that a 3B Llama model trained with PPO (zero-RL) reaches 22.5% on the test set, above the 20.18% of the much larger DeepSeek V3. The dataset and training pipeline are both released open-source.

What carries the argument

The load-bearing object is the three-tier difficulty construct: L1 (direct attribute lookups), L2 (multi-hop relational chains), and L3 (long entity chains hidden behind dense, obfuscated narratives, designed to force at least 15 independent search queries). Difficulty is operationalized as search-tool call count, and the construction pipeline—entity extraction from HotpotQA, 2WikiMultihopQA, and MuSiQue; chain building with a rule that no single page contains more than two consecutive chain entities; and progressive entity obfuscation—automates this scaling. On the training side, the mechanism is DeepResearch-R1, an open framework that supports search-tool interaction, PPO/GRPO, and a rewar

Load-bearing premise

The whole edifice rests on the reliability of DeepSeek V3 as an LLM judge: it labels which teacher trajectories are correct, scores every model's final answers, and supplies the reward signal in both reinforcement-learning stages; if the judge is biased toward a particular style of answer—especially on highly obfuscated L3 questions—then the verifiable-answer label and the reported accuracies, including the comparison against DeepSeek V3, are not trustworthy.

What would settle it

Take a random sample of about 300 L3 questions, have two independent human annotators verify the ground-truth answers against the cited sources (for example, the Wikipedia pages involved in the chain), and compare their verdicts with the LLM judge's correct/incorrect labels; if the judge's labels do not reproduce under human verification at a high rate (say, above 90%), the verifiability claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A model trained on DeepResearch-9K trajectories with reinforcement learning can achieve higher accuracy on the benchmark than the much larger model that provided the judge and reward signal.
  • L3 performance of the teacher (about 23.7%) closely matches its performance on BrowseComp-Plus (about 24.9%), which the authors interpret as evidence that L3 captures real-world research difficulty.
  • The pipeline costs about $200 in search API fees and roughly 8,000 A100 GPU-hours, so other groups can afford to synthesize similarly hard tasks without large human annotation.
  • Because difficulty is defined by search-call counts, future dataset builders can calibrate task hardness automatically, without human judges.
  • The results suggest a training recipe—SFT for cold-start mitigation plus RL, or zero-RL with PPO—that practitioners can copy for small open agents.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the same model family (DeepSeek-V3) both synthesizes the questions, judges the answers, and provides the reward signal, there is a risk of style bias or self-support that is not controlled for; a fair test would re-verify with human labelers or a different judge.
  • If the difficulty calibration holds beyond this benchmark, the same synthesis recipe could be ported to other domains—scientific literature, financial filings, or code archaeology—where 'research' means long chains of independent lookups.
  • The claim that a 3B model beats the much larger teacher on this benchmark should be tested on the broader BrowseComp-Plus set; the paper only reports the test set of DeepResearch-9K, so the generalization of the 'small beats large' result remains open.
  • The most direct testable extension: apply the same training pipeline to BrowseComp-Plus questions and see whether the small model's margin persists outside the synthetic dataset.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces DeepResearch-9K, a dataset of 9,000 multi-hop web-research questions with search trajectories generated by Tongyi-DeepResearch-30B-A3B, partitioned into three difficulty tiers (L1–L3) based on required search tool calls, and with answers labeled correct/incorrect by DeepSeek-V3 as an LLM-as-judge. The authors also release DeepResearch-R1, an open-source training framework supporting SFT and RL (PPO/GRPO) with rule-based and LLM-as-judge rewards. Experiments report that 3B-parameter models trained on DeepResearch-9K achieve state-of-the-art results, including outperforming DeepSeek V3 on a test set composed of teacher-failed samples.

Significance. If the evaluation issues are addressed, this is a valuable contribution: it provides a low-cost, open-source pipeline for synthesizing deep-research training data from existing QA benchmarks, a public dataset with trajectories and difficulty tiers, and a framework for training small models for long-horizon web-research tasks. The comparison of zero-RL versus SFT+RL and PPO versus GRPO is useful for practitioners. The reproducibility of the pipeline (released code and data) is a concrete strength.

major comments (3)
  1. [§4.1.2–§4.1.3, Table 1, Figure 4] The same model, DeepSeek-V3, is used for trajectory labeling (Table 1), as the RL reward signal in both stages (§4.1.2), and as the LLM-as-judge for final answer correctness (§4.1.3). The paper cites preference leakage in LLM-as-judge [27] but provides no human-agreement study, no independent judge, and no sensitivity analysis. Consequently, the central claims—that the dataset has 'verifiable answers' and that Llama-3.2-3B (22.50%) achieves SOTA over DeepSeek V3 (20.18%) in Figure 4—could be artifacts of judge bias or reward hacking rather than genuine capability. Please report a human-verified subset (e.g., 200–300 samples) with inter-annotator agreement, or an independent judge model, and a comparison of scores.
  2. [§4.1.1, Figure 4] The test set is constructed by placing all 5,026 teacher-correct trajectories and 2,200 randomly selected teacher-incorrect trajectories into training, leaving the remaining 1,774 teacher-failed samples as the test set. Thus the evaluation in Figure 4 is on a non-representative hard subset, not on DeepResearch-9K as a whole. The claim that trained agents achieve state-of-the-art results on DeepResearch-9K is not supported by these numbers. The authors should evaluate on a random split of all 9,000 instances (reporting per-tier accuracy) or clearly present performance on the full set and on the hard subset separately.
  3. [§2.2, §3.2] The difficulty tiers are defined a priori by required tool-call counts (L1: 1–2, L2: multi-hop, L3: ≥15 searches), and the 'validation' in Figure 2 reports the teacher model's own tool-call counts. This is partially self-confirming: the teacher's search behavior is used to confirm a difficulty construct defined in terms of search counts. The claim in §4.2 that L3 'accurately captures the intricate relationships and high reasoning demands of real-world research' is based solely on matching the teacher's accuracy on BrowseComp-Plus (23.73% vs. 24.94%), which is a single-model comparison and does not establish correspondence with human-judged research difficulty. Please add independent difficulty ratings (e.g., human judges or a different teacher) or a more direct analysis of the required reasoning steps.
minor comments (6)
  1. [§3.4] The rule-based quality assurance is underspecified. 'Deconstructing high-quality examples from BrowseComp-Plus using LLM-based analysis' does not state what rules are extracted, how they are applied, or how the 'cross-document independence checks' are implemented.
  2. [§3 (end of first paragraph)] Duplicate phrase: 'These steps are detailed in the following sections, which will be detailed in the following sections' should be cleaned up.
  3. [Figure 4] The legend and bar labels ('Best RL only SFT+RL') are difficult to parse. Please label each model configuration explicitly (e.g., 'Qwen-2.5-3B PPO Zero-RL') and, if possible, include error bars or the number of runs.
  4. [§2.1] The statement 'we assume that the more difficult the problem, the more times the search tool is called' should be framed as a design assumption, not a validated property. The text later treats the observed teacher search counts in Figure 2 as evidence of difficulty, which conflates assumption with result.
  5. [§4.1.3] The evaluation metric description says 'we aim to evaluate whether the L1-L3 tasks possess a clear difficulty gradation' but the paper presents Figure 2 as a validation of that gradation. Clarify that the gradation is observed for the teacher model, not for the trained models.
  6. [§3, Abstract] The abstract emphasizes a low-cost pipeline with a cost of 200 US dollars, but §3 also reports 8,064 A100 GPU hours. Please state the full compute cost and clarify that the $200 figure refers only to API fees, not total computational cost.

Circularity Check

3 steps flagged

DeepSeek-V3 is the sole correctness oracle — labeling data, supplying RL rewards, and scoring accuracy — so SOTA claims measure agreement with the training judge itself; difficulty tiers are 'validated' by the same tool-call constraints used to build them.

specific steps
  1. fitted input called prediction [§2.1; §4.1.2; §4.1.3; Figure 4]
    "Each sample has a verifiable final answer. Specifically, in the following, LLM-as-judge is adopted to verify the correctness of the answer. [§2.1] The reward signal in both RL phases is provided by DeepSeek-V3, which evaluates the quality and correctness of the agent’s reasoning trajectories. [§4.1.2] We evaluate the correctness of the final prediction using an LLM-as-judge paradigm... processed by DeepSeek-V3. [§4.1.3]"

    Correctness is defined by DeepSeek-V3 at every stage: it verifies the dataset answers, labels the 5,026 'correct' SFT trajectories, supplies the RL reward in both PPO and GRPO phases, and scores the final accuracy. Agents are therefore optimized to maximize DeepSeek-V3's verdicts, and the reported accuracy / SOTA standing (Llama-3.2-3B 22.50% vs DeepSeek V3 20.18%, Figure 4) is measured by that same verdict function. No human or independent verification is reported anywhere, so 'state-of-the-art' is a statement about agreement with the training judge rather than an independent measurement; the paper even cites LLM-as-judge preference leakage [27] without mitigation.

  2. self definitional [§2.2; §3.2; Figure 3]
    "We adopt the number of autonomous searches as the primary target for research difficulty. [§2.2] ... L1 tasks are strictly constrained to 1–2 search iterations; L3 ... ensuring a minimum of 15 search iterations for successful resolution. [§3.2] ... The mean number of tool calls increases substantially with each level: from 4.30 at L1 to 10.74 at L2, and reaching 20.23 at L3. This consistent increase validates the effectiveness of our dataset construction strategy. [§2.2]"

    Difficulty is defined as required search count, and the construction prompts impose those counts per tier (L1: 1–2 searches; L3: ≥15 searches, per Figure 3 'Difficulty Target'). The paper then presents the teacher model's mean tool-call counts per tier as empirical validation of the difficulty ladder. The validation metric is the same variable that the construction constraints set, so the observed L1<L2<L3 escalation is guaranteed by construction, not discovered. 'Difficulty is defined by search count' and 'search count validates difficulty' are the same assertion; only the magnitude (e.g., L3 mean 20.23 vs the 15-search floor) carries independent empirical content.

  3. self definitional [§3; §4.1.1; Figure 4]
    "We introduce DeepResearch-Hard, a challenging subset ... with 3,974 instances. Specifically, it only contains the samples on which the teacher model Tongyi-DeepResearch-30B-A3B fails to provide a correct answer, as verified by our LLM-as-judge framework. [§3] ... we combine all 5,026 correct trajectories with a randomly selected subset of 2,200 incorrect samples, forming a training set of 7,226 instances. The remaining 1,774 samples form the test set. [§4.1.1]"

    The test set on which every trained-model comparison and the benchmark-difficulty conclusion rest ('DeepSeek V3 achieves only 20.18% underscores the substantial challenge posed by DeepResearch-9K') is the complement of the judge-approved teacher trajectories: membership is defined by DeepSeek-V3's 'incorrect' verdicts on the teacher. The test set's hard character is thus co-defined by the same judge used as RL reward and final scorer, and every accuracy number on it is the judge's own score. The 'challenging' property of the benchmark is partly an artifact of this judge-defined selection rather than an independent measurement, and the figure conflates this 1,774-sample hard subset with the full 9K dataset's difficulty.

full rationale

The two load-bearing claims — that DeepResearch-9K has 'verifiable answers' with a validated L1–L3 difficulty ladder, and that agents trained on it achieve state-of-the-art results — both reduce, in substantial part, to a single unvalidated instrument: DeepSeek-V3 as LLM-as-judge. Correctness is defined by that judge (ground-truth verification, teacher-trajectory labeling, SFT selection), used as the RL reward in both training phases, and then used as the evaluation metric; no human or independent verification is reported. The headline comparison (3B model 22.50% vs DeepSeek V3 20.18%) is therefore a within-judge comparison of models trained to maximize that same judge's approval, so the SOTA claim is confounded with the reward signal — the paper cites preference leakage [27] but never mitigates it. Separately, difficulty tiers are defined as tool-call counts, imposed in the construction prompts (L1 1–2; L3 ≥15), and then 'validated' by the teacher's measured tool-call counts; this validation is the construction constraint echoed back. The test set is also the set of teacher-failed samples, where failure is the judge's own negative verdict, further co-defining the measured difficulty. Credit where due: seeds come from open-source QA benchmarks, the L3-vs-BrowseComp-Plus accuracy match provides an external anchor, and no load-bearing self-citations were found. But the correctness and difficulty claims central to the paper's contribution reduce by construction to the judge and to the imposed constraints, warranting a score of 7.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

No new physical or conceptual entities are postulated; the paper's contribution is a dataset and framework. The key external inputs are hand-selected difficulty thresholds, an unvalidated LLM judge, and a teacher model whose trajectories are assumed to be high-quality demonstrations.

free parameters (3)
  • Difficulty-tier thresholds (L1 1-2, L3 >=15 searches; chain length 5-6; max 2 consecutive entities per source) = hand-selected in prompt design
    These thresholds define the L1–L3 labels (§3.2) and are subsequently 'validated' by the same teacher model's mean tool-call counts (§2.2), so the difficulty scale is not independently calibrated.
  • Balanced instance distribution (3,000 per difficulty level) = 3,000
    The equal split across L1/L2/L3 is a design choice, not a principled sample from a natural difficulty distribution (§2.2, §3).
  • Number of teacher-failed trajectories added to training set = 2,200
    The choice to include 2,200 incorrect trajectories as negative training data is arbitrary and not justified; it also forces the test set to be the remaining 1,774 teacher-failed samples (§4.1.1).
axioms (4)
  • domain assumption DeepSeek-V3 LLM-as-judge gives correct verdicts for answer verification
    Used to label teacher trajectories correct/incorrect (§3, Table 1), to measure final accuracy (§4.1.3), and as RL reward (§4.1.2); no human agreement or calibration reported.
  • domain assumption Search tool call count is a valid proxy for research difficulty
    The paper defines difficulty in terms of required tool calls and validates L1–L3 by the teacher's mean calls (§2.2, §3.2).
  • domain assumption Teacher model trajectories are high-quality demonstrations when judged correct
    SFT uses 5,026 'correct' trajectories without manual review; trajectory quality beyond final answer correctness is assumed (§4.1.2).
  • domain assumption Entity extraction and graph expansion by DeepSeek-V3 preserve answer verifiability
    The pipeline uses LLM extraction/expansion to create new questions; no independent check that generated questions have unique, verifiable target answers (§3.1).

pith-pipeline@v1.3.0-alltime-deepseek · 13810 in / 15505 out tokens · 144084 ms · 2026-08-02T19:41:04.464943+00:00 · methodology

0 comments
read the original abstract

Deep-research agents are capable of executing multi-step web exploration, targeted retrieval, and sophisticated question answering. Despite their powerful capabilities, deep-research agents face two critical bottlenecks: (1) the lack of large-scale, challenging datasets with real-world difficulty, and (2) the absence of accessible, open-source frameworks for data synthesis and agent training. To bridge these gaps, we first construct DeepResearch-9K, a large-scale challenging dataset specifically designed for deep-research scenarios built from open-source multi-hop question-answering (QA) datasets via a low-cost autonomous pipeline. Notably, it consists of (1) 9000 questions spanning three difficulty levels from L1 to L3 (2) high-quality search trajectories with reasoning chains from Tongyi-DeepResearch-30B-A3B, a state-of-the-art deep-research agent, and (3) verifiable answers. Furthermore, we develop an open-source training framework DeepResearch-R1 that supports (1) multi-turn web interactions, (2) different reinforcement learning (RL) approaches, and (3) different reward models such as rule-based outcome reward and LLM-as-judge feedback. Finally, empirical results demonstrate that agents trained on DeepResearch-9K under our DeepResearch-R1 achieve state-of-the-art results on challenging deep-research benchmarks. We release the DeepResearch-9K dataset on https://huggingface.co/datasets/artillerywu/DeepResearch-9K and the code of DeepResearch-R1 on https://github.com/Applied-Machine-Learning-Lab/DeepResearch-R1.

Figures

Figures reproduced from arXiv: 2603.01152 by Dawei Yin, Shuaiqiang Wang, Tongzhou Wu, Xiangyu Zhao, Xinyu Ma, Xiuqiang He, Yuhao Wang.

Figure 1
Figure 1. Figure 1: A data sample from DeepResearch-9K including [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Distribution and mean of search tool call frequency [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Detailed prompt template for Level-3 (L3) deep￾research tasks in the DeepResearch-9K dataset. Besides, we introduce DeepResearch-Hard, a challenging sub￾set of DeepResearch-9K with 3,974 instances. Specifically, it only contains the samples on which the teacher model Tongyi-DeepResearch￾30B-A3B fails to provide a correct answer, as verified by our LLM￾as-judge framework [27, 61]. The complete LLM-as-judge … view at source ↗
Figure 4
Figure 4. Figure 4: Model performance comparison on the test set of [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. FORT-Searcher: Synthesizing Shortcut-Resistant Search Tasks for Training Deep Search Agents

    cs.CL 2026-06 unverdicted novelty 7.0

    FORT synthesizes shortcut-resistant search tasks by controlling four identified shortcut risks across entity selection, graph construction, question formulation, and refinement, producing training data that yields age...

  2. SciDataSailor: Deep Scientific Data Exploring

    cs.AI 2026-07 conditional novelty 6.0

    SciDataSailor uses Monte Carlo tree search to synthesize tool-use trajectories for training and benchmarking LLM agents that explore raw scientific repositories.

  3. MemSearch-o1: Empowering Large Language Models with Reasoning-Aligned Memory Growth in Agentic Search

    cs.IR 2026-04 unverdicted novelty 6.0

    MemSearch-o1 uses reasoning-aligned memory growth from seed tokens, retracing via contribution functions, and path reorganization to mitigate memory dilution in LLM agentic search.

  4. MemSearch-o1: Empowering Large Language Models with Reasoning-Aligned Memory Growth in Agentic Search

    cs.IR 2026-04 unverdicted novelty 6.0

    MemSearch-o1 mitigates memory dilution in agentic LLM search through reasoning-aligned token-level memory growth, retracing with a contribution function, and path reorganization, improving reasoning activation on benchmarks.

  5. SpecHop: Continuous Speculation for Accelerating Multi-Hop Retrieval Agents

    cs.CL 2026-05 unverdicted novelty 5.0

    SpecHop accelerates multi-hop LLM tool use via continuous multi-threaded speculation with asynchronous verification, approaching oracle latency gains and reducing latency up to 40% on retrieval tasks.

Reference graph

Works this paper leans on

77 extracted references · 31 linked inside Pith · cited by 4 Pith papers

  1. [1]

    Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, et al

  2. [2]

    Muhammad Arslan, Hussam Ghanem, Saba Munawar, and Christophe Cruz. 2024. A Survey on RAG with LLMs.Procedia computer science246 (2024), 3781–3790

    2024

  3. [3]

    Luyi Bai, Wenting Yu, Mingzhuo Chen, and Xiangnan Ma. 2021. Multi-hop reasoning over paths in temporal knowledge graphs using reinforcement learning. Applied Soft Computing103 (2021), 107144

    2021

  4. [4]

    Ma Chang, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhen- zhong Lan, Lingpeng Kong, and Junxian He. 2024. Agentboard: An analytical evaluation board of multi-turn llm agents.Advances in neural information pro- cessing systems37 (2024), 74325–74362

    2024

  5. [5]

    Anthony Chen, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Eval- uating question answering evaluation. InProceedings of the 2nd workshop on machine reading for question answering. 119–124

    2019

  6. [6]

    Danqi Chen and Wen-tau Yih. 2020. Open-domain question answering. InPro- ceedings of the 58th annual meeting of the association for computational linguistics: tutorial abstracts. 34–37

    2020

  7. [7]

    Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, et al. 2025. Browsecomp- plus: A more fair and transparent evaluation benchmark of deep-research agent. arXiv preprint arXiv:2508.06600(2025)

    Pith/arXiv arXiv 2025

  8. [8]

    Yaswanth Chittepu, Raghavendra Addanki, Tung Mai, Anup Rao, and Branislav Kveton. 2025. ML-Tool-Bench: Tool-Augmented Planning for ML Tasks.arXiv preprint arXiv:2512.00672(2025)

    arXiv 2025

  9. [9]

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multi- modality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261(2025)

    Pith/arXiv arXiv 2025

  10. [10]

    Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. 2025. DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents. arXiv preprint arXiv:2506.11763(2025)

    Pith/arXiv arXiv 2025

  11. [11]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models.arXiv e-prints(2024), arXiv–2407

    2024

  12. [12]

    Yiwen Gao, Ruochen Zhao, Yang Deng, and Wenxuan Zhang. 2026. DR-Arena: an Automated Evaluation Framework for Deep Research Agents.arXiv preprint arXiv:2601.10504(2026)

    Pith/arXiv arXiv 2026

  13. [13]

    Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Am- manamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna Clinciu, Dipanjan Das, Kaustubh Dhole, et al. 2021. The gem benchmark: Natural language generation, its evaluation and metrics. InProceedings of the 1st Workshop on Natural Language Generation, Evaluation, ...

    2021

  14. [14]

    Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies.Transactions of the Association for Computational Linguistics9 (2021), 346–361

    2021

  15. [15]

    Haisong Gong, Jing Li, Junfei Wu, Qiang Liu, and Shu Wu. 2026. Strive: Struc- tured reasoning for self-improvement in claim verification.Machine Intelligence Research23, 1 (2026), 185–199

    2026

  16. [16]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al . 2024. A survey on llm-as-a-judge.The Innovation(2024)

    2024

  17. [17]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al . 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948(2025)

    Pith/arXiv arXiv 2025

  18. [18]

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps.arXiv preprint arXiv:2011.01060(2020)

    Pith/arXiv arXiv 2020

  19. [19]

    Jiabao Ji, Min Li, Priyanshu Kumar, Shiyu Chang, and Saloni Potdar. 2025. DEEP- AMBIGQA: Ambiguous Multi-hop Questions for Benchmarking LLM Answer Completeness.arXiv preprint arXiv:2511.01323(2025)

    arXiv 2025

  20. [20]

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516(2025)

    Pith/arXiv arXiv 2025

  21. [21]

    Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. 1996. Rein- forcement learning: A survey.Journal of artificial intelligence research4 (1996), 237–285

    1996

  22. [22]

    Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan Van Esch, Nasanba- yar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, et al. 2022. Quality at a glance: An audit of web-crawled multilingual datasets.Transactions of the Association for Computational Linguistics10 (2022), 50–72

    2022

  23. [23]

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics7 (2019), 453–466

    2019

  24. [24]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Mem- ory Management for Large Language Model Serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles

    2023

  25. [25]

    Matthew Lamm, Jennimaria Palomaki, Chris Alberti, Daniel Andor, Eunsol Choi, Livio Baldini Soares, and Michael Collins. 2021. Qed: A framework and dataset for explanations in question answering.Transactions of the Association for com- putational Linguistics9 (2021), 790–806

    2021

  26. [26]

    Tian Lan, Bin Zhu, Qianghuai Jia, Junyang Ren, Haijun Li, Longyue Wang, Zhao Xu, Weihua Luo, and Kaifu Zhang. 2025. Deepwidesearch: Benchmarking depth and width in agentic information seeking.arXiv preprint arXiv:2510.20168(2025)

    arXiv 2025

  27. [27]

    Dawei Li, Renliang Sun, Yue Huang, Ming Zhong, Bohan Jiang, Jiawei Han, Xiangliang Zhang, Wei Wang, and Huan Liu. 2025. Preference leakage: A con- tamination problem in llm-as-a-judge.arXiv preprint arXiv:2502.01534(2025)

    arXiv 2025

  28. [28]

    Haonan Li, Xudong Han, Hao Wang, Yuxia Wang, Minghan Wang, Rui Xing, Yilin Geng, Zenan Zhai, Preslav Nakov, and Timothy Baldwin. 2025. Loki: An open-source tool for fact verification. InProceedings of the 31st International Conference on Computational Linguistics: System Demonstrations. 28–36

    2025

  29. [29]

    Kuan Li, Zhongwang Zhang, Huifeng Yin, Liwen Zhang, Litu Ou, Jialong Wu, Wenbiao Yin, Baixuan Li, Zhengwei Tao, Xinyu Wang, et al. 2025. WebSailor: Nav- igating Super-human Reasoning for Web Agent.arXiv preprint arXiv:2507.02592 (2025)

    Pith/arXiv arXiv 2025

  30. [30]

    Yuchen Li, Hengyi Cai, Rui Kong, Xinran Chen, Jiamin Chen, Jun Yang, Haojie Zhang, Jiayi Li, Jiayi Wu, Yiqun Chen, et al. 2025. Towards AI Search Paradigm. arXiv preprint arXiv:2506.17188(2025)

    arXiv 2025

  31. [31]

    Xiao Ling, Sameer Singh, and Daniel S Weld. 2015. Design challenges for entity linking.Transactions of the Association for Computational Linguistics3 (2015), 315–328

    2015

  32. [32]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437(2024)

    Pith/arXiv arXiv 2024

  33. [33]

    Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al . 2025. Deepseek- v3. 2: Pushing the frontier of open large language models.arXiv preprint arXiv:2512.02556(2025)

    Pith/arXiv arXiv 2025

  34. [34]

    Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang

  35. [35]

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. 2023. Agentbench: Evaluating llms as agents.arXiv preprint arXiv:2308.03688(2023)

    Pith/arXiv arXiv 2023

  36. [36]

    Zhiwei Liu, Weiran Yao, Jianguo Zhang, Le Xue, Shelby Heinecke, Rithesh Murthy, Yihao Feng, Zeyuan Chen, Juan Carlos Niebles, Devansh Arpit, et al. 2023. Bolaa: Benchmarking and orchestrating llm-augmented autonomous agents.arXiv preprint arXiv:2308.05960(2023)

    Pith/arXiv arXiv 2023

  37. [37]

    Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2023. Gaia: a benchmark for general ai assistants. InThe Twelfth Inter- national Conference on Learning Representations

    2023

  38. [38]

    Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2019. Multi-hop reading comprehension through question decomposition and rescoring. arXiv preprint arXiv:1906.02916(2019)

    Pith/arXiv arXiv 2019

  39. [39]

    Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, et al

  40. [40]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schul- man, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Pe- ter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human...

    Pith/arXiv arXiv 2022

  41. [41]

    Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, et al. 2021. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua...

    2021

  42. [42]

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al . 2025. Humanity’s last exam.arXiv preprint arXiv:2501.14249(2025)

    Pith/arXiv arXiv 2025

  43. [43]

    Peng Qi, Xiaowen Lin, Leo Mehr, Zijian Wang, and Christopher D Manning. 2019. Answering complex open-domain questions through iterative query generation. arXiv preprint arXiv:1910.07000(2019). Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Tongzhou Wu et al

    Pith/arXiv arXiv 2019

  44. [44]

    Hongjin Qian and Zheng Liu. 2025. Scent of Knowledge: Optimizing Search- Enhanced Reasoning with Information Foraging.arXiv preprint arXiv:2505.09316 (2025)

    arXiv 2025

  45. [45]

    Hongjin Qian, Zheng Liu, Chao Gao, Yankai Wang, Defu Lian, and Zhicheng Dou. 2025. HawkBench: Investigating Resilience of RAG Methods on Stratified Information-Seeking Tasks.arXiv preprint arXiv:2502.13465(2025)

    arXiv 2025

  46. [46]

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis.arXiv preprint arXiv:2307.16789(2023)

    Pith/arXiv arXiv 2023

  47. [47]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. InFirst Conference on Language Modeling

    2024

  48. [48]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

  49. [49]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300(2024)

    Pith/arXiv arXiv 2024

  50. [50]

    Ying Shen, Ning Ding, Hai-Tao Zheng, Yaliang Li, and Min Yang. 2020. Modeling relation paths for knowledge graph completion.IEEE Transactions on Knowledge and Data Engineering33, 11 (2020), 3607–3617

    2020

  51. [51]

    Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al

  52. [52]

    Shashi Kant Singh, Shubham Kumar, and Pawan Singh Mehra. 2023. Chat gpt & google bard ai: A review. In2023 International Conference on IoT, Communication and Automation Technology (ICICAT). IEEE, 1–6

    2023

  53. [53]

    Tongyi DeepResearch Team, Baixuan Li, Bo Zhang, Dingchu Zhang, Fei Huang, Guangyu Li, Guoxin Chen, Huifeng Yin, Jialong Wu, Jingren Zhou, et al. 2025. Tongyi DeepResearch Technical Report.arXiv preprint arXiv:2510.24701(2025)

    Pith/arXiv arXiv 2025

  54. [54]

    Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models.arXiv preprint arXiv:2104.08663(2021)

    Pith/arXiv arXiv 2021

  55. [55]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal

  56. [56]

    Chaojie Wang, Yanchen Deng, Zhiyi Lyu, Liang Zeng, Jujie He, Shuicheng Yan, and Bo An. 2024. Q*: Improving multi-step reasoning for llms with deliberative planning.arXiv preprint arXiv:2406.14283(2024)

    Pith/arXiv arXiv 2024

  57. [57]

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2024. A survey on large language model based autonomous agents.Frontiers of Computer Science18, 6 (2024), 186345

    2024

  58. [58]

    Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese

  59. [59]

    Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents.Transactions of the Association for Computational Linguistics6 (2018), 287–302

    2018

  60. [60]

    Mengyuan Wu, Ke Li, Sam Kwong, Qingfu Zhang, and Jun Zhang. 2018. Learning to decompose: A paradigm for decomposition-based multiobjective optimization. IEEE Transactions on Evolutionary Computation23, 3 (2018), 376–390

    2018

  61. [61]

    Transactions of the Association for Computational Linguistics10 (2022), 539–554

    MuSiQue: Multi-hop Questions via Single-hop Question Composition. Transactions of the Association for Computational Linguistics10 (2022), 539–554

    2022

  62. [62]

    Wenhan Xiong, Xiang Lorraine Li, Srini Iyer, Jingfei Du, Patrick Lewis, William Yang Wang, Yashar Mehdad, Wen-tau Yih, Sebastian Riedel, Douwe Kiela, et al. 2020. Answering complex open-domain questions with multi-hop dense retrieval.arXiv preprint arXiv:2009.12756(2020)

    Pith/arXiv arXiv 2020

  63. [63]

    Renjun Xu and Jingwen Peng. 2025. A Comprehensive Survey of Deep Research: Systems, Methodologies, and Applications.arXiv preprint arXiv:2506.12594(2025)

    Pith/arXiv arXiv 2025

  64. [64]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Cheng- peng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfen...

    Pith/arXiv arXiv 2024

  65. [65]

    arXiv preprint arXiv:2504.12516(2025)

    Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12516(2025)

    Pith/arXiv arXiv 2025

  66. [66]

    Huihan Yao, Ying Chen, Qinyuan Ye, Xisen Jin, and Xiang Ren. 2021. Refining language models with compositional explanations.Advances in neural information processing systems34 (2021), 8954–8967

    2021

  67. [67]

    Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. 2025. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild.arXiv preprint arXiv:2503.18892(2025)

    Pith/arXiv arXiv 2025

  68. [68]

    Xingpeng Xiao, Yaomin Zhang, Jian Xu, Wenkun Ren, and Junyi Zhang. 2025. Assessment methods and protection strategies for data leakage risks in large language models.Journal of Industrial Engineering and Applied Science3, 2 (2025), 6–15

    2025

  69. [69]

    Dawei Zhou, Tongliang Liu, Bo Han, Nannan Wang, Chunlei Peng, and Xinbo Gao. 2021. Towards defending against adversarial examples via attack-invariant features. InInternational conference on machine learning. PMLR, 12835–12845

    2021

  70. [72]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. InProceedings of the 2018 conference on empirical methods in natural language processing. 2369–2380

    2018

  71. [75]

    Siyun Zhao, Yuqing Yang, Zilong Wang, Zhiyuan He, Luna K Qiu, and Lili Qiu

  72. [2017]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347 (2017)

    Pith/arXiv arXiv 2017

  73. [2020]

    Logiqa: A challenge dataset for machine reading comprehension with logical reasoning.arXiv preprint arXiv:2007.08124(2020)

    Pith/arXiv arXiv 2007

  74. [2022]

    Lila: A unified benchmark for mathematical reasoning.arXiv preprint arXiv:2210.17517(2022)

    Pith/arXiv arXiv 2022

  75. [2023]

    InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

    Mega: Multilingual evaluation of generative ai. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 4232–4267

    2023

  76. [2024]

    Retrieval augmented generation (rag) and beyond: A comprehensive sur- vey on how to make your llms use external data more wisely.arXiv preprint arXiv:2409.14924(2024)

    Pith/arXiv arXiv 2024

  77. [2025]

    Openai gpt-5 system card.arXiv preprint arXiv:2601.03267(2025)

    Pith/arXiv arXiv 2025