REVIEW 3 major objections 37 references
A verifiable offline search world lets web agents improve from their own tool traces without stronger teachers.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-14 15:40 UTC pith:W25HYHEG
load-bearing objection Solid systems paper on offline-verifiable self-distillation for search agents; main live-web numbers still mix in a small GRPO patch that is not isolated. the 3 major comments →
DeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environment
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Deterministic, entity-verifiable offline search-and-read tools plus scaffold-to-ReAct self-distillation are enough for a 9B agent to climb to competitive deep-search performance from its own filtered rollouts, without trajectories from stronger models.
What carries the argument
DeepSearch-World and DeepSearch-Evolve: an offline BM25/SQLite Wikipedia tool pair with order-free entity progress verification and staged grounded reflection, feeding an evolving supervised loop that rewrites scaffolded teacher traces (plan, memory, failure recovery) into deployable ReAct student data.
Load-bearing premise
Skills learned under entity checkmarks inside a closed Wikipedia simulator will transfer to messy live web tools and non-encyclopedia domains.
What would settle it
Retrain the same self-evolving loop with answer-correct trajectories only, without entity-level verification and without scaffold-to-ReAct conversion, and check whether BrowseComp and GAIA scores fall sharply; or evaluate the offline-only agent on live search tools without the small real-tool adaptation stage and measure the drop.
If this is right
- Open-source search agents can close much of the tool-use gap by iterating in verifiable sandboxes rather than copying proprietary traces.
- Entity-level process verification can substitute for sparse outcome rewards when training long-horizon search agents.
- Scaffold planning and reflection can be stripped before training and still leave memory tracking and self-correction inside a standard ReAct policy.
- Larger multi-hop task pools sustain gains across more self-evolving rounds by covering more failure and recovery patterns.
- Schema-aligned offline tools can later be swapped for live search and visit APIs with limited extra adaptation.
Where Pith is reading between the lines
- The same entity-walk construction and progress checks could be ported to other structured corpora beyond encyclopedias.
- Once process labels are reliable, soft on-policy distillation may become more stable than it is in noisy live tool settings.
- Competitive deep-search numbers from pure self-experience imply proprietary trajectory distillation is less necessary for search agents than is often assumed.
- Failure-recovery behaviors induced by staged reflection may be the main transferable skill when the agent leaves Wikipedia.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces DeepSearch-World, a deterministic offline Wikipedia environment with BM25 search and SQLite visit tools plus 420K multi-hop QA tasks from entity-level random walks, and DeepSearch-Evolve, an iterative self-distillation loop that generates scaffold (Plan–Act–End) trajectories, filters them by answer correctness and quality, converts them to ReAct format via state internalization and reflection rewriting, and fine-tunes the student. Starting from Qwen3.5-9B-Instruct, DeepSearch-World-9B is trained for 11 evolving-SFT rounds and a final GRPO stage on 1,600 live-tool instances, reporting 31.2% BrowseComp, 61.5% GAIA, and 93.4% HotpotQA without distillation from stronger proprietary models. Ablations on SearchQA/DeepSearch-Val isolate rejection sampling, quality filtering, and scaffold-to-ReAct conversion; behavior analyses show longer horizons and higher advanced-capability scores.
Significance. If the attribution holds, the work is a solid systems contribution to self-evolving tool-use agents: a reproducible, entity-verifiable offline environment that supplies process-level signals (progress verification, grounded reflection, failure recovery) and an evolving-SFT recipe that avoids sparse-reward RL instability and proprietary trajectory distillation. Planned release of the environment, 420K pool, validation set, model, and code is a genuine community asset. Large gains over the same backbone and competitive open-source numbers would support the thesis that verifiable environments can scale long-horizon web-agent self-improvement. The significance is tempered until live-tool competitiveness is cleanly attributed to the offline loop rather than the final GRPO patch.
major comments (3)
- Sec. 4.1 and Table 1: the headline claim that offline verifiable self-evolution alone yields competitive open-source deep-search agents is not isolated. DeepSearch-World-9B is evaluated after 11 evolving-SFT rounds plus GRPO on 1,600 SerpAPI/Jina instances, yet no checkpoint after offline-only training is reported under the same live tools used for baselines. Ablations (Tables 2–3, Fig. 5) stay on SearchQA/DeepSearch-Val. Without an offline-only live evaluation (and ideally a GRPO-only control from the backbone), competitiveness cannot be attributed to DeepSearch-World/DeepSearch-Evolve rather than conventional live-tool RL. This is load-bearing for the abstract’s central claim.
- Sec. 3.1.3 and 3.2: teacher rollouts receive privileged environment-grounded reflection that can escalate to canonical entity names or fuzzy descriptions of unresolved walk entities. Student training rewrites these signals, but teacher trajectory quality—and thus the self-evolution loop—depends on process labels unavailable in open-web deployment. The paper should quantify how much performance depends on this privileged process supervision (e.g., teacher with answer-only filtering vs. entity-progress reflection) and clarify that “self-evolution” here is self-distillation under construction-time entity ground truth, not unsupervised discovery of intermediate progress.
- Table 1 / Sec. 4.1–4.2: baseline comparison fairness is under-specified for the competitiveness claim. Open-source agents span 7B–9B scales, heterogeneous tool backends, and often stronger-model distillation; proprietary systems use different tool stacks. The paper notes some of this but does not report a controlled same-tools, same-budget evaluation of DeepSearch-World-9B vs. the strongest open baselines (e.g., MiroThinker, Marco-DR, WebExplorer). Without that, “competitive with open-source agents” remains only partially supported even if the GRPO confound is fixed.
Circularity Check
No load-bearing circularity: self-evolution is gated by external ground-truth answers and entity sets, and headline scores are external-benchmark measurements—not quantities forced by the training objective.
specific steps
-
other
[Sec. 3.3 Trajectory generation and verification; Training Configuration Sec. 4.1]
"Then verified successful trajectories are converted into ReAct-format supervision to train the student πθR+1, which will be the next teacher. ... We retain trajectories with correct answer and further prompt Qwen3.5-9B (Qwen Team, 2026) to apply trajectory-level filtering... We train DeepSearch-World-9B from Qwen3.5-9B for 11 self-evolving rounds"
Later rounds train on trajectories produced by earlier versions of the same agent, and the quality filter is the same 9B family as the student. This is ordinary iterative self-training, not a definitional loop: keep/reject still depends on external answer correctness and entity-level environment checks, and reported gains are on held-out external benchmarks. Flagged only as minor self-reference, not as a forced prediction.
full rationale
DeepSearch-Evolve is iterative SFT on the agent’s own rollouts, but retention is not self-referential by construction. Trajectory generation is filtered by answer correctness against the multi-hop QA targets built from Wikipedia random walks (Sec. 3.1.1, 3.3), and intermediate progress is checked against stored ground-truth entity sets Ti via order-free matching (Sec. 3.1.3)—both external to the model’s beliefs. Scaffold-to-ReAct conversion rewrites planning/reflection into standard ReAct traces (Eq. 2) and trains with hard-label NLL/SFT (Eq. 3); that is imitation of verified traces, not a fitted parameter renamed as a prediction. Main results (BrowseComp 31.2%, GAIA 61.5%, HotpotQA 93.4%) are evaluations on external benchmarks under a shared tool setting, not in-sample reconstructions of the 420K training pool. Self-citations (e.g., Geng et al. 2025 for random walks; Fang et al. 2026 for scaffold phases) supply methodology, not uniqueness theorems that force the competitive claim. The only mild self-reference is that quality filtering uses Qwen3.5-9B (same family as the student) and later rounds retrain on earlier self-generated data—standard self-training, not circular derivation. Attribution gaps (offline SFT vs final SerpAPI/Jina GRPO) affect causal credit for the headline numbers but do not make those numbers tautological. Score 1 for that minor self-reference only.
Axiom & Free-Parameter Ledger
free parameters (4)
- self-evolving rounds R =
11 (main)
- importance-sampling decay γ =
0.5
- trajectory budget Tmax and generation/filter thresholds =
Tmax=30; Cmin=4000
- GRPO real-tool stage size =
1600 instances
axioms (4)
- domain assumption An offline Wikipedia corpus with BM25 search and deterministic page visit is a sufficiently faithful proxy for web search/browse tool schemas.
- domain assumption Entity-set membership after each tool call is an objective, order-free measure of intermediate progress for multi-hop QA.
- domain assumption Answer-correct trajectories after quality filtering are adequate hard targets for SFT self-distillation of planning and recovery skills.
- ad hoc to paper Scaffold progress state and environment reflections can be rewritten into standard ReAct <think> blocks without destroying the learned behaviors.
invented entities (2)
-
DeepSearch-World
no independent evidence
-
DeepSearch-Evolve / scaffold teacher (Plan–Act–End)
no independent evidence
read the original abstract
Training tool-use agents to improve from their own experience remains challenging, as supervised fine-tuning relies on fixed teacher-distilled trajectories, while sparse-reward reinforcement learning provides weak supervision for long-horizon interactions. We present DeepSearch-Evolve, a self-distillation framework for web agents built on DeepSearch-World, a deterministic and verifiable environment with reproducible search and page-reading tools. DeepSearch-World contains 420K multi-hop QA tasks constructed from entity-level random walks and supports key agentic cognitive behaviors useful for self-evolving, including progress verification, grounded reflection, and failure recovery. DeepSearch-Evolve iteratively performs trajectory generation, filtering, data mixing, and fine-tuning to train stronger agents. Without distillation from more capable models, DeepSearch-World-9B achieves competitive performance compared with open-source agents, reaching 31.2% on BrowseComp, 61.5% on GAIA, and 93.4% on HotpotQA, showing that verifiable environments enable scalable self-evolution for long-horizon web agents. We will release the environment, 420K training pool, validation set, model, and code to facilitate future research on self-improving deep search agents.
Figures
Reference graph
Works this paper leans on
-
[1]
Cognitive kernel-pro: A framework for deep research agents and agent foundation models train- ing. Preprint, arXiv:2508.00414. Jiaxuan Gao, Wei Fu, Minyang Xie, Shusheng Xu, Chuyi He, Zhiyu Mei, Banghua Zhu, and Yi Wu
-
[2]
arXiv preprint arXiv:2508.07976
Beyond ten turns: Unlocking long-horizon agentic search with large-scale asynchronous rl. arXiv preprint arXiv:2508.07976. Xinyu Geng, Peng Xia, Zhen Zhang, Xinyu Wang, Qiuchen Wang, Ruixue Ding, Chenxi Wang, Jia- long Wu, Yida Zhao, Kuan Li, and 1 others
-
[3]
arXiv preprint arXiv:2508.05748
Webwatcher: Breaking new frontier of vision-language deep research agent. arXiv preprint arXiv:2508.05748. Xinyu Geng, Y anjing Xiao, Y uyang Zhang, Hanwen Wang, Xinyan Liu, Rui Min, Tianqing Fang, and Yi R Fung. 2026. Geobrowse: A geolocation bench- mark for agentic tool use with expert-annotated rea- soning traces. arXiv preprint arXiv:2604.04017. Googl...
Pith/arXiv arXiv 2026
-
[4]
Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations. Proceedings of the 44th Interna- tional ACM SIGIR Conference on Research and De- velopment in Information Retrieval . Junteng Liu, Y unji Li, Chi Zhang, Jingyang Li, Aili Chen, Ke Ji, Weiyu Cheng, Zijia Wu, Chengyu Du, Qidi Xu, and 1 others. 2...
arXiv 2025
-
[5]
arXiv preprint arXiv:2602.23166
Agentvista: Evaluating multimodal agents in ultra-challenging realistic visual scenarios. arXiv preprint arXiv:2602.23166. Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Y ong Jiang, Pengjun Xie, Y an Zhang, Fei Huang, and Jingren Zhou. 2025. Zerosearch: Incen- tivize the search capability of llms without searching. arXiv preprint arXiv:2505.045...
arXiv 2025
-
[6]
arXiv preprint arXiv:2601.18734
Self-distilled reasoner: On-policy self- distillation for large language models. arXiv preprint arXiv:2601.18734. Y aowei Zheng, Richong Zhang, Junhao Zhang, Y anhan Y e, Zheyan Luo, Zhangchi Feng, and Y ongqiang Ma
-
[7]
In Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (V olume 3: System Demonstra- tions), Bangkok, Thailand
Llamafactory: Unified efficient fine-tuning of 100+ language models . In Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (V olume 3: System Demonstra- tions), Bangkok, Thailand. Association for Compu- tational Linguistics. Y uxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Y e, Pengrui Lu, and Pengfei Liu
-
[8]
DeepResearcher: Scaling deep research via reinforcement learning in real-world environments . In Proceedings of the 2025 Conference on Empiri- cal Methods in Natural Language Processing, pages 414–431, Suzhou, China. Association for Computa- tional Linguistics. Peilin Zhou, Bruce Leon, Xiang Ying, Can Zhang, Yifan Shao, Qichen Y e, Dading Chong, Zhiling J...
Pith/arXiv arXiv 2025
-
[9]
For each seed article, we perform breadth-first exploration
Knowledge Tree Construction. For each seed article, we perform breadth-first exploration. At each level, an LLM (Gemini-3-Pro) selects the b most relevant hyperlinks (default b=3) from the current page. Exploration continues for up to dmax levels (default dmax=4) or until pmax pages (default pmax=6) have been visited. Each edge in the tree records the sema...
-
[10]
From the complete knowl- edge tree, we randomly sample 2–4 subtrees
Subtree Sampling. From the complete knowl- edge tree, we randomly sample 2–4 subtrees. Each subtree defines a set of entities (nodes) and their relationships (edges)
-
[11]
For each sampled sub- tree, the LLM generates a complex multi- hop question that requires locating and cross- referencing information about all entities in the subtree
Question Generation. For each sampled sub- tree, the LLM generates a complex multi- hop question that requires locating and cross- referencing information about all entities in the subtree. The question is designed to be answer- able only by consulting multiple Wikipedia ar- ticles
-
[12]
2014” ! “the 2010s
Feature Fuzzification. To prevent memoriza- tion and ensure genuine multi-step reasoning, we apply feature fuzzification : specific values in the question are replaced with approximate descriptors (e.g., “2014” ! “the 2010s”, “Bei- jing, China” ! “a major Chinese city”). This forces the agent to search for and verify specific facts rather than pattern-matchin...
2014
-
[13]
Wikipedia XML dumps are processed via WikiExtractor to obtain clean article text
-
[14]
Extracted articles are merged into a single JSONL file with {id, contents, caption} schema
-
[15]
A SQLite offset index is built for random ac- cess by article title
-
[16]
A Pyserini BM25 index is constructed over the full corpus for keyword retrieval
-
[17]
The resulting corpus contains approximately 10 million entries, providing comprehensive cover- age of English Wikipedia
Missing text (articles not in the dump) is sup- plemented via controlled online scraping with rate limiting. The resulting corpus contains approximately 10 million entries, providing comprehensive cover- age of English Wikipedia. A.2 Tool Implementation Details web_search_wiki. web_search_wiki imple- ments the search interface of the virtual Wikipedia env...
-
[18]
query":
over the offline Wikipedia corpus. Each indexed document contains three fields: keyword, caption, and contents, corresponding to the article title, lead-section summary or feature description, and full article text, respectively. For each retrieved article, the tool returns a structured result ri = fkeywordi, captioni, urlig, i = 1, . . . , k, (5) where url...
2024
-
[19]
Check: Is there any overlap or repeated request?
Information Non-Redundancy: The requested information or action in the tool call is not already provided or easily derivable from prior dialogue, the users current question, or the as- sistants previous answers. Check: Is there any overlap or repeated request?
-
[20]
Check: Does it advance the users main objective?
Goal Alignment: The tool calls pur- pose and expected result directly serve the users explicit intent or core need in this turn. Check: Does it advance the users main objective?
-
[21]
name": "tool name here
Logical Reasoning and Accuracy: The assistants thought process shows clear, correct logic and reliable grounding – no unfounded guesses or fabrications. The <think> sec- tion should be concise. Check: Is the reasoning well-structured and evidence-based? Instruction: Compare the users question and the models generated snippet (includ- ing <tool_call> and <...
-
[22]
**Final Result**: Carefully examine the outputs from the previous steps to decide the final output
-
[23]
how many thousands
**Output Rules**: Your final output should be a number OR as few words as possible OR a comma separated list of numbers and/or strings. Do NOT include any unnecessary information in the output. 18 - **Number**: If you are asked for a number, directly output the number itself. Don 't use comma to write your number. Be careful about what the question is ask...
-
[24]
`assistant`: Your response `<think>...</think>` then `<tool_call>...</tool_call>`
-
[25]
`user`: The tool result `<too ⌋ l_response>...</tool_response>`
-
[26]
`user`: A `[REFLECTION]` message environment-injected progress feedback
-
[27]
`user`: A `[STATE_UPDATE]` prompt asking you (in your state-tracker role) to update the Progress State
-
[28]
**When you are acting as the action module (this prompt)**, you produce messages 1 and 5
`assistant`: Your state-update response a JSON block with the updated Progress State. **When you are acting as the action module (this prompt)**, you produce messages 1 and 5. Messages 2 and 3 are injected by the environment. Message 4 is a separate prompt sent to you in a state-tracker role. Do NOT generate `[REFLECTION]` or `[STATE_UPDATE]` in your acti...
-
[29]
Call one tool at a time; wait for its result before deciding the next step. 19
-
[31]
Pass all necessary context explicitly in the arguments
Each tool call is stateless its internal state is discarded after return. Pass all necessary context explicitly in the arguments
-
[32]
NEVER put next steps, search plans, or action descriptions inside `<answer>`
Use `<answer>` ONLY when you already have the confirmed, specific answer (a fact, name, number, etc.) obtained from tool results. NEVER put next steps, search plans, or action descriptions inside `<answer>`
-
[33]
Found target keyword X
`[REFLECTION]` messages are **system-injected** progress-tracking signals they are added automatically by the environment after tool calls. **You must NEVER generate or output `[REFLECTION]` yourself.** Keywords in "Found target keyword X" or "remaining targets" are intermediate research targets to look up they are NOT the answer. Only output `<answer>` a...
-
[34]
name": "web_search_wiki
** `<think>` is MANDATORY**: You MUST output `<think>...</think>` BEFORE every `<tool_call>` or `<answer>`. A response without `<think>` is invalid and will be rejected. ## Tool Call Guidelines - **Mandatory two-step pattern NO EXCEPTIONS**: `web_search_wiki` `visit_wiki` extract fact answer. - After `web_search_wiki` returns, you MUST call `visit_wiki` n...
1939
-
[35]
Call one tool at a time; wait for its result before deciding the next step
-
[36]
If a tool call fails or returns irrelevant results, try a different query or tool do not repeat the same call
-
[37]
Each tool call is stateless pass all necessary context explicitly in the arguments
-
[38]
completed_list
Use `<answer>` ONLY when you already have the confirmed answer from tool results. NEVER put next steps or search plans inside `<answer>`. ## Progress State Four fields (ALL writable update every round): - completed_list: Finished steps and confirmed findings. - todo_list: **Living checklist** remove finished items, add new follow-ups, rewrite failing sear...
1906
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.