Pith. sign in

REVIEW 3 major objections 5 minor 74 references

Beyond Outcome Rewards: Step-Level Self-Distilled Policy Optimization for Deep Search Agents

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Step-level self-distillation beats GRPO at half the steps.

desk verdict A real, clearly-documented step-level process supervision method for search agents with an honest ablation story, undercut by thin validation of the evidence anchors that carry the gain. read the letter →

arxiv 2608.12764 v1 pith:BYEWVJHL submitted 2026-08-13 cs.LG cs.AI

Haoze Wu , Chuqiao Kuang , Tianyi Zhuang , Xiaoguang Li This is my paper · ORCID
classification cs.LGcs.AI
keywords self-distillationdeepsearchagentsGRPOstep-levelcreditassignmentprocesssupervisionevidenceanchorswebreinforcementlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Deep search agents produce trajectories with dozens of steps but receive only one binary reward, so ordinary policy-gradient training cannot tell which steps helped. This paper claims that the missing credit assignment can be supplied by the agent itself: a teacher conditioned on web-derived evidence snippets scores each search step, and the resulting teacher–student disagreement is converted into step-level weights inside the GRPO objective, applied only to failed trajectories. On Qwen3-8B, the proposed method (SSPO) improves accuracy over GRPO on BrowseComp, GAIA, and FRAMES, and 100 training steps beat GRPO trained for 200 steps with roughly 5% extra compute per step. If correct, fine-grained process supervision for search agents is attainable without a separate reward model or hand-labeled process data.

What carries the argument

The central object is the step-level privileged-information gain $$\$\Delta$^{\mathrm{step}}_\tau = \mathrm{sg}\left(\log P_T(t_\tau,a_\tau) - \log P_S(t_\tau,a_\tau)\right),$$ the logged ratio of teacher to student joint likelihood for one search step (thought plus tool call), under a teacher conditioned on Evidence Anchors and a student that is not. It enters the GRPO surrogate as a multiplicative weight $w_\tau = \min(\exp(\mathrm{sign}(A^{(i)})\Delta^{\mathrm{step}}_\tau), 1+\epsilon)$ on every token of that step, for incorrect trajectories only. Evidence Anchors are the privileged input: compact, step-level web snippets that support the ground-truth answer, collected in advance by prompting a strong language model. This machinery decouples update direction (set by the group-relative outcome advantage) from update magnitude (set by teacher–student agreement), and aligns supervision with the action that actually performs search.

What would settle it

A manual audit of, say, 100 sampled Evidence Anchors—checking whether each snippet's content actually supports the ground-truth answer—would settle the claim if a large share fail while SSPO's win over GRPO persists; alternatively, replacing anchors with random web snippets of similar length should drop performance if the anchor semantics are doing the work.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that privileged-information leakage in self-distillation—normally a failure mode—can be repurposed as a score: instead of trying to make the student match a teacher that has seen the answer, SSPO keeps the environment reward as the only update direction and lets the teacher's log-likelihood ratio modulate how strongly each step is reinforced or penalized. The modulation is step-level, treating a thought plus its tool call as one information-seeking action, and is applied only to incorrect trajectories so correct behavior stays diverse. Empirically this yields consistently higher accuracy than GRPO across three open-ended search benchmarks and matches or exceeds GRPO trained twice as long.

Load-bearing premise

The method assumes the Evidence Anchors truly support the correct answer; the paper validates only that their URLs are accessible and their titles match, not that the snippet content itself is accurate support.

Editorial extensions

If this is right

  • At the same number of training steps, SSPO attains higher accuracy than GRPO on all three benchmarks, and 100-step SSPO surpasses 200-step GRPO, so step-level process supervision substantially accelerates credit assignment in deep search.
  • Using the self-distillation signal as an advantage weight rather than a direct optimization target is necessary: direct distribution matching collapses tool use and underperforms even GRPO.
  • Step-level weighting beats token-level weighting, supporting the view that the thought-plus-tool-call pair, not the token, is the natural unit of credit assignment for search agents.
  • The training dynamics show aligned entropy but sustained gradient norms relative to GRPO, indicating the gains come from more effective policy updates rather than increased exploration.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same step-level advantage scheme could transfer to other long-horizon tool-use agents, such as coding or OS control, if a privileged evidence source of comparable quality can be built; the paper only tests web search.
  • Editorial inference: anchor quality, not optimization, may become the bottleneck at larger scale; an ablation that varies anchor count or introduces corrupted anchors would reveal how sensitive SSPO is to teacher input quality.
  • Editorial inference: a direct comparison against an explicit process-reward-model baseline on identical trajectories would clarify whether self-distilled advantage weights are a full substitute for learned process rewards or just a cheaper approximation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper addresses sparse outcome rewards in deep web-search agents by introducing Step-Level Self-Distilled Policy Optimization (SSPO). The method constructs Evidence Anchors—concise web-derived snippets supporting the ground-truth answer—as privileged information for a self-teacher, then converts teacher–student disagreement into step-level advantage weights inside GRPO, applied only to incorrect trajectories. The authors evaluate on Qwen3-8B across BrowseComp, GAIA, and FRAMES, reporting that SSPO outperforms GRPO at equal training steps and surpasses or matches GRPO trained for twice as many steps, with roughly 5% additional per-step computational overhead from one extra forward pass. The paper also includes ablations separating direct distillation from advantage weighting, token-level from step-level weighting, and the roles of Evidence Anchors versus incorrect-answer feedback.

Significance. If the results hold, the paper makes a useful contribution: it offers a way to inject process-level supervision into long-horizon search-agent RL without an external reward model, and it aligns the supervision granularity with the natural unit of search actions. The design is well motivated, the ablations are informative, and the analysis of which steps receive amplified or reduced penalties is a nice qualitative check. The authors also release code and report training dynamics and overhead, which strengthens reproducibility. However, the evidence-anchor validation gap and the absence of uncertainty quantification leave the central empirical claim not fully established.

major comments (3)
  1. [Appendix G, Table 5] The validation of Evidence Anchors checks only URL accessibility and whether the retrieved page title matches the reported title; it does not verify that the webpage content actually supports the condition it is claimed to support. This is load-bearing because the teacher distribution PT in Eq. (6) is conditioned on these anchors, and Table 5 shows that removing anchors drops BC-Sub from 14.5 to 12.4, below the GRPO baseline of 12.8. A hallucinated or semantically irrelevant anchor would corrupt the step-level advantage weights in Eq. (8), potentially inverting credit assignment. The authors should either add a semantic validity check (e.g., human evaluation on a sample, or an automatic consistency check between anchor explanations and page content) or report a sensitivity analysis showing that the gains are robust to anchor quality. The current statement that only a small fraction of anchors have inaccessible URLs does not address hallucinated content on accessible pages.
  2. [Section 4.2, Table 3, Figure 10] All comparisons are based on a single training run without confidence intervals or multiple seeds. The reported differences are often modest—for example, BrowseComp 15.7 versus 13.6, GAIA 49.3 versus 47.3, and FRAMES 73.0 versus 69.8—and the Avg@4 evaluation reduces but does not eliminate evaluation noise. The central claim that SSPO consistently outperforms GRPO and matches or surpasses GRPO at twice the training steps would be much stronger with either multiple seeds or at least bootstrap confidence intervals on the evaluation metric. Without this, the reader cannot assess whether the reported gains are statistically reliable.
  3. [Section 4.3, Table 5] The ablation separating Evidence Anchors from incorrect-answer feedback reports a single number per condition (14.5 vs. 14.0 vs. 12.4) with no variance information. Given that the difference between the full setting and the no-anchor setting is about two percentage points, and that the no-anchor setting falls below the GRPO baseline, this ablation is central to the claim that Evidence Anchors are the primary driver. Reporting multiple runs or at least a bootstrap interval on these ablation numbers would substantially increase confidence in the mechanism.
minor comments (5)
  1. [Figure 1] The label 'BC-EN' in the right panel is not defined or used elsewhere; it likely refers to BrowseComp or BC-Sub and should be made consistent with the rest of the text.
  2. [Table 3] The benchmark name is written as 'Frames' in the table header but as 'FRAMES' in the text and elsewhere; please unify the notation.
  3. [Section 3.2, Eq. (8)] The hyperparameter epsilon is introduced in Eq. (8) but its value and choice are only mentioned in Appendix B; consider stating the default value and that it is not swept early in the main text.
  4. [Appendix G] The claim that 'only a very small fraction of Evidence Anchors contain inaccessible URLs' would be more informative with the actual number or percentage of anchors filtered, along with the number of QA pairs removed from training.
  5. [Table 2] The row for A(i)>0 under 'incorrect trajectories' is explained in the footnote, but the table alone could confuse readers; consider adding a short parenthetical in the table caption or renaming the axis to make clear this case arises from format rewards.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: SSPO's advantage weights are defined from the outcome advantage and teacher/student likelihoods, and the claimed benchmark gains are externally evaluated rather than derived from the method's inputs.

full rationale

The central derivation is self-contained. Equation 7 defines the privileged-information gain as the log-likelihood ratio between teacher and student for a step, and Equation 8 defines the SSPO weight as min(exp(sign(A)*Delta), 1+epsilon); neither quantity is fitted to the benchmarks on which the method is evaluated. The teacher is the student's own policy conditioned on privileged evidence anchors, which is the self-distillation premise being tested rather than a hidden input from which the outcome is derived. The empirical claim that SSPO outperforms GRPO on BrowseComp, GAIA, and FRAMES is an externally evaluated result, not a consequence of the equations by construction. The paper's citations to the authors' prior work ([18], [46], [47]) are contextual and non-load-bearing; the key design choices are credited to external work (RLSD [54], SRPO [17]) and supported by the paper's own ablations. Appendix G's anchor validation check, which verifies only URL accessibility and title matching, is a real evidence-quality limitation, since hallucinated anchor content could corrupt the teacher distribution in Equation 6 and hence the step-level gain in Equation 7, but this is a data-validity risk and not a circular reduction of the kind required for a circularity flag. No step of the derivation reduces, by definition or by self-citation, to the result it is used to support.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The central claim rests on a small set of hand-chosen hyperparameters (epsilon, teacher refresh interval), the validity of teacher log-likelihood ratios as step-quality signals, and the semantic accuracy of machine-generated evidence anchors. No formal verification is provided; the evidence is empirical and conditional on these assumptions.

free parameters (2)
  • epsilon (advantage clipping threshold) = 0.2
    Hand-chosen in Eq. 8; caps the multiplicative weight and controls how strongly teacher agreement can change GRPO advantages. No sensitivity analysis is reported.
  • teacher reinitialization interval = 50 training steps
    The teacher is refreshed from the current policy every 50 steps; staleness of the teacher is a design choice not ablated.
assumptions (4)
  • standard math Autoregressive factorization of teacher and student step probabilities (Eq. 6) is valid.
    Standard product-of-token-conditional probabilities; not controversial.
  • domain assumption Teacher-student log-likelihood ratio sign (Eq. 7-8) is a valid proxy for step quality in search.
    The method assumes a step the teacher favors under privileged evidence is a better information-seeking action; no direct process-level ground truth validates this.
  • domain assumption Evidence anchors are semantically accurate and sufficient support for the ground-truth answer.
    Appendix G validates only URL accessibility and title matching, not whether anchor content actually supports the answer; hallucinated anchors would corrupt teacher signal.
  • domain assumption Outcome reward (R_correct via LLM judge) correctly identifies trajectory correctness.
    Correctness is judged by an LLM judge; the prompt and reliability are not specified, and all training signal directions depend on this binary label.
invented entities (1)
  • Evidence Anchors
    purpose: Step-level evidence snippets extracted from the web and inserted into the teacher prompt as privileged information.
    They are a new construct central to the method, but the paper releases neither the anchors nor semantic validation; only URL/title checks are described, so no external falsifiable handle is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Outcome Rewards: Step-Level Self-Distilled Policy Optimization for Deep Search Agents." pith.science (2026). https://pith.science/paper/BYEWVJHL

@misc{pith2026260812764,
  author       = {Pith},
  title        = {Pith review of: Beyond Outcome Rewards: Step-Level Self-Distilled Policy Optimization for Deep Search Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BYEWVJHL}},
  note         = {Machine review of arXiv:2608.12764}
}
read the original abstract

Deep search agents operate over trajectories spanning dozens of steps, yet standard reinforcement learning provides only a single outcome reward per trajectory, which is far too sparse for effective credit assignment. On-policy self-distillation (OPSD) addresses this by using the model's own logits as dense token-level teachers, but extending it to search agents introduces a fundamental tension: the teacher, having access to privileged information such as the correct answer, produces a distribution that differs systematically from the student's exploration-based reasoning, and naive distillation causes the student to inherit this information asymmetry rather than learn better search strategies. We resolve this tension through two contributions. First, we construct Evidence Anchors, which are concise, step-level evidence snippets extracted from the web, as privileged information that captures key reasoning steps without revealing the entire answer path. Second, we propose Step-Level Self-Distilled Policy Optimization (SSPO), which converts teacher-student disagreement into step-level advantage weights within GRPO, applied exclusively to incorrect trajectories. This design decouples what to update from how much to update: the outcome reward determines the direction of policy change, while the teacher modulates its magnitude at each step. Correct trajectories are left untouched, preserving their diversity. On Qwen3-8B, SSPO consistently outperforms GRPO across BrowseComp, GAIA, and FRAMES, surpassing or matching GRPO trained with twice as many gradient steps while adding only about 5 percent overhead per step from a single additional forward pass.

Figures

Figures reproduced from arXiv: 2608.12764 by the authors.

Figure 1
Figure 1. Comparison of performance across training methods. Left: After introducing step-level self-distilled advantage weights (SSPO), performance improves substantially faster during training, surpassing GRPO trained for 200 steps after only 100 steps. Right: We report results for the cold-start model, GRPO after 100 and 200 steps, and SSPO after 100 steps. 1Code is available at https://github.com/hkust-nlp/SSPO. Correspon… view at source ↗
Figure 2
Figure 2. An example question with three evidence anchors. The differently colored evidence anchors [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Teacher prompt template incorporating evidence anchors and previously generated incorrect [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Comparison of training dynamics between GRPO and SSPO (EMA Smoothed). [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 6
Figure 6. Figure 6: Average number of queries under differ￾ent step-level privileged-information gains. 5 Case Study: Which Steps are Exempted from Penalty? To better understand why SSPO is effective, we analyze how the teacher assigns penalties across different types of steps. We find th…
Figure 7
Figure 7. Figure 7: ∆step τ > 0: compact single-query verification targeting one specific entity-attribute relation. <think> ERNI matches the consultancy with offices across Europe and Asia. Now find an ERNI employee who is a "medical technology software specialist" and holds a "foundatio…
Figure 8
Figure 8. Figure 8: ∆step τ < 0: diffuse multi-query search with high lexical overlap but poor localization. in a setting where neither reference solutions nor execution feedback are available, we introduced Evidence Anchors—structured, step-aligned privileged information whose granularit…
Figure 9
Figure 9. Figure 9: Modified Prompt template used in the Iterative Query Evolution stage. Accuracy #Tool #Browse #Search All 0.74 32.1 10.5 21.6 Correct - 20.5 8.3 12.2 Incorrect - 65.5 16.9 48.6 [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Training dynamics of GRPO and SSPO on BC-Sub, GAIA, and Frames-Sub. [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Statistics of Evidence Anchor Counts. H Limitations and Future Work Due to the high cost associated with API usage (e.g., Serper, Jina, and LLM services), the scale of both data construction and experimental evaluation is constrained. For both cold-start and on￾policy…
Figure 12
Figure 12. Figure 12: Prompt template used to collect Evidence Anchors. [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

74 extracted references · 49 canonical work pages

  1. [1]

    Glm-4.6.https://docs.z.ai/guides/llm/glm-4.6, 2025

    Zhipu AI. Glm-4.6.https://docs.z.ai/guides/llm/glm-4.6, 2025

    work page 2025

  2. [2]

    Glm-5.1.https://docs.z.ai/guides/llm/glm-5.1, 2026

    Zhipu AI. Glm-5.1.https://docs.z.ai/guides/llm/glm-5.1, 2026. 10

    work page 2026

  3. [3]

    Introducing Claude 4 — anthropic.com

    Anthropic. Introducing Claude 4 — anthropic.com. https://www.anthropic.com/news/ claude-4, 2025. [Accessed 19-03-2026]

    work page 2025

  4. [4]

    Introducing claude opus 4.7

    Anthropic. Introducing claude opus 4.7. https://www.anthropic.com/news/ claude-opus-4-7, 2026

    work page 2026

  5. [5]

    Is gpt-oss good? a comprehensive evaluation of openai’s latest open source models, 2025

    Ziqian Bi, Keyu Chen, Chiung-Yi Tseng, Danyang Zhang, Tianyang Wang, Hongying Luo, Lu Chen, Junming Huang, Jibin Guan, Junfeng Hao, Xinyuan Song, and Junhao Song. Is gpt-oss good? a comprehensive evaluation of openai’s latest open source models, 2025

    work page 2025

  6. [6]

    SFT memorizes, RL generalizes

    Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. SFT memorizes, RL generalizes. InProceedings of the 42nd International Conference on Machine Learning, pages 10818–10838. PMLR, 2025

    work page 2025

  7. [7]

    Gemini 3.1 pro: Best for complex tasks and bringing creative concepts to life.https://deepmind.google/models/gemini/pro/, 2026

    Google Deepmind. Gemini 3.1 pro: Best for complex tasks and bringing creative concepts to life.https://deepmind.google/models/gemini/pro/, 2026

    work page 2026

  8. [8]

    Deepseek-v4: Towards highly efficient million-token context intelligence, 2026

    DeepSeek-AI. Deepseek-v4: Towards highly efficient million-token context intelligence, 2026

    2026

Show all 74 references
  1. [9]

    Deepseek-v3.2: Pushing the frontier of open large language models, 2025

    DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenhao Xu, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, et al. Deepseek-v3.2: Pushing the frontier...

    2025

  2. [10]

    Openthoughts: Data recipes for reasoning models, 2025

    Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, Ashima Suvarna, Benjamin Feuer, Liangyu Chen, Zaid Khan, Eric Frankel, et al. Openthoughts: Data recipes for reasoning models, 2025

    2025

  3. [11]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, et al. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning.Nature, 645(8081):63...

    2025

  4. [12]

    Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. InProceedings of the 28th International Conference on Computational Linguistics, pages 6609–6625, Barcelona, Spain (Online), De...

    2020

  5. [13]

    Reinforcement learning via self-distillation.arXiv preprint arXiv:2601.20802, 2026

    Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, and Andreas Krause. Reinforcement learning via self-distillation.arXiv preprint arXiv:2601.20802, 2026

    2026 arXiv

  6. [14]

    An empirical study on reinforcement learning for reasoning-search interleaved llm agents.arXiv preprint arXiv:2505.15117, 2025

    Bowen Jin, Jinsung Yoon, Priyanka Kargupta, Sercan O Arik, and Jiawei Han. An empirical study on reinforcement learning for reasoning-search interleaved llm agents.arXiv preprint arXiv:2505.15117, 2025

    2025 arXiv

  7. [15]

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516, 2025

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Za- mani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516, 2025

    2025 arXiv

  8. [16]

    Fact, fetch, and reason: A unified evaluation of retrieval-augmented generation, 2024

    Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, and Manaal Faruqui. Fact, fetch, and reason: A unified evaluation of retrieval-augmented generation, 2024

    2024

  9. [17]

    Unifying group-relative and self-distillation policy optimization via sample routing, 2026

    Gengsheng Li, Tianyu Yang, Junfeng Fang, Mingyang Song, Mao Zheng, Haiyun Guo, Dan Zhang, Jinqiao Wang, and Tat-Seng Chua. Unifying group-relative and self-distillation policy optimization via sample routing, 2026

    2026

  10. [18]

    The tool decathlon: Benchmarking language agents for diverse, realistic, and long-horizon task execution, 2026

    Junlong Li, Wenshuo Zhao, Jian Zhao, Weihao Zeng, Haoze Wu, Xiaochen Wang, Rui Ge, Yuxuan Cao, Yuzhen Huang, Wei Liu, Junteng Liu, Zhaochen Su, Yiyang Guo, Fan Zhou, Lueyang Zhang, Juan Michelini, Xingyao Wang, Xiang Yue, Shuyan Zhou, Graham Neubig, and Junxian He. The tool de...

    2026

  11. [19]

    Websailor-v2: Bridging the chasm to proprietary agents via synthetic data and scalable reinforcement learning, 2025

    Kuan Li, Zhongwang Zhang, Huifeng Yin, Rui Ye, Yida Zhao, Liwen Zhang, Litu Ou, Dingchu Zhang, Xixi Wu, Jialong Wu, Xinyu Wang, Zile Qiao, Zhen Zhang, Yong Jiang, Pengjun Xie, Fei Huang, and Jingren Zhou. Websailor-v2: Bridging the chasm to proprietary agents via synthetic dat...

    2025

  12. [20]

    Websailor: Navigating super-human reasoning for web agent, 2025

    Kuan Li, Zhongwang Zhang, Huifeng Yin, Liwen Zhang, Litu Ou, Jialong Wu, Wenbiao Yin, Baixuan Li, Zhengwei Tao, Xinyu Wang, Weizhou Shen, Junkai Zhang, Dingchu Zhang, Xixi Wu, Yong Jiang, Ming Yan, Pengjun Xie, Fei Huang, and Jingren Zhou. Websailor: Navigating super-human rea...

    2025

  13. [21]

    Chain-of-agents: End-to-end agent foundation models via multi-agent distillation and agentic rl, 2025

    Weizhen Li, Jianbo Lin, Zhuosong Jiang, Jingyi Cao, Xinpeng Liu, Jiayu Zhang, Zhenqiang Huang, Qianben Chen, Weichen Sun, Qiexiang Wang, Hongxuan Lu, Tianrui Qin, Chenghao Zhu, Yi Yao, Shuying Fan, Xiaowan Li, Tiannan Wang, Pai Liu, King Zhu, He Zhu, Dingfeng Shi, Piaohong Wan...

    2025

  14. [22]

    Webthinker: Empowering large reasoning models with deep research capability, 2025

    Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yongkang Wu, Ji-Rong Wen, Yutao Zhu, and Zhicheng Dou. Webthinker: Empowering large reasoning models with deep research capability, 2025

    2025

  15. [23]

    Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016, 2026

    Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan-ang Gao, Wenkai Yang, Zhiyuan Liu, et al. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016, 2026

    2026 arXiv

  16. [24]

    Code-r1: Reproducing r1 for code with reliable rewards

    Jiawei Liu and Lingming Zhang. Code-r1: Reproducing r1 for code with reliable rewards. 2025

    2025

  17. [25]

    Synlogic: Synthesizing verifiable reasoning data at scale for learning logical reasoning and beyond, 2025

    Junteng Liu, Yuanxiang Fan, Zhuo Jiang, Han Ding, Yongyi Hu, Chi Zhang, Yiqi Shi, Shitong Weng, Aili Chen, Shiqi Chen, Yunan Huang, Mozhi Zhang, Pengyu Zhao, Junjie Yan, and Junxian He. Synlogic: Synthesizing verifiable reasoning data at scale for learning logical reasoning an...

    2025

  18. [26]

    Webexplorer: Explore and evolve for training long-horizon web agents, 2025

    Junteng Liu, Yunji Li, Chi Zhang, Jingyang Li, Aili Chen, Ke Ji, Weiyu Cheng, Zijia Wu, Chengyu Du, Qidi Xu, Jiayuan Song, Zhengmao Zhu, Wenhu Chen, Pengyu Zhao, and Junxian He. Webexplorer: Explore and evolve for training long-horizon web agents, 2025

    2025

  19. [27]

    On-policy distillation.Thinking Machines Lab: Connec- tionism, 2025

    Kevin Lu and Thinking Machines Lab. On-policy distillation.Thinking Machines Lab: Connec- tionism, 2025. https://thinkingmachines.ai/blog/on-policy-distillation

    2025

  20. [28]

    Deepdive: Advancing deep search agents with knowledge graphs and multi-turn rl, 2025

    Rui Lu, Zhenyu Hou, Zihan Wang, Hanchen Zhang, Xiao Liu, Yujiang Li, Shi Feng, Jie Tang, and Yuxiao Dong. Deepdive: Advancing deep search agents with knowledge graphs and multi-turn rl, 2025

    2025

  21. [29]

    Gaia: a benchmark for general ai assistants, 2023

    Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants, 2023

    2023

  22. [30]

    Minimax m2.7: Early echoes of self-evolution, 2026

    MiniMax. Minimax m2.7: Early echoes of self-evolution, 2026

    2026

  23. [31]

    Gpt-5-nano

    OpenAI. Gpt-5-nano. https://developers.openai.com/api/docs/models/ gpt-5-nano, 2025

    2025

  24. [32]

    Introducing gpt-oss

    OpenAI. Introducing gpt-oss. https://openai.com/index/introducing-gpt-oss/ , 2025

    2025

  25. [33]

    Introducing gpt -5.5

    OpenAI. Introducing gpt -5.5. https://openai.com/index/introducing-gpt-5-5/ , 2026

    2026

  26. [34]

    Qwen3.6-Plus: Towards real world agents, April 2026

    Qwen Team. Qwen3.6-Plus: Towards real world agents, April 2026

    2026

  27. [35]

    Crisp: Compressed reasoning via iterative self-policy distillation, 2026

    Hejian Sang, Yuanda Xu, Zhengze Zhou, Ran He, Zhipeng Wang, and Jiachen Sun. Crisp: Compressed reasoning via iterative self-policy distillation, 2026. 12

    2026

  28. [36]

    Generalization in generation: A closer look at exposure bias

    Florian Schmidt. Generalization in generation: A closer look at exposure bias. InProceedings of the 3rd Workshop on Neural Generation and Translation, pages 157–167. Association for Computational Linguistics, 2019

    2019

  29. [37]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024

    2024

  30. [38]

    Self-distillation enables continual learning, 2026

    Idan Shenfeld, Mehul Damani, Jonas Hübotter, and Pulkit Agrawal. Self-distillation enables continual learning, 2026

    2026

  31. [39]

    A survey of on-policy distillation for large language models, 2026

    Mingyang Song and Mao Zheng. A survey of on-policy distillation for large language models, 2026

    2026

  32. [40]

    Webshaper: Agentically data synthesizing via information-seeking formalization, 2025

    Zhengwei Tao, Jialong Wu, Wenbiao Yin, Junkai Zhang, Baixuan Li, Haiyang Shen, Kuan Li, Liwen Zhang, Xinyu Wang, Yong Jiang, Pengjun Xie, Fei Huang, and Jingren Zhou. Webshaper: Agentically data synthesizing via information-seeking formalization, 2025

    2025

  33. [41]

    Kimi k2.6: Advancing open-source coding

    Kimi Team. Kimi k2.6: Advancing open-source coding. https://www.kimi.com/blog/ kimi-k2-6, 2026

    2026

  34. [42]

    Mirothinker: Pushing the performance boundaries of open-source research agents via model, context, and interactive scaling.arXiv preprint arXiv:2511.11793, 2025

    MiroMind Team, Song Bai, Lidong Bing, Carson Chen, Guanzheng Chen, Yuntao Chen, Zhe Chen, Ziyi Chen, Xuan Dong, et al. Mirothinker: Pushing the performance boundaries of open-source research agents via model, context, and interactive scaling.arXiv preprint arXiv:2511.11793, 2025

    2025 arXiv

  35. [43]

    Browsecomp: A simple yet challenging benchmark for browsing agents, 2025

    Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents, 2025

    2025

  36. [44]

    Chain-of-thought prompting elicits reasoning in large language models, 2023

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023

    2023

  37. [45]

    Smartsearch: Process reward-guided query refinement for search agents, 2026

    Tongyu Wen, Guanting Dong, and Zhicheng Dou. Smartsearch: Process reward-guided query refinement for search agents, 2026

    2026

  38. [46]

    Mirage or method? how model-task alignment induces divergent rl conclusions, 2025

    Haoze Wu, Cheng Wang, Wenshuo Zhao, and Junxian He. Mirage or method? how model-task alignment induces divergent rl conclusions, 2025

    2025

  39. [47]

    Recode: Updating code api knowledge with reinforcement learning, 2025

    Haoze Wu, Yunzhi Yao, Wenhao Yu, and Ningyu Zhang. Recode: Updating code api knowledge with reinforcement learning, 2025

    2025

  40. [48]

    On the generalization of sft: A reinforcement learning perspective with reward rectification, 2026

    Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng, Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Ming-Hsuan Yang, and Xu Yang. On the generalization of sft: A reinforcement learning perspective with reward rectification, 2026

    2026

  41. [49]

    Grok 4.1 fast and agent tools api.https://x.ai/news/grok-4-1-fast, 2025

    X.AI. Grok 4.1 fast and agent tools api.https://x.ai/news/grok-4-1-fast, 2025

    2025

  42. [50]

    Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning, 2025

    Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning, 2025

    2025

  43. [51]

    Principle process reward for search agents, 2026

    Peiran Xu, Zhuohao Li, Xiaoying Xing, Guannan Zhang, Debiao Li, and Kunyu Shi. Principle process reward for search agents, 2026

    2026

  44. [52]

    Tip: Token importance in on-policy distillation, 2026

    Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, and Alborz Geramifard. Tip: Token importance in on-policy distillation, 2026

    2026

  45. [53]

    Qwen3 technical report, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, et al. Qwen3 technical report, 2025. 13

    2025

  46. [54]

    Self-distilled rlvr, 2026

    Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, and Nan Duan. Self-distilled rlvr, 2026

    2026

  47. [55]

    Cohen, Ruslan Salakhut- dinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhut- dinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering, 2018

    2018

  48. [56]

    React: Synergizing reasoning and acting in language models, 2023

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023

    2023

  49. [57]

    On-policy context distillation for language models, 2026

    Tianzhu Ye, Li Dong, Xun Wu, Shaohan Huang, and Furu Wei. On-policy context distillation for language models, 2026

    2026

  50. [58]

    Dapo: An open-source llm reinforcement learning system at scale, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen...

    2025

  51. [59]

    Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild

    Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. InSecond Conference on Language Modeling, 2025

    2025

  52. [60]

    The landscape of agentic reinforcement learning for LLMs: A survey.Transactions on Machine Learning Research,

    Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhong-Zhi Li, Xiangyuan Xue, Yijiang Li, Yifan Zhou, Yang Chen, Chen Zhang, Yutao Fan, Zihu Wang, Songtao Huang, Francisco Piedrahita Velez, Yue Liao, Hongru W ANG, Mengyue Yang, Heng Ji, J...

  53. [61]

    Tool-r1: Sample-efficient reinforcement learning for agentic tool use, 2025

    Yabo Zhang, Yihan Zeng, Qingyun Li, Zhen Hu, Kavin Han, and Wangmeng Zuo. Tool-r1: Sample-efficient reinforcement learning for agentic tool use, 2025

    2025

  54. [62]

    Criticsearch: Fine-grained credit assignment for search agents via a retrospective critic, 2025

    Yaocheng Zhang, Haohuan Huang, Zijun Song, Yuanheng Zhu, Qichao Zhang, Zijie Zhao, and Dongbin Zhao. Criticsearch: Fine-grained credit assignment for search agents via a retrospective critic, 2025

    2025

  55. [63]

    Self-distilled reasoner: On-policy self-distillation for large language models, 2026

    Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models, 2026

    2026

  56. [64]

    aha moments

    Yuhang Zhou, Kai Zheng, Qiguang Chen, Mengkang Hu, Qingfeng Sun, Can Xu, and Jingjing Chen. Offseeker: Online reinforcement learning is not all you need for deep research agents, 2026. 14 A Related Work A.1 Reinforcement Learning with Verifiable Rewards (RLVR) Building on work...

    2026

  57. [66]

    **Remove highly specific clues**: delete dates, numbers, full names, institutions, awards, locations, publication names, etc.; keep only the minimal signals needed for unique identification

  58. [67]

    **Blur and generalize**: replace concrete information with uncertain or approximate descriptions, while keeping the truth uniquely resolvable

  59. [68]

    **Add distractor-like similar entities**: introduce misleading cues so shallow reasoning fails and deeper inference is required

  60. [69]

    **Refer indirectly to the entity**: use less-common identifiers--- predecessor/successor relationships, indirect influence, obscure nicknames, abstract impact, associated figures, etc

  61. [70]

    **Iterative escalation**: perform **5 evolution steps**; each step must remove or obscure at least one previously clear attribute, and be ** strictly harder** than the previous version

  62. [71]

    Ensure **uniqueness of the final truth**: despite the ambiguity, the question must still map to the exact same truth You may use search and browsing tools to verify uniqueness during rewriting. --- Output format: For each iteration: <question> {more difficult question version}...

  63. [72]

    DECOMPOSE: Break the [Question] into independent, verifiable conditions

  64. [73]

    who is the COOP leader in Amarillo

    ANCHORING SEARCH: For each condition, your search query MUST include the [ Standard Answer] or be related to the [Standard Answer]. - BAD Search: "who is the COOP leader in Amarillo" (This is solving) - GOOD Search: "Angela Margrave National Weather Service Amarillo COOP leade...

  65. [74]

    PROVE SATISFACTION: A condition is only ’Verified’ if you find a source that explicitly links the [Standard Answer] to that specific requirement. OUTPUT FORMAT (Strict XML): <evidences> <evidence> <condition>The specific requirement/condition extracted from the [Question ]</co...

  66. [2026]

    Survey Certification

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.