Pith. sign in

REVIEW 4 major objections 3 minor 5 cited by

LLMs can act as their own search engine when training RL agents, the paper claims.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

SSRL, a training pipeline that uses an LLM's own repeated sampling as a search environment for RL, improves question answering without external tools and transfers to real search engines.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Plausible method, overreaching abstract: sim-to-real claim needs reward-alignment evidence that isn't shown here. the 4 major comments →

arxiv 2508.10874 v1 pith:IRBHERLC submitted 2025-08-14 cs.CL

SSRL: Self-Search Reinforcement Learning

Yuchen Fan , Kaiyan Zhang , Heng Zhou , Yuxin Zuo , Yanxu Chen , Yu Fu , Xinwei Long , Xuekai Zhu
show 10 more authors
Che Jiang Yuchen Zhang Li Kang Gang Chen Cheng Huang Zhizhou He Bingning Wang Lei Bai Ning Ding Bowen Zhou
This is my paper
classification cs.CL
keywords self-searchreinforcement learningLLM agentssim-to-real transferpass@k scalinghallucination reductionBrowseCompsearch-engine-free RL
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a large language model can double as its own search engine during reinforcement learning, so that training a search-agent policy no longer requires expensive calls to an external search engine. It first measures how well LLMs answer hard questions by repeatedly sampling structured prompts, a behavior it calls Self-Search, and finds performance climbs steadily with the inference budget, with high pass@k on tasks like BrowseComp. It then trains policies with SSRL, using format-based and rule-based rewards on the model's own samples, and reports that the resulting policies are a stable, cheap environment for RL and that they integrate with real external search engines without extra fine-tuning. If correct, this would lower the cost of training RL agents for web-scale tasks and turn internal knowledge retrieval into a trainable skill.

Core claim

The central claim is that the LLM itself is a sufficient simulator for search-driven RL. The authors measure intrinsic search ability by prompting the model to produce a structured search record (queries and reasoning) and sampling many times, defining Self-Search; pass@k grows with inference budget, reaching high scores on BrowseComp. They then use those internal self-search trajectories as the RL environment, rewarding format compliance and rule-based outcomes instead of external retrieval results. The reported finding is that SSRL-trained policies improve in tool-free question answering, exhibit less hallucination, and—critically—transfer to real external search engines without additional

What carries the argument

Self-Search RL (SSRL): using the LLM's own repeated-sampling search trajectories as the environment for policy-gradient training, with format-based and rule-based rewards substituting for external search-engine result quality. The Self-Search measurement (structured prompting plus repeated sampling, scored as pass@k) is the mechanism that makes the environment feasible: it quantifies how much correctness scales with inference budget and supplies the reward signal for RL.

Load-bearing premise

The rewards based on format and internal rules capture the same notion of correctness as a real search engine, so that a policy trained in the internal environment still improves when attached to the external one.

What would settle it

Compare SSRL-trained policies with rule rewards against policies trained with outcome rewards from a live search engine on the same prompts, then evaluate both with real web search on unseen questions. If the rule-rewarded policy underperforms the outcome-rewarded one, the internal environment is not aligned with external correctness and sim-to-real transfer would fail.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Search-agent RL training can be run without live search-engine APIs, cutting cost and avoiding rate limits.
  • Higher inference budget at training or inference time should yield better self-search performance in a predictable, scaling-like way.
  • Policies trained only on internal self-search can be paired with an external search engine at deployment with no extra training, retaining the gains.
  • Training to search the model's own memory is also a route to reducing hallucination, since the model learns to consult its own knowledge before answering.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the reward-alignment assumption holds, the same self-simulation trick could extend to other tools, such as code interpreters or database interfaces, where a model's own sampled outputs stand in for tool feedback.
  • The paper's scaling result suggests pass@k can be treated as a controllable resource: a deployer could trade inference budget for accuracy without retraining, a property the paper observes but does not fully exploit.
  • A strong test of the sim-to-real claim would be to compare rule-rewarded self-search against outcome-rewarded training on a live search engine; if the rule-rewarded policy wins or ties, the internal environment is genuinely aligned with external correctness.
  • The reduction in hallucination claim implies that RL against internal search rewards may be acting as a form of self-consistency training, which could generalize to other consistency-based objectives.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper investigates whether LLMs can serve as efficient simulators for agentic search tasks in reinforcement learning, reducing dependence on costly interactions with external search engines. It first proposes 'Self-Search' — eliciting intrinsic search capability via structured prompting and repeated sampling — and reports that LLMs exhibit strong scaling with inference budget, achieving high pass@k on QA benchmarks including BrowseComp. It then introduces Self-Search RL (SSRL), which trains policies with format-based and rule-based rewards so that the model refines its internal knowledge without external tools. The abstract claims SSRL-trained policies form a cost-effective and stable RL environment and facilitate robust sim-to-real transfer to real search engines. The stated conclusions are that LLMs' world knowledge can be elicited, that SSRL can reduce hallucination, and that SSRL-trained models integrate with external search engines without additional effort.

Significance. If all claims are substantiated, the contribution would be practically significant: replacing external search engines with the LLM itself as the RL environment would substantially lower the cost of training search agents, and the claimed sim-to-real transfer would make such training directly usable. The evaluation is anchored to an external benchmark, BrowseComp, which is a reasonable external check. However, the abstract provides no quantitative magnitudes, baselines, error bars, or ablations, and the sim-to-real transfer claim rests on an unvalidated assumption that internal format/rule rewards align with external search correctness. As presented, the work is a promising research direction rather than a demonstrated result.

major comments (4)
  1. [Abstract] The central quantitative claims — 'high pass@k', 'cost-effective and stable environment', and 'robust sim-to-real transfer' — are stated without any supporting magnitudes, baselines, error bars, or ablations. For a journal submission, the abstract should report at least the key numbers (e.g., pass@k on BrowseComp, comparison to a real-search-engine-trained policy, and variance across seeds/tasks). Without these, the paper's main claims cannot be assessed.
  2. [Abstract, sim-to-real claim] The training rewards are described as 'format-based and rule-based', computed on the model's own samples. The external correctness of a search trajectory is not shown to be correlated with these internal rewards. This is the load-bearing alignment assumption behind 'facilitating robust sim-to-real transfer'. The paper needs either (a) a direct correlation analysis between internal reward and external outcome quality, or (b) an ablation comparing SSRL against the same policy trained with rewards from a real search engine, plus an explicit transfer evaluation. The abstract currently provides no evidence for this alignment.
  3. [Abstract, 'high pass@k'] pass@k with structured prompting and repeated sampling measures the coverage of the sampling distribution, not the expected reward of the optimized policy after RL. The inference-budget scaling result does not by itself imply that SSRL increases the probability of generating correct answers; the format/rule-based rewards may instead increase format compliance or self-consistency. The paper should report pass@1 or mean reward before and after SSRL, and ideally the distribution of correctness probabilities, to distinguish true capability gains from proxy optimization.
  4. [Abstract, 'cost-effective and stable'] No definitions or measurements are given for 'cost-effective' or 'stable'. Cost should be quantified (e.g., tokens, wall-clock, or API calls per training episode) and stability should be quantified (e.g., reward variance or success-rate variance across seeds/tasks). Without such metrics, the comparison to external search engines is not meaningful.
minor comments (3)
  1. [Abstract] The terms 'Self-Search' and 'sim-to-real' are used without definition. 'Sim-to-real' is borrowed from robotics and should be explicitly defined in the context of search-agent training.
  2. [Abstract] The phrase 'we draw the following conclusions' followed by three bullet-like conclusions is unusual for an abstract; consider moving detailed conclusions to the introduction and keeping the abstract outcome-focused.
  3. [Overall] The manuscript should position itself against existing work on self-rewarding/self-improving LLMs and internal-knowledge elicitation; no related work is visible in the provided text.

Circularity Check

0 steps flagged

No significant circularity; SSRL's claims are anchored to external benchmarks and the central sim-to-real concern is an external-validity gap, not a circular reduction.

full rationale

The abstract's derivation chain is not circular by any of the enumerated patterns. The initial measurement of 'Self-Search' is defined operationally as 'structured prompting and repeated sampling' and is evaluated against an external benchmark ('high pass@k on question-answering benchmarks, including the challenging BrowseComp task'). This is an external falsifiability anchor, not a self-defined metric. SSRL then trains with 'format-based and rule-based rewards'—these are distinct from the pass@k metric itself, and the paper does not claim the rewards are defined in terms of the evaluation outcome. There are no visible fitted parameters that are renamed as predictions, no self-citation chain, and no uniqueness theorem imported from the authors. The 'sim-to-real transfer' claim ('facilitating robust sim-to-real transfer') is indeed an unsupported empirical assumption in the abstract—the internal reward alignment with external search correctness is not demonstrated—but absence of evidence is an external-validity or correctness risk, not circularity under the hard rules. The term 'Self-Search' repackages known sampling-based techniques, but the paper explicitly discloses the components ('structured prompting and repeated sampling'), so it is not a disguised renaming of a known result. Under the given definition of circularity, this paper warrants a score of 0.

Axiom & Free-Parameter Ledger

2 free parameters · 2 axioms · 0 invented entities

The method rests on two unstated domain assumptions (sufficient internal knowledge; reward-answer alignment) and two undisclosed control values (sampling budget k, reward thresholds). The abstract provides no numbers for either.

free parameters (2)
  • pass@k inference budget k
    Number of samples per query controls the self-search scaling curves; value chosen by authors and not reported in the abstract.
  • rule-based reward thresholds and format constraints
    Define the SSRL training signal; whether they encode true answer correctness rather than self-consistency is not disclosed in the abstract.
axioms (2)
  • domain assumption LLM parametric knowledge covers a large fraction of the target question-answering space
    Tool-free self-search can only surface answers already present in the weights; the whole approach presupposes this, stated as conclusion 1 of the abstract.
  • domain assumption Format-based and rule-based rewards track genuine answer correctness
    SSRL optimizes these rewards; if they proxy self-consistency rather than truth, the hallucination-reduction and sim-to-real claims do not follow. Reward design is asserted, not derived.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of SSRL: Self-Search Reinforcement Learning." pith.science (2026). https://pith.science/paper/IRBHERLC

@misc{pith2026250810874,
  author       = {Pith},
  title        = {Pith review of: SSRL: Self-Search Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IRBHERLC}},
  note         = {Machine review of arXiv:2508.10874}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We investigate the potential of large language models (LLMs) to serve as efficient simulators for agentic search tasks in reinforcement learning (RL), thereby reducing dependence on costly interactions with external search engines. To this end, we first quantify the intrinsic search capability of LLMs via structured prompting and repeated sampling, which we term Self-Search. Our results reveal that LLMs exhibit strong scaling behavior with respect to the inference budget, achieving high pass@k on question-answering benchmarks, including the challenging BrowseComp task. Building on these observations, we introduce Self-Search RL (SSRL), which enhances LLMs' Self-Search capability through format-based and rule-based rewards. SSRL enables models to iteratively refine their knowledge utilization internally, without requiring access to external tools. Empirical evaluations demonstrate that SSRL-trained policy models provide a cost-effective and stable environment for search-driven RL training, reducing reliance on external search engines and facilitating robust sim-to-real transfer. We draw the following conclusions: 1) LLMs possess world knowledge that can be effectively elicited to achieve high performance; 2) SSRL demonstrates the potential of leveraging internal knowledge to reduce hallucination; 3) SSRL-trained models integrate seamlessly with external search engines without additional effort. Our findings highlight the potential of LLMs to support more scalable RL agent training.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Qwen-AgentWorld: Language World Models for General Agents

    cs.CL 2026-06 unverdicted novelty 6.0

    Qwen-AgentWorld are language world models that simulate multi-domain agent environments and boost general agent capabilities via decoupled RL simulation and unified foundation model training.

  2. The Landscape of Agentic Reinforcement Learning for LLMs: A Survey

    cs.AI 2025-09 accept novelty 6.0

    Survey that defines agentic RL for LLMs via POMDPs, introduces a taxonomy of planning/tool-use/memory/reasoning capabilities and domains, and compiles open environments from over 500 papers.

  3. Dynamic Collaboration of Multi-Language Models based on Minimal Complete Semantic Units

    cs.AI 2025-08 conditional novelty 6.0

    MCSU-based vocabulary alignment plus distance-based dynamic selection (DDS) lets several LLMs vote token-by-token, beating single models and prior ensemble baselines on multiple reasoning benchmarks without training.

  4. Erase to Improve: Erasable Reinforcement Learning for Search-Augmented LLMs

    cs.CL 2025-10 unverdicted novelty 5.0

    ERL trains LLMs to erase faulty reasoning steps and regenerate them in place, yielding gains of up to 8.48% EM on multi-hop QA benchmarks like HotpotQA.

  5. A Survey of Reinforcement Learning for Large Reasoning Models

    cs.CL 2025-09 accept novelty 3.0

    A survey compiling RL methods, challenges, data resources, and applications for enhancing reasoning in large language models and large reasoning models since DeepSeek-R1.

Reference graph

Works this paper leans on

72 extracted references · 8 canonical work pages · cited by 5 Pith papers · 2 internal anchors

  1. [1]

    Le, Christopher Ré, and Azalia Mirhoseini

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling, 2024. URL https://arxiv.org/abs/2407.21787

    Pith/arXiv arXiv 2024

  2. [2]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  3. [3]

    Pan, Wen Zhang, Huajun Chen, Fan Yang, Zenan Zhou, and Weipeng Chen

    Mingyang Chen, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z. Pan, Wen Zhang, Huajun Chen, Fan Yang, Zenan Zhou, and Weipeng Chen. Research: Learning to reason with search for llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2503.19470

    Pith/arXiv arXiv 2025

  4. [4]

    Process reinforcement through implicit rewards

    Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456, 2025 a

    Pith/arXiv arXiv 2025

  5. [5]

    The entropy mechanism of reinforcement learning for reasoning language models

    Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617, 2025 b

    Pith/arXiv arXiv 2025

  6. [6]

    A survey of sim-to-real methods in rl: Progress, prospects and challenges with foundation models, 2025

    Longchao Da, Justin Turnau, Thirulogasankar Pranav Kutralingam, Alvaro Velasquez, Paulo Shakarian, and Hua Wei. A survey of sim-to-real methods in rl: Progress, prospects and challenges with foundation models, 2025. URL https://arxiv.org/abs/2502.13187

    Pith/arXiv arXiv 2025

  7. [7]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2501.12948

    Pith/arXiv arXiv 2025

  8. [8]

    Citations and trust in llm generated responses

    Yifan Ding, Matthew Facciani, Ellen Joyce, Amrit Poudel, Sanmitra Bhattacharya, Balaji Veeramani, Sal Aguinaga, and Tim Weninger. Citations and trust in llm generated responses. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp.\ 23787--23795, 2025

    work page 2025

  9. [9]

    Competitive programming with large reasoning models

    Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, et al. Competitive programming with large reasoning models. arXiv preprint arXiv:2502.06807, 2025

    Pith/arXiv arXiv 2025

  10. [10]

    Retool: Reinforcement learning for strategic tool use in llms, 2025

    Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms, 2025. URL https://arxiv.org/abs/2504.11536

    Pith/arXiv arXiv 2025

  11. [11]

    Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D. Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars, 2025. URL https://arxiv.org/abs/2503.01307

    Pith/arXiv arXiv 2025

  12. [12]

    A survey of self-evolving agents: On path to artificial super intelligence

    Huan-ang Gao, Jiayi Geng, Wenyue Hua, Mengkang Hu, Xinzhe Juan, Hongzhang Liu, Shilong Liu, Jiahao Qiu, Xuan Qi, Yiran Wu, et al. A survey of self-evolving agents: On path to artificial super intelligence. arXiv preprint arXiv:2507.21046, 2025

    Pith/arXiv arXiv 2025

  13. [13]

    Enabling large language models to generate text with citations

    Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models to generate text with citations. arXiv preprint arXiv:2305.14627, 2023

    Pith/arXiv arXiv 2023

  14. [14]

    The llama 3 herd of models, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, and et al. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

    Pith/arXiv arXiv 2024

  15. [15]

    Is your llm secretly a world model of the internet? model-based planning for web agents

    Yu Gu, Kai Zhang, Yuting Ning, Boyuan Zheng, Boyu Gou, Tianci Xue, Cheng Chang, Sanjari Srivastava, Yanan Xie, Peng Qi, et al. Is your llm secretly a world model of the internet? model-based planning for web agents. arXiv preprint arXiv:2411.06559, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    Reasoning with language model is planning with world model

    Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992, 2023

    Pith/arXiv arXiv 2023

  17. [17]

    Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps, 2020

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps, 2020. URL https://arxiv.org/abs/2011.01060

    Pith/arXiv arXiv 2020

  18. [18]

    Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models, 2025

    Jian Hu, Jason Klein Liu, and Wei Shen. Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models, 2025. URL https://arxiv.org/abs/2501.03262

    Pith/arXiv arXiv 2025

  19. [19]

    Arik, and Jiawei Han

    Bowen Jin, Jinsung Yoon, Priyanka Kargupta, Sercan O. Arik, and Jiawei Han. An empirical study on reinforcement learning for reasoning-search interleaved llm agents, 2025 a . URL https://arxiv.org/abs/2505.15117

    Pith/arXiv arXiv 2025

  20. [20]

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516, 2025 b

    Pith/arXiv arXiv 2025

  21. [21]

    Weld, and Luke Zettlemoyer

    Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, 2017. URL https://arxiv.org/abs/1705.03551

    Pith/arXiv arXiv 2017

  22. [22]

    Sim2real transfer for reinforcement learning without dynamics randomization

    Manuel Kaspar, Juan D Mu \ n oz Osorio, and J \"u rgen Bock. Sim2real transfer for reinforcement learning without dynamics randomization. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp.\ 4383--4388. IEEE, 2020

    work page 2020

  23. [23]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transac...

    doi:10.1162/tacl_a_00276 2019

  24. [24]

    Scalable agent alignment via reward modeling: a research direction

    Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871, 2018

    Pith/arXiv arXiv 2018

  25. [25]

    Gonzalez, and Ion Stoica

    Dacheng Li, Shiyi Cao, Chengkun Cao, Xiuyu Li, Shangyin Tan, Kurt Keutzer, Jiarong Xing, Joseph E. Gonzalez, and Ion Stoica. S*: Test time scaling for code generation, 2025 a . URL https://arxiv.org/abs/2502.14382

    Pith/arXiv arXiv 2025

  26. [26]

    Emergent world representations: Exploring a sequence model trained on a synthetic task

    Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Vi \'e gas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on a synthetic task. ICLR, 2023

    work page 2023

  27. [27]

    CorpusLM: Towards a Unified Language Model on Corpus for Knowledge-Intensive Tasks

    Xiaoxi Li, Zhicheng Dou, Yujia Zhou, and Fangchao Liu. Corpuslm: Towards a unified language model on corpus for knowledge-intensive tasks, 2024 a . URL https://arxiv.org/abs/2402.01176

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  28. [28]

    Search-o1: Agentic search-enhanced large reasoning models, 2025 b

    Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-o1: Agentic search-enhanced large reasoning models, 2025 b . URL https://arxiv.org/abs/2501.05366

    Pith/arXiv arXiv 2025

  29. [29]

    From matching to generation: A survey on generative information retrieval

    Xiaoxi Li, Jiajie Jin, Yujia Zhou, Yuyao Zhang, Peitian Zhang, Yutao Zhu, and Zhicheng Dou. From matching to generation: A survey on generative information retrieval. ACM Transactions on Information Systems, 43 0 (3): 0 1--62, 2025 c

    work page 2025

  30. [30]

    A survey of generative search and recommendation in the era of large language models

    Yongqi Li, Xinyu Lin, Wenjie Wang, Fuli Feng, Liang Pang, Wenjie Li, Liqiang Nie, Xiangnan He, and Tat-Seng Chua. A survey of generative search and recommendation in the era of large language models. arXiv preprint arXiv:2404.16924, 2024 b

    Pith/arXiv arXiv 2024

  31. [31]

    Learning to rank in generative retrieval

    Yongqi Li, Nan Yang, Liang Wang, Furu Wei, and Wenjie Li. Learning to rank in generative retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 8716--8723, 2024 c

    work page 2024

  32. [32]

    McIlraith, and Yilun Du

    Shalev Lifshitz, Sheila A. McIlraith, and Yilun Du. Multi-agent verification: Scaling test-time compute with multiple verifiers, 2025. URL https://arxiv.org/abs/2502.20379

    Pith/arXiv arXiv 2025

  33. [33]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  34. [34]

    Can 1b llm surpass 405b llm? rethinking compute-optimal test-time scaling, 2025 a

    Runze Liu, Junqi Gao, Jian Zhao, Kaiyan Zhang, Xiu Li, Biqing Qi, Wanli Ouyang, and Bowen Zhou. Can 1b llm surpass 405b llm? rethinking compute-optimal test-time scaling, 2025 a . URL https://arxiv.org/abs/2502.06703

    Pith/arXiv arXiv 2025

  35. [35]

    Understanding r1-zero-like training: A critical perspective, 2025 b

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective, 2025 b . URL https://arxiv.org/abs/2503.20783

    Pith/arXiv arXiv 2025

  36. [36]

    Generative multi-modal knowledge retrieval with large language models

    Xinwei Long, Jiali Zeng, Fandong Meng, Zhiyuan Ma, Kaiyan Zhang, Bowen Zhou, and Jie Zhou. Generative multi-modal knowledge retrieval with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 18733--18741, 2024

    2024

  37. [37]

    Openai o1 system card, 2024

    OpenAI. Openai o1 system card, 2024. URL https://arxiv.org/abs/2412.16720

    Pith/arXiv arXiv 2024

  38. [38]

    Proof or bluff? evaluating llms on 2025 usa math olympiad

    Ivo Petrov, Jasper Dekoninck, Lyuben Baltadzhiev, Maria Drencheva, Kristian Minchev, Mislav Balunovi \'c , Nikola Jovanovi \'c , and Martin Vechev. Proof or bluff? evaluating llms on 2025 usa math olympiad. arXiv preprint arXiv:2503.21934, 2025

    Pith/arXiv arXiv 2025

  39. [39]

    Smith, and Mike Lewis

    Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models, 2023. URL https://arxiv.org/abs/2210.03350

    Pith/arXiv arXiv 2023

  40. [40]

    Toolrl: Reward is all tool learning needs

    Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-T \"u r, Gokhan Tur, and Heng Ji. Toolrl: Reward is all tool learning needs. arXiv preprint arXiv:2504.13958, 2025

    Pith/arXiv arXiv 2025

  41. [41]

    WebCPM: Interactive Web Search for Chinese Long-form Question Answering

    Yujia Qin, Zihan Cai, Dian Jin, Lan Yan, Shihao Liang, Kunlun Zhu, Yankai Lin, Xu Han, Ning Ding, Huadong Wang, Ruobing Xie, Fanchao Qi, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Webcpm: Interactive web search for chinese long-form question answering, 2023. URL https://arxiv.org/abs/2305.06849

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  42. [42]

    Treebon: Enhancing inference-time alignment with speculative tree-search and best-of-n sampling, 2024

    Jiahao Qiu, Yifu Lu, Yifan Zeng, Jiacheng Guo, Jiayi Geng, Huazheng Wang, Kaixuan Huang, Yue Wu, and Mengdi Wang. Treebon: Enhancing inference-time alignment with speculative tree-search and best-of-n sampling, 2024. URL https://arxiv.org/abs/2410.16033

    Pith/arXiv arXiv 2024

  43. [43]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and et al. Qwen2.5 technical report, 2025. URL https://arxiv.org/abs/2412.15115

    Pith/arXiv arXiv 2025

  44. [44]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347

    Pith/arXiv arXiv 2017

  45. [45]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300

    Pith/arXiv arXiv 2024

  46. [46]

    Thinking vs

    Junhong Shen, Hao Bai, Lunjun Zhang, Yifei Zhou, Amrith Setlur, Shengbang Tong, Diego Caples, Nan Jiang, Tong Zhang, Ameet Talwalkar, and Aviral Kumar. Thinking vs. doing: Agents that reason by scaling test-time interaction, 2025. URL https://arxiv.org/abs/2506.07976

    Pith/arXiv arXiv 2025

  47. [47]

    Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face, 2023

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face, 2023. URL https://arxiv.org/abs/2303.17580

    Pith/arXiv arXiv 2023

  48. [48]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024. URL https://arxiv.org/abs/2408.03314

    Pith/arXiv arXiv 2024

  49. [49]

    Zerosearch: Incentivize the search capability of llms without searching, 2025

    Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Yan Zhang, Fei Huang, and Jingren Zhou. Zerosearch: Incentivize the search capability of llms without searching, 2025. URL https://arxiv.org/abs/2505.04588

    Pith/arXiv arXiv 2025

  50. [50]

    Worldcoder, a model-based llm agent: Building world models by writing code and interacting with the environment

    Hao Tang, Darren Key, and Kevin Ellis. Worldcoder, a model-based llm agent: Building world models by writing code and interacting with the environment. Advances in Neural Information Processing Systems, 37: 0 70148--70212, 2024

    work page 2024

  51. [51]

    Transformer memory as a differentiable search index

    Yi Tay, Vinh Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, et al. Transformer memory as a differentiable search index. Advances in Neural Information Processing Systems, 35: 0 21831--21843, 2022

    2022

  52. [52]

    Kimi k2: Open agentic intelligence

    Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025

    Pith/arXiv arXiv 2025

  53. [53]

    Musique: Multihop questions via single-hop question composition, 2022

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition, 2022. URL https://arxiv.org/abs/2108.00573

    Pith/arXiv arXiv 2022

  54. [54]

    A neural corpus indexer for document retrieval

    Yujing Wang, Yingyan Hou, Haonan Wang, Ziming Miao, Shibin Wu, Qi Chen, Yuqing Xia, Chengmin Chi, Guoshuai Zhao, Zheng Liu, et al. A neural corpus indexer for document retrieval. Advances in Neural Information Processing Systems, 35: 0 25600--25614, 2022

    2022

  55. [55]

    Octothinker: Mid-training incentivizes reinforcement learning scaling, 2025 a

    Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. Octothinker: Mid-training incentivizes reinforcement learning scaling, 2025 a . URL https://arxiv.org/abs/2506.20512

    Pith/arXiv arXiv 2025

  56. [56]

    Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning, 2025 b

    Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, Yiping Lu, Kyunghyun Cho, Jiajun Wu, Li Fei-Fei, Lijuan Wang, Yejin Choi, and Manling Li. Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning, 2025 b . URL https://arxiv.org/a...

    Pith/arXiv arXiv 2025

  57. [57]

    Chain-of-thought prompting elicits reasoning in large language models, 2023

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. URL https://arxiv.org/abs/2201.11903

    Pith/arXiv arXiv 2023

  58. [58]

    Measuring short-form factuality in large language models, 2024

    Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models, 2024. URL https://arxiv.org/abs/2411.04368

    Pith/arXiv arXiv 2024

  59. [59]

    Browsecomp: A simple yet challenging benchmark for browsing agents, 2025

    Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents, 2025. URL https://arxiv.org/abs/2504.12516

    Pith/arXiv arXiv 2025

  60. [60]

    Towards large reasoning models: A survey of reinforced reasoning with large language models, 2025

    Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, Chenyang Shao, Yuwei Yan, Qinglong Yang, Yiwen Song, Sijian Ren, Xinyuan Hu, Yu Li, Jie Feng, Chen Gao, and Yong Li. Towards large reasoning models: A survey of reinforced reasoning with large language models, 2025. URL ht...

    Pith/arXiv arXiv 2025

  61. [62]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025 b

    Pith/arXiv arXiv 2025

  62. [63]

    Gta1: Gui test-time scaling agent, 2025 c

    Yan Yang, Dongxu Li, Yutong Dai, Yuhao Yang, Ziyang Luo, Zirui Zhao, Zhiyuan Hu, Junzhe Huang, Amrita Saha, Zeyuan Chen, Ran Xu, Liyuan Pan, Caiming Xiong, and Junnan Li. Gta1: Gui test-time scaling agent, 2025 c . URL https://arxiv.org/abs/2507.05791

    Pith/arXiv arXiv 2025

  63. [64]

    Cohen, Ruslan Salakhutdinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering, 2018. URL https://arxiv.org/abs/1809.09600

    Pith/arXiv arXiv 2018

  64. [65]

    Dapo: An open-source llm reinforcement learning system at scale, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, and et al. Dapo: An open-source llm reinforcement learning system at scale, 2025. URL https://arxiv.org/abs/2503.14476

    Pith/arXiv arXiv 2025

  65. [66]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?, 2025

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?, 2025. URL https://arxiv.org/abs/2504.13837

    Pith/arXiv arXiv 2025

  66. [67]

    Deepresearcher: Scaling deep research via reinforcement learning in real-world environments, 2025

    Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments, 2025. URL https://arxiv.org/abs/2504.03160

    Pith/arXiv arXiv 2025

  67. [68]

    Scaling test-time compute for llm agents, 2025

    King Zhu, Hanhao Li, Siwei Wu, Tianshun Xing, Dehua Ma, Xiangru Tang, Minghao Liu, Jian Yang, Jiaheng Liu, Yuchen Eleanor Jiang, Changwang Zhang, Chenghua Lin, Jun Wang, Ge Zhang, and Wangchunshu Zhou. Scaling test-time compute for llm agents, 2025. URL https://arxiv.org/abs/2506.12928

    Pith/arXiv arXiv 2025

  68. [69]

    Ttrl: Test-time reinforcement learning

    Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, et al. Ttrl: Test-time reinforcement learning. arXiv preprint arXiv:2504.16084, 2025

    Pith/arXiv arXiv 2025

  69. [70]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  70. [71]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  71. [72]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  72. [73]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.