Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

CLEANER claims that rewriting failed tool-use steps out of reinforcement-learning trajectories — replacing them with the model's own successful corrections — turns noisy agentic training into clean signal, yielding average accuracy gains of

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 08:56 UTC pith:MPRKBPNK

load-bearing objection A genuinely useful trajectory-purification idea for agentic RL, but the GRPO estimator on counterfactual paths is not justified and the empirical shell is too thin to trust the numbers as-is. the 3 major comments →

arxiv 2601.15141 v2 pith:MPRKBPNK submitted 2026-01-21 cs.LG

CLEANER: Self-Purified Trajectories Boost Agentic Reinforcement Learning

Tianshi Xu , Yuteng Chen , Meng Li This is my paper
classification cs.LG
keywords agentic reinforcement learningtrajectory purificationself-correctionsimilarity-aware adaptive rollbackGRPOtool usecode interpreterLLM reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper is trying to establish that the main obstacle to reinforcement learning for small tool-using language models is not missing rewards or exploration strategy, but the accumulation of execution-error context inside training trajectories. It argues that failed code and tracebacks contaminate the model's context and receive spurious positive credit in outcome-only RL, so defeating that noise matters more than adding dense rewards or supersampling. The proposed solution, CLEANER, uses a Similarity-Aware Adaptive Rollback (SAAR) mechanism to detect when the model has corrected its own code within a rollout and rewrite the history to show only the successful version. Trained on these self-purified paths, models internalize correct reasoning rather than error-recovery loops. A sympathetic reader would care because the method is data-level and cheap: it replaces expensive oversampling and bypasses reward hacking, and the paper reports consistent gains across math, science, and code benchmarks.

Core claim

The paper's central claim is that trajectory-level execution noise — not token-level reasoning error — is the binding constraint in agentic RL for 4B–7B models. CLEANER converts a noisy exploration log into a counterfactual clean trajectory: when the model's code fails and the same rollout later contains a self-correction that executes successfully, SAAR rolls the history back to the failure point and grafts in the corrected code (shallow replacement if code similarity is high, deep reasoning replacement if low), then recomputes the policy-gradient log-probabilities under the purified context. Training on these purified paths suppresses erroneous tool calls, improves pass@1 accuracy, and rea

What carries the argument

Similarity-Aware Adaptive Rollback (SAAR): an online trajectory-rewriting mechanism triggered by execution errors. It freezes the history at the failure, asks the policy to produce a corrected code, verifies it, then uses `difflib.SequenceMatcher` similarity between original and corrected code (threshold γ=0.5) to decide whether to swap in only the corrected code or the whole corrected reasoning+code turn. Log-probabilities are recomputed under the purified context (using a KV-cache-reuse attention strategy) so the update is grounded in the causal path, and for the 7B model 30% of trajectories are left un-purified to preserve self-correction ability.

Load-bearing premise

The load-bearing premise is that the GRPO objective evaluated on the rewritten 'purified' trajectory is a valid policy-gradient estimator; because τ_purified was never actually sampled from the old policy, and the importance-weight re-computation does not correct the sampling distribution, the update may instead amount to behavior-cloning the corrected code. If that estimator is biased, the claimed RL-superiority and efficiency gains are on weaker footing.

What would settle it

Train a model on exactly the same SAAR-purified trajectories, but with the GRPO surrogate replaced by plain supervised imitation (max-likelihood on the purified code and reasoning). If the pass@1 gains on AIME24/25 and LiveCodeBench largely persist, the paper's credit-assignment mechanism is not what drives the gains; the method would then be an SFT data-cleaning trick. Conversely, if the gains vanish, the RL-specific claim is supported.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Execution-failure frequency during RL is a bottleneck; suppressing it via purification yields consistent pass@1 improvements across four benchmarks and two model sizes.
  • CLEANER reaches roughly the same accuracy as a strong agentic baseline in about one-third of the training steps, implying data quality can substitute for compute in this regime.
  • The benefit is internalized: with SAAR disabled at evaluation, accuracy drops by only ~0.6–2.5 percentage points, while deploying SAAR at inference adds only 8.8% latency.
  • Starting from a degraded policy (step 200 of a baseline), SAAR restores stability and improves AIME24/25 accuracy by 5.2% and 1.0%, but does not fully match from-scratch training — so the mechanism belongs in the training loop, not just after.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: If the credit-assignment story is right, the same retrospective-purification idea should transfer to other tool calls (search, shell, APIs) and to other policy-gradient objectives; the similarity heuristic may even need to be replaced by a learned or model-based judgment of 'same attempt'.
  • Editorial inference: The paper's GRPO estimator on `τ_purified` is a counterfactual rewrite rather than a sample from the old policy; until an unbiasedness argument or a control experiment is supplied, the reported gains could plausibly come from supervised copying of corrected code instead of a novel RL exploration signal. A clean test: train on the same purified trajectories with plain behavior
  • Editorial inference: Appendix B reports that using the discarded erroneous actions as negative samples (via online DPO with token masking) failed and even caused training collapse; that self-reported negative result is consistent with the paper's framing that the benefit comes from removing noise, not from contrastive learning, but it also leaves open whether a better-shaped negative signal could
  • Editorial inference: The 70/30 curriculum mix is a stability lever: the paper keeps 30% raw trajectories for the 7B model to preserve self-correction. Varying this ratio across task difficulty or training stage might further improve the trade-off between error avoidance and error recovery.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes CLEANER, a data-level intervention for agentic reinforcement learning with Python tool use. When a rollout contains an execution error that is later corrected within the same trajectory, the Similarity-Aware Adaptive Rollback (SAAR) mechanism constructs a 'purified' trajectory by replacing the failed step with the successful correction. The replacement is shallow (only the code action is replaced) when the erroneous and corrected code are semantically similar, and deep (the whole reasoning step is replaced) otherwise. The purified trajectories are then used to update the policy with GRPO, with log-probabilities of the corrected code recomputed under the purified context using RadixAttention. Experiments on Qwen3-4B-Instruct and Qwen2.5-7B-Instruct report gains over a DAPO baseline on AIME24/25, GPQA, and LiveCodeBench, and claim state-of-the-art efficiency, matching DemyAgent-4B with one-third of the training steps.

Significance. If the method works as described, it would be a practically useful way to reduce the cost of agentic RL for small models, and the promised release of code and processed datasets would support reproducibility. The core idea of eliminating error-contaminated context during collection is simple and plausible. However, the paper currently lacks a valid derivation of the RL update on purified trajectories; the importance ratio in Eq. (5) is computed against a distribution that did not generate the data. Without either a correct estimator or a control separating imitation from RL exploration, the central claim that CLEANER 'boosts agentic RL' is not established. The empirical evaluation is also based on single runs and, for the Qwen2.5 experiments, on an underspecified data filter.

major comments (3)
  1. [§4.3 and Eq. (5)] The GRPO update on purified trajectories is not a valid policy-gradient update as written. Eq. (5) requires the group trajectories {τ_i} to be sampled from π_θold. SAAR, however, constructs τ_purified by a counterfactual transformation T of a rollout that was sampled from π_θold: the corrected code c'_t is generated from the error-augmented context h~_t, not from the purified context. The actual proposal distribution for τ_purified is q(τ_purified) = Σ_{τ: T(τ)=τ_purified} π_θold(τ), which is not equal to π_θold(τ_purified). Recomputing log-probabilities under the purified context (as described in §4.3) fixes the conditioning for c'_t but does not correct the denominator of ρ_i; moreover, for the deep-replacement case the auxiliary reasoning r'_aux is also generated from the error-augmented context and its log-probability is not recomputed at all. The objective is therefore a biased surr
  2. [§5.1 and Table 2] The empirical evidence relies on single-run comparisons without error bars or multiple seeds. Differences of 1–3% in AIME/GPQA/LiveCodeBench are within the range of typical seed-to-seed variance for small-model RL. In addition, the Qwen2.5-7B experiments filter 'trivially easy or unsolvable' instances, but the filtering criterion is never defined. The filter is applied only to the Qwen2.5 experiments and not to the Qwen3-4B experiments, making the cross-model comparison asymmetric. The 70/30 curriculum mixing ratio is also introduced only for Qwen2.5-7B. I would ask for at least three seeds per condition, a precise description of the data filter (or removal of the filter), and a justification for the asymmetric treatment of the two model scales.
  3. [§5.3, Table 3, and Appendix A] Key hyperparameters — similarity threshold γ=0.5, retry limit K=3, and the learning rate 2e-6 for the 4B model — are selected by ablations on the target evaluation benchmarks. The DAPO baseline shares the same final configuration, so the head-to-head comparison is fair, but the absolute gains over the 'RL w/ Tools' condition in Table 2 may partly reflect benchmark tuning rather than the SAAR mechanism. The paper should either report a separate validation split for hyperparameter selection or provide a sensitivity analysis showing that the main conclusions hold across a range of γ, K, and learning rates. The claim in the abstract that gains are 'average 6% on AIME, 3% on GPQA, 5% on LiveCodeBench' also does not match Table 2, which reports 4% on GPQA for Qwen3-4B.
minor comments (4)
  1. [Table 1 / §5.1] The baseline is called 'Demystify-4B' in §5.1 and 'DemyAgent-4B' in Table 1 and elsewhere. Please use one consistent name.
  2. [Appendix A] The hardware section refers to a 'Qwen2.5-4B' model, but the experiments use Qwen3-4B and Qwen2.5-7B. This appears to be a typo.
  3. [Table 2 and Table 1] The 'RL w/o Tools' row for Qwen2.5-7B reports LiveCodeBench-v6 of 1.1, while Table 1 lists the same base model with 15.2 on LiveCodeBench. The evaluation settings differ, but this is not explained; please clarify.
  4. [General] The manuscript is marked 'Work in progress' on every page and contains corrupted font artifacts in Section 3 and Figures 4–5. The unicode fragments and missing figure text make parts of the paper difficult to read. Please copyedit and regenerate the figures.

Circularity Check

0 steps flagged

No significant circularity: the reported gains are empirical, externally benchmarked, and not forced by construction.

full rationale

The paper's central claims are empirical: SAAR constructs counterfactually cleaned trajectories and the model is trained on them, with accuracy measured on external benchmarks (AIME24/25, GPQA, LiveCodeBench). The method does not derive a prediction from fitted constants: the similarity threshold, retry limit, and learning rate are hyperparameters tuned through ablations, not quantities whose values are then renamed as results. The importance-ratio concern raised by the reader is a correctness/validity issue about applying GRPO to trajectories that were not actually sampled from π_θold; it does not make the outcome equivalent to the input by construction. The paper also provides an independent check by evaluating with SAAR deactivated (Table 4), so the measured performance is not simply the scaffolding operating at test time. No load-bearing self-citation appears: the cited baselines, datasets, and algorithms (DAPO, Demystify, rStar2, etc.) are external prior work with no overlapping authorship with this paper. Thus no circular step reduces the derivation to its own inputs.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The method rests on several unverified assumptions and a small set of hand-tuned hyperparameters; no new entities are postulated. The heaviest burden is the GRPO estimator on counterfactual trajectories, which is assumed without derivation.

free parameters (4)
  • Similarity threshold γ = 0.5
    Hand-set default; controls shallow vs deep replacement in §4.2. The paper says it is robust but the value is chosen via ablations.
  • Retry limit K = 3
    Max correction attempts per failure in Phase I; chosen by hand as a trade-off between recovery rate and computational cost.
  • SAAR mixing ratio (Qwen2.5-7B) = 0.7
    70% purified / 30% raw trajectories; hand-set curriculum in §4.3, applied only to the 7B model.
  • Learning rate = 2e-6 (4B) / 1e-6 (7B)
    Tuned; Table 3 ablation selects 2e-6 based on AIME24/25 performance.
axioms (5)
  • ad hoc to paper GRPO with importance ratios computed on counterfactual purified trajectories is a valid policy-gradient estimator
    Eq. (5) in §3 and the logit recomputation in §4.3 assume this, but τ_purified was not sampled from π_θold; no unbiasedness proof is given.
  • domain assumption The model can self-correct a large enough fraction of execution errors within K=3 attempts
    Phase I (§4.1) depends on successful correction to build purified trajectories; the paper shows error-rate evolution but not a correction-success-rate analysis.
  • domain assumption String similarity (difflib.SequenceMatcher) between broken and fixed code indicates whether the original reasoning is sound
    Case A/B split in §4.2 uses threshold γ=0.5; no ground-truth validation of this proxy is provided.
  • domain assumption Training on error-free 'immediate-success' histories does not degrade test-time robustness or teach shortcut behavior
    The method assumes the purified path is a good learning signal; Table 4 gives a single-run ablation, but the assumption is not independently established.
  • standard math Standard GRPO definitions (reward normalization, clipping) from Guo et al. hold
    Background from preliminaries §2.2; assumed uncontroversial.

pith-pipeline@v1.3.0-alltime-deepseek · 18014 in / 14404 out tokens · 144514 ms · 2026-08-03T08:56:46.959554+00:00 · methodology

0 comments
read the original abstract

Agentic Reinforcement Learning (RL) has empowered Large Language Models (LLMs) to utilize tools like Python interpreters for complex problem-solving. However, for parameter-constrained models (e.g., 4B--7B), the exploration phase is often plagued by frequent execution failures, creating noisy trajectories that hinder policy optimization. Under standard outcome-based reward settings, this noise leads to a critical credit assignment issue, where erroneous actions are inadvertently reinforced alongside successful outcomes. Existing mitigations face a dilemma: dense rewards often trigger reward hacking, while supersampling incurs prohibitive computational costs. To address these challenges, we propose CLEANER. Distinct from external filtering methods, CLEANER exploits the model's intrinsic self-correction capabilities to eliminate error-contaminated context directly during data collection. At its core, the Similarity-Aware Adaptive Rollback (SAAR) mechanism autonomously constructs clean, purified trajectories by retrospectively replacing failures with successful self-corrections. Based on semantic similarity, SAAR adaptively regulates replacement granularity from shallow execution repairs to deep reasoning substitutions. By training on these self-purified paths, the model internalizes correct reasoning patterns rather than error-recovery loops. Empirical results on AIME24/25, GPQA, and LiveCodeBench show average accuracy gains of 6%, 3%, and 5% over baselines. Notably, CLEANER matches state-of-the-art performance using only one-third of the training steps, highlighting trajectory purification as a scalable solution for efficient agentic RL. Our models and code are available at GitHub

Figures

Figures reproduced from arXiv: 2601.15141 by Meng Li, Tianshi Xu, Yuteng Chen.

Figure 1
Figure 1. Figure 1: Left: Illustration of the differences between the standard baseline and our CLEANER. Right: By reducing the number of tool execution failures within trajectories during training, our method improves pass@1 accuracy on AIME’25 by 8.1%. 1 arXiv:2601.15141v1 [cs.LG] 21 Jan 2026 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Impact of execution noise. Spikes in the average number of tool execution fail￾ures per trajectory correlate directly with ac￾curacy degradation on AIME25, highlighting the sensitivity of policy optimization to error￾contaminated trajectories. Unlike internal Chain-of-Thought reasoning, agentic workflows introduce external stochasticity via inter￾actions with the environment E. This uncertainty manifests a… view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of our Similarity-Aware Adaptive Rollback (SAAR). core objective is to distill the learning signal by retrospectively eliminating execution failures from exploration rollouts, thereby constructing clean, self-purified trajectories. In these synthesized paths, the agent appears to solve the task fluently, enabling the optimizer to reinforce correct reasoning logic rather than error-recovery loo… view at source ↗
Figure 4
Figure 4. Figure 4: Evolution of training metrics during RL. Compared to the DAPO-baseline, CLEANER effectively suppresses erroneous tool calls in trajectories, leading to significant performance gains. removed, and the auxiliary correction thought r ′ aux is adopted as the canonical reasoning, forming the consistent tuple (r ′ aux, c ′ t , o′ t ). Through this adaptive mechanism, we synthesize the self-purified trajectory: τ… view at source ↗
Figure 5
Figure 5. Figure 5: Recovery from suboptimal policies. Comparison of training metrics before and after in￾troducing CLEANER at step 200. The inclusion of CLEANER effectively stabilizes the optimization process, leading to a marked improvement in final performance. Compared to Previous Works. The main results comparing CLEANER with existing works are summarized in [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Adapting the Interface, Not the Model: Runtime Harness Adaptation for Deterministic LLM Agents

    cs.AI 2026-05 unverdicted novelty 7.0

    Life-Harness evolves reusable interventions from training trajectories to enhance frozen LLM agents on unseen tasks across seven deterministic environments, yielding 88.5% average relative improvement in 116 of 126 mo...

  2. Adapting the Interface, Not the Model: Runtime Harness Adaptation for Deterministic LLM Agents

    cs.AI 2026-05 unverdicted novelty 6.0

    Life-Harness evolves reusable runtime interventions from training failures to improve frozen LLM agents by 88.5% on average across 126 settings in seven deterministic environments while transferring across 18 model backbones.

Reference graph

Works this paper leans on

31 extracted references · 27 linked inside Pith · cited by 1 Pith paper

  1. [2]

    Minimax-m1: Scaling test-time compute efficiently with lightning attention.arXiv preprint arXiv:2506.13585,

    Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, et al. Minimax-m1: Scaling test-time compute efficiently with lightning attention.arXiv preprint arXiv:2506.13585,

    Pith/arXiv arXiv

  2. [5]

    Retool: Reinforcement learning for strategic tool use in llms

    Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms. arXiv preprint arXiv:2504.11536,

    Pith/arXiv arXiv

  3. [6]

    Areal: A large-scale asynchronous reinforcement learning system for language reasoning.arXiv preprint arXiv:2505.24298,

    Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, et al. Areal: A large-scale asynchronous reinforcement learning system for language reasoning.arXiv preprint arXiv:2505.24298,

    Pith/arXiv arXiv

  4. [7]

    Tora: A tool-integrated reasoning agent for mathematical problem solving.arXiv preprint arXiv:2309.17452,

    Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. Tora: A tool-integrated reasoning agent for mathematical problem solving.arXiv preprint arXiv:2309.17452,

    Pith/arXiv arXiv

  5. [9]

    Skywork open reasoner 1 technical report.arXiv preprint arXiv:2505.22312,

    Jujie He et al. Skywork open reasoner 1 technical report.arXiv preprint arXiv:2505.22312,

    Pith/arXiv arXiv

  6. [10]

    10 Work in progress Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report.arXiv preprint arXiv:2409.12186,

    Pith/arXiv arXiv

  7. [11]

    Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974,

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974,

    Pith/arXiv arXiv

  8. [12]

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516,

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516,

    Pith/arXiv arXiv

  9. [13]

    Training language models to self-correct via reinforcement learning.arXiv preprint arXiv:2409.12917,

    Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning.arXiv preprint arXiv:2409.12917,

    Pith/arXiv arXiv

  10. [14]

    Websailor: Navigating super-human reasoning for web agent.arXiv preprint arXiv:2507.02592, 2025a

    Kuan Li, Zhongwang Zhang, Huifeng Yin, Liwen Zhang, Litu Ou, Jialong Wu, Wenbiao Yin, Baixuan Li, Zhengwei Tao, Xinyu Wang, et al. Websailor: Navigating super-human reasoning for web agent.arXiv preprint arXiv:2507.02592, 2025a. Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yongkang Wu, Ji-Rong Wen, Yutao Zhu, and Zhicheng Dou. Webthinker: Empowerin...

    Pith/arXiv arXiv

  11. [15]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437,

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437,

    Pith/arXiv arXiv

  12. [16]

    Aaron Parisi, Yao Zhao, and Noah Fiedel

    URLhttps://openreview.net/forum?id=hO0c2jD5c3. Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models.arXiv preprint arXiv:2205.12255,

    Pith/arXiv arXiv

  13. [17]

    Defeating the training-inference mismatch via fp16.arXiv preprint arXiv:2510.26788,

    Penghui Qi, Zichen Liu, Xiangxin Zhou, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Defeating the training-inference mismatch via fp16.arXiv preprint arXiv:2510.26788,

    arXiv

  14. [18]

    Toolrl: Reward is all tool learning needs.arXiv preprint arXiv:2504.13958,

    Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. Toolrl: Reward is all tool learning needs.arXiv preprint arXiv:2504.13958,

    Pith/arXiv arXiv

  15. [19]

    Toolllm: Facilitating large language models to master 16000+ real-world apis.arXiv preprint arXiv:2307.16789,

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis.arXiv preprint arXiv:2307.16789,

    Pith/arXiv arXiv

  16. [20]

    pocoo.org/2025/10/17/code/

    URLhttps://lucumr. pocoo.org/2025/10/17/code/. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools.Advances in Neural Information Processing Systems, 36:68539–68551,

    2025

  17. [21]

    rstar2-agent: Agentic reasoning technical report.arXiv preprint arXiv:2508.20722,

    11 Work in progress Ning Shang, Yifei Liu, Yi Zhu, Li Lyna Zhang, Weijiang Xu, Xinyu Guan, Buze Zhang, Bingcheng Dong, Xudong Zhou, Bowen Zhang, et al. rstar2-agent: Agentic reasoning technical report.arXiv preprint arXiv:2508.20722,

    Pith/arXiv arXiv

  18. [22]

    Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256,

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256,

    Pith/arXiv arXiv

  19. [23]

    Alfworld: Aligning text and embodied environments for interactive learning.arXiv preprint arXiv:2010.03768,

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning.arXiv preprint arXiv:2010.03768,

    Pith/arXiv arXiv 2010

  20. [25]

    R1-searcher: Incentivizing the search capability in llms via reinforcement learning

    Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-searcher: Incentivizing the search capability in llms via reinforcement learning. arXiv preprint arXiv:2503.05592,

    Pith/arXiv arXiv

  21. [26]

    Zerosearch: Incentivize the search capability of llms without searching.arXiv preprint arXiv:2505.04588,

    Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Yan Zhang, Fei Huang, and Jingren Zhou. Zerosearch: Incentivize the search capability of llms without searching.arXiv preprint arXiv:2505.04588,

    Pith/arXiv arXiv

  22. [27]

    True knowledge comes from practice: Aligning llms with embodied environments via reinforcement learning.arXiv preprint arXiv:2401.14151,

    Weihao Tan, Wentao Zhang, Shanqi Liu, Longtao Zheng, Xinrun Wang, and Bo An. True knowledge comes from practice: Aligning llms with embodied environments via reinforcement learning.arXiv preprint arXiv:2401.14151,

    Pith/arXiv arXiv

  23. [28]

    Otc: Optimal tool calls via reinforcement learning

    Hongru Wang, Cheng Qian, Wanjun Zhong, Xiusi Chen, Jiahao Qiu, Shijue Huang, Bowen Jin, Mengdi Wang, Kam-Fai Wong, and Heng Ji. Otc: Optimal tool calls via reinforcement learning. arXiv e-prints, pp. arXiv–2504, 2025a. Taiyi Wang, Zhihao Wu, Jianheng Liu, Jianye Hao, Jun Wang, and Kun Shao. Distrl: An asyn- chronous distributed reinforcement learning fram...

    Pith/arXiv arXiv

  24. [29]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  25. [30]

    Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025a

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025a. 12 Work in progress Zhaochen Yu, Ling Yang, Jiaru Zou, Shuicheng Yan, and Mengdi Wang. Demystifying reinforcement learning ...

    Pith/arXiv arXiv

  26. [31]

    purified

    13 Work in progress A IMPLEMENTATIONDETAILS Table 5: Hyperparameters for Reinforcement Learning. Hyperparameter Value Learning Rate2×10 −6 (4B) /1×10 −6 (7B) Max Prompt Length2,560 Max Response Length20,480(Avg.≈7,000) LR Warmup Steps20 PPO Clip Ratio (ϵ−, ϵ+)0.20,0.28 Retry LimitK3 Similarity Thresholdγ0.5 Reward Type Outcome-only{−1,1} Table 6: Sampling...

    2025

  27. [2020]

    Agentic reasoning and tool integration for llms via reinforcement learning.arXiv preprint arXiv:2505.01441,

    Joykirat Singh, Raghav Magazine, Yash Pandya, and Akshay Nambi. Agentic reasoning and tool integration for llms via reinforcement learning.arXiv preprint arXiv:2505.01441,

    Pith/arXiv arXiv

  28. [2022]

    Agentic entropy-balanced policy optimization

    Guanting Dong, Licheng Bao, Zhongyuan Wang, Kangzhi Zhao, Xiaoxi Li, Jiajie Jin, Jinghan Yang, Hangyu Mao, Fuzheng Zhang, Kun Gai, et al. Agentic entropy-balanced policy optimization. arXiv preprint arXiv:2510.14545, 2025a. Guanting Dong, Yifei Chen, Xiaoxi Li, Jiajie Jin, Hongjin Qian, Yutao Zhu, Hangyu Mao, Guorui Zhou, Zhicheng Dou, and Ji-Rong Wen. To...

    arXiv

  29. [2023]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Pith/arXiv arXiv

  30. [2024]

    Forest-of-thought: Scaling test-time compute for enhancing llm reasoning.arXiv preprint arXiv:2412.09078,

    Zhenni Bi, Kai Han, Chuanjian Liu, Yehui Tang, and Yunhe Wang. Forest-of-thought: Scaling test-time compute for enhancing llm reasoning.arXiv preprint arXiv:2412.09078,

    Pith/arXiv arXiv

  31. [2025]

    Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks.arXiv preprint arXiv:2211.12588,

    Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks.arXiv preprint arXiv:2211.12588,

    Pith/arXiv arXiv