Pith. sign in

REVIEW 2 major objections 5 minor 33 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

Per-Task Adaptive Clipping Stabilizes Multi-Task LLM Agent Training

2026-07-09 18:11 UTC pith:77XBYIIC

load-bearing objection Task-wise entropy clipping for multi-task agentic RL: solid empirical gains, one missing control the 2 major comments →

arxiv 2607.07178 v1 pith:77XBYIIC submitted 2026-07-08 cs.LG cs.AI

Entropy Pacing Policy Optimization for Multi-Task Agentic Reinforcement Learning

Zetian Hu , Shunyu Liu , Junjie Zhang , Yongcheng Jing , Ting-En Lin , Yongbin Li , Dacheng Tao This is my paper
classification cs.LG cs.AI
keywords multi-task reinforcement learningagentic LLMspolicy entropyadaptive clippingGRPOexploration-exploitationtrust region optimization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper identifies a phenomenon called 'pace mismatch' in multi-task reinforcement learning for LLM agents: when training a single policy on multiple heterogeneous tasks simultaneously, easier tasks collapse to deterministic behavior early while harder tasks remain exploratory. This creates inter-task entropy crossovers and late-stage entropy spikes that destabilize the shared policy. The authors propose Entropy Pacing Policy Optimization (EPPO), which replaces the global fixed clipping range in Group Relative Policy Optimization (GRPO) with a task-wise dynamic clipping mechanism. This mechanism tracks each task's policy entropy, computes a normalized collapse ratio to compare learning pace across tasks, and adjusts the clipping range accordingly—tightening updates for over-confident tasks while relaxing them for under-explored ones. A stability-aware trend constraint further dampens updates when entropy rises. Experiments on five agentic benchmarks show that EPPO reduces entropy crossovers, suppresses late-stage spikes, and improves average success rates over multi-task baselines.

Core claim

The paper claims that the primary cause of instability in multi-task agentic GRPO is the global fixed clipping range, which cannot simultaneously accommodate tasks at different stages of exploration-exploitation. By replacing this global constraint with a task-wise dynamic clipping range driven by relative entropy pace, the paper demonstrates that inter-task entropy dynamics can be coordinated, leading to more stable training and improved performance.

What carries the argument

Task-wise dynamic clipping mechanism (EPPO): an entropy tracker (EMA-smoothed policy entropy), a progress pacer (cohort-normalized z-score mapped to a bounded clipping range via tanh), and a stability-aware trend constraint (downscaling clip range when entropy increases).

Load-bearing premise

The core assumption is that policy entropy is a valid, task-comparable proxy for a task's exploration-exploitation state, and that modulating the clipping range based on this signal effectively coordinates learning pace across heterogeneous tasks without unintended side effects.

What would settle it

If one could show that the observed entropy crossovers and spikes are symptoms of a deeper issue (e.g., gradient conflict or reward sparsity) rather than the clipping range itself, then adjusting clipping based on entropy would be treating the symptom rather than the cause, and the improvements would be spurious.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If task-wise entropy pacing generalizes, it could become a standard component in multi-task LLM agent training pipelines, enabling more efficient scaling to diverse task mixtures.
  • The principle of using policy entropy as a control signal for update aggressiveness could extend beyond clipping ranges to other trust-region or learning rate scheduling mechanisms.
  • The 'pace mismatch' phenomenon may also manifest in other forms of multi-objective optimization with shared parameters, such as multi-modal model training or continual learning scenarios.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's mechanism implicitly assumes that entropy collapse is undesirable for easier tasks, but in some cases, rapid convergence to deterministic behavior may be optimal if the task is truly solved. The pacing mechanism might artificially slow down convergence on easy tasks to maintain 'balance'.
  • The approach could be extended to dynamically adjust other hyperparameters beyond clipping, such as per-task learning rates or KL penalty coefficients, based on the same entropy pacing signal.
  • The effectiveness of entropy as a pacing signal might diminish in very large action spaces or tasks with inherently high entropy ceilings, where collapse ratios become less meaningful.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper identifies a phenomenon called 'pace mismatch' in multi-task agentic RL with LLMs, where different tasks exhibit asynchronous entropy collapse dynamics under a shared policy, leading to inter-task entropy crossovers and late-stage entropy spikes. The authors attribute this partly to GRPO's global fixed clipping range, which cannot simultaneously accommodate tasks at different exploration-exploitation stages. They propose EPPO, which replaces the global clip with a task-wise dynamic clipping mechanism driven by each task's relative entropy pace. The mechanism has three components: (1) an entropy tracker using EMA-smoothed collapse ratios, (2) a progress pacer that maps standardized collapse ratios to bounded clip adjustments via tanh, and (3) a stability-aware trend constraint that downscales the clip when entropy rises. Experiments on five agentic benchmarks (ALFWorld, DB, KG, OS, WebShop) show EPPO improves average success rate over AgentRL, DCPO, and BAPO baselines, with cleaner entropy dynamics. Theoretical analysis in Appendix D provides boundedness and contraction guarantees for the clipping mechanism.

Significance. The paper addresses a practically important problem in multi-task agentic RL training. The identification of the pace mismatch phenomenon — with quantitative diagnostics (crossover count, trend R², spike amplitude) — is a useful empirical observation for the community. The proposed solution is lightweight, modular, and pluggable into existing GRPO pipelines. The theoretical analysis (Appendix D, Propositions 1–2) provides formal boundedness and Lipschitz guarantees, which is commendable, though these are properties of the mechanism's design rather than convergence guarantees. The experimental evaluation spans multiple benchmarks and model scales, and the RLOO variant (Appendix B) demonstrates generalizability beyond GRPO. The sensitivity analysis (Table 5) and component ablations (Table 4) add credibility. The main empirical claim — that entropy-aware task-wise clipping stabilizes multi-task training — is supported by the results, though the mechanistic specificity of the claim could be stronger (see Major Comment 1).

major comments (2)
  1. §4.1, §4.2, Table 4: The paper's central mechanistic claim is that entropy-aware task-wise clipping specifically addresses pace mismatch. However, the ablation in Table 4 only compares EPPO (full) against 'w/o task-wise clip' (i.e., reverting to global clipping) and 'w/o trend constraint.' This confirms that per-task adaptation helps but does not isolate entropy as the correct signal. A natural control would be a per-task clipping baseline driven by a non-entropy signal (e.g., per-task fixed ε values based on reward variance, success rate, or task difficulty). Without such a control, one cannot distinguish 'entropy-aware pacing is the key mechanism' from 'any per-task clipping would similarly reduce inter-task interference.' The comparisons with DCPO and BAPO do not close this gap because those methods operate at token/sample level, not task level. This matters because the paper's stated
  2. contribution is specifically the entropy-pacing design (Eqs. 7–11), not the general idea of per-task clipping. If a simpler per-task heuristic achieves comparable gains, the novelty and mechanistic claim weaken substantially. The authors should either add this control experiment or explicitly scope their claim to 'per-task clipping with entropy as one effective signal.'
minor comments (5)
  1. §4.2, Eq. (9): The base clipping range is denoted ϵ₀ here but ϵ_base in Appendix A.2. Please unify notation.
  2. Table 1: The KG result for EPPO (29.6) is slightly lower than AgentRL (30.0). The text in §5.2 mentions 'KG shows a mild trade-off,' but this is a negative result that should be discussed more transparently, including whether it reflects task-specific interference or noise.
  3. Figure 1c: The 'Trend R²' metric is described as 'average coefficient of determination from a linear fit of task entropy versus log t.' It would help to clarify whether higher R² is always better — a task that collapses very early and stays flat would have high R², but so would a task that never explores. The metric's interpretability could be briefly discussed.
  4. Appendix C: The discrepancy between reproduced and reported AgentRL results is attributed to code-level modifications, hardware constraints, and other factors. While reasonable, it would strengthen reproducibility to release the modified code or at least specify the code-level changes more precisely (e.g., which files/functions were modified).
  5. §5.1: The paper uses 8 H20 GPUs vs. AgentRL's 16 H800 GPUs. It would be useful to note whether the reduced compute could systematically favor or disadvantage certain methods (e.g., if EPPO's stability advantages are more pronounced at smaller batch sizes).

Circularity Check

0 steps flagged

No significant circularity; theoretical propositions are trivially true by construction but presented as guarantees, not predictions

full rationale

The paper's derivation chain is: empirical observation (entropy crossovers/spikes) → hypothesis (global clipping causes pace mismatch) → proposed mechanism (entropy-aware task-wise dynamic clipping, Eqs. 5–11) → empirical validation (success rates on standard benchmarks + entropy dynamics analysis). Each step is defined independently. The entropy collapse ratio is computed from policy entropy, not from the clipping range or the success metric. The clipping range is computed from the collapse ratio, not from task success. The theoretical properties (Propositions 1 and 2 in Appendix D.1) are straightforward mathematical consequences of the construction (tanh ∈ [-1,1] implies boundedness; 1/(1+δ) < 1 when δ > 0 implies contraction), but the paper presents them as algorithmic guarantees, not as empirical predictions or fitted results renamed as findings. No self-citation chain exists — BAPO, CE-GPPO, DCPO, and DAPO are all by different author groups. The primary success metric is task success rate (Table 1), not entropy stability, so the evaluation is not tautologically tied to the diagnostic that motivated the method. The skeptic's concern about a missing non-entropy per-task clipping baseline is a valid experimental design issue but falls under correctness risk, not circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 3 axioms · 0 invented entities

The method introduces several hyperparameters that are fitted or chosen ad hoc (alpha, beta, epsilon bounds). The core axioms are standard domain assumptions in RL (entropy as exploration proxy), but the specific choice of z-score standardization for pacing is an ad-hoc design decision specific to this paper.

free parameters (5)
  • base clipping range (epsilon_0) = 0.2
    Standard PPO/GRPO default, used as the anchor for the dynamic clipping.
  • pacing strength (alpha) = 0.4
    Controls the magnitude of clipping adjustment based on entropy z-score. Chosen via sensitivity analysis (Table 5).
  • entropy EMA decay (beta) = 0.95
    Smoothing factor for the entropy tracker. A standard hyperparameter for moving averages, but its value is chosen ad hoc.
  • epsilon_min = 0.15
    Hard floor for the clipping range, mentioned in Appendix A.2 but not in the main method section.
  • epsilon_max = 0.3
    Hard ceiling for the clipping range, mentioned in Appendix A.2 but not in the main method section.
axioms (3)
  • domain assumption Policy entropy is a valid proxy for a task's exploration-exploitation state.
    Stated in Section 4.2: 'The key design choice is to treat entropy as an online indicator of a task's position on the exploration-exploitation trajectory.' This is a standard RL assumption but central to the method's design.
  • domain assumption GRPO's global clipping range is a key contributor to pace mismatch.
    Stated in Section 4.1: 'We identify GRPO's global clipping range epsilon as a key contributor to this phenomenon.' This motivates the entire approach.
  • ad hoc to paper Standardizing collapse ratios across tasks makes pacing comparable.
    Section 4.2 assumes that using cohort-level z-scores of entropy collapse ratios is the correct way to compare and coordinate pace across heterogeneous tasks.

pith-pipeline@v1.1.0-glm · 20858 in / 2241 out tokens · 163192 ms · 2026-07-09T18:11:01.711854+00:00 · methodology

0 comments
read the original abstract

Recent breakthroughs of Reinforcement Learning (RL) have highlighted its potential for complex agentic Large Language Model (LLM) tasks. However, existing efforts largely focus on single-task settings, whereas real-world deployment necessitates a generalist agent capable of solving multiple tasks simultaneously. In this work, we identify a critical yet underexplored phenomenon in multi-task agentic RL: different tasks can exhibit exploration-exploitation pace mismatch. Specifically, easier tasks may converge early to low-entropy policies that hinder learning on harder tasks, while harder tasks can, in turn, push easier tasks back toward high-entropy exploration. This back-and-forth interaction creates inter-task entropy crossovers and frequent entropy spikes. Inspired by this observation, we introduce Entropy Pacing Policy Optimization (EPPO) for multi-task agentic LLMs, which coordinates entropy across tasks to stabilize multi-task optimization. At the core of EPPO is a task-wise dynamic clipping mechanism that replaces the fixed clipping threshold in Group Relative Policy Optimization (GRPO) with a task entropy-aware adaptive bound, tightening updates for over-confident tasks while relaxing them for under-explored ones. Experiments on the multi-task agentic benchmarks demonstrate that the proposed EPPO yields results superior to its counterparts.

Figures

Figures reproduced from arXiv: 2607.07178 by Dacheng Tao, Junjie Zhang, Shunyu Liu, Ting-En Lin, Yongbin Li, Yongcheng Jing, Zetian Hu.

Figure 1
Figure 1. Figure 1: Task-wise policy entropy dynamics during multi-task agentic training. (a) The AgentRL [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: EPPO overview. Per-task entropy is tracked with EMA and converted to a normalized [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Validation learning curves (pass rate) comparing EPPO [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Entropy z-score trajectories (three-task setting) diag￾nosing pacing stability. (a) Without the trend constraint, z-scores exhibit high-frequency oscillations and spikes. (b) With the trend constraint, z-scores remain smoother and within a narrower band, indicating stabilized pacing. Why the trend constraint mat￾ters [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Per-task success rates under five-task multi-task training. We compare representative [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Per-task success rates under three-task multi-task training (ALFWorld, DB, OS) with a [PITH_FULL_IMAGE:figures/full_fig_p017_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Task-wise entropy dynamics under different ablation settings. [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 33 canonical work pages · 25 internal anchors

  1. [1]

    com/claude-4-system-card

    URL https://www.anthropic. com/claude-4-system-card. Philipp Becker, Niklas Freymuth, Serge Thilges, Fabian Otto, and Gerhard Neumann. Troll: Trust re- gions improve reinforcement learning for large language models.arXiv preprint arXiv:2510.03817,

    work page arXiv

  2. [2]

    Reasoning with Exploration: An Entropy Perspective

    Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective.arXiv preprint arXiv:2506.14758,

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    Training Verifiers to Solve Math Word Problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models

    Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models.arXiv preprint arXiv:2505.22617,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    Cotbal: Com- prehensive task balancing for multi-task visual instruction tuning.arXiv preprint arXiv:2403.04343,

    Yanqi Dai, Zebin You, Dong Jing, Yutian Luo, Nanyi Fei, Guoxing Yang, and Zhiwu Lu. Cotbal: Com- prehensive task balancing for multi-task visual instruction tuning.arXiv preprint arXiv:2403.04343,

    work page arXiv

  6. [6]

    ReTool: Reinforcement Learning for Strategic Tool Use in LLMs

    Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms. arXiv preprint arXiv:2504.11536,

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming–the rise of code intelligence.arXiv preprint arXiv:2401.14196,

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    Measuring Mathematical Problem Solving With the MATH Dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [10]

    Rethinking entropy regularization in large reasoning models.arXiv preprint arXiv:2509.25133,

    Yuxian Jiang, Yafu Li, Guanxu Chen, Dongrui Liu, Yu Cheng, and Jing Shao. Rethinking entropy regularization in large reasoning models.arXiv preprint arXiv:2509.25133,

    work page arXiv

  11. [11]

    Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516,

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [12]

    InternBootcamp Technical Report: Boosting LLM Reasoning with Verifiable Task Scaling

    Peiji Li, Jiasheng Ye, Yongkang Chen, Yichuan Ma, Zijie Yu, Kedi Chen, Ganqu Cui, Haozhan Li, Jiacheng Chen, Chengqi Lyu, et al. Internbootcamp technical report: Boosting llm reasoning with verifiable task scaling.arXiv preprint arXiv:2508.08636, 2025a. Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yongkang Wu, Ji-Rong Wen, Yutao Zhu, and Zhicheng D...

    work page internal anchor Pith review Pith/arXiv arXiv

  13. [13]

    DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models

    Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al. Deepseek-v3. 2: Pushing the frontier of open large language models.arXiv preprint arXiv:2512.02556, 2025a. Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning.A...

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [14]

    AgentBench: Evaluating LLMs as Agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents.arXiv preprint arXiv:2308.03688,

    work page internal anchor Pith review Pith/arXiv arXiv

  15. [15]

    InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners

    Yuhang Liu, Pengxiang Li, Congkai Xie, Xavier Hu, Xiaotian Han, Shengyu Zhang, Hongxia Yang, and Fei Wu. Infigui-r1: Advancing multimodal gui agents from reactive actors to deliberative reasoners.arXiv preprint arXiv:2504.14239, 2025b. Junyu Luo, Weizhi Zhang, Ye Yuan, Yusheng Zhao, Junwei Yang, Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, ...

    work page internal anchor Pith review Pith/arXiv arXiv

  16. [16]

    Multi-Turn Reinforcement Learning for Tool-Calling Agents with Iterative Reward Calibration

    Wachiravit Modecrua, Krittanon Kaewtawee, Krittin Pachtrachai, and Touchapon Kraisingkorn. Multi-turn reinforcement learning for tool-calling agents with iterative reward calibration.arXiv preprint arXiv:2604.02869,

    work page internal anchor Pith review Pith/arXiv arXiv

  17. [17]

    Aske Plaat, Max van Duijn, Niki van Stein, Mike Preuss, Peter van der Putten, and Kees Joost Batenburg

    URL https://openai.com/index/o3-o 4-mini-system-card/. Aske Plaat, Max van Duijn, Niki van Stein, Mike Preuss, Peter van der Putten, and Kees Joost Batenburg. Agentic large language models, a survey.arXiv preprint arXiv:2503.23037,

    work page arXiv

  18. [18]

    Qwen2.5 Technical Report

    Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2.5 technical report.arXiv preprint arXiv:2412.15115,

    work page internal anchor Pith review Pith/arXiv arXiv

  19. [19]

    ALFWorld: Aligning Text and Embodied Environments for Interactive Learning

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning.arXiv preprint arXiv:2010.03768,

    work page internal anchor Pith review Pith/arXiv arXiv 2010

  20. [20]

    OpenAI GPT-5 System Card

    Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. Openai gpt-5 system card.arXiv preprint arXiv:2601.03267,

    work page internal anchor Pith review Pith/arXiv arXiv

  21. [21]

    CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learning

    Zhenpeng Su, Leiyu Pan, Minxuan Lv, Yuntao Li, Wenping Hu, Fuzheng Zhang, Kun Gai, and Guorui Zhou. Ce-gppo: Coordinating entropy via gradient-preserving clipping policy optimization in reinforcement learning.arXiv preprint arXiv:2509.20712,

    work page internal anchor Pith review Pith/arXiv arXiv

  22. [22]

    Position: Agent Should Invoke External Tools ONLY When Epistemically Necessary

    Hongru Wang, Cheng Qian, Manling Li, Jiahao Qiu, Boyang Xue, Mengdi Wang, Heng Ji, and Kam- Fai Wong. Toward a theory of agents as tool-use decision-makers.arXiv preprint arXiv:2506.00886,

    work page internal anchor Pith review Pith/arXiv arXiv

  23. [23]

    WebDancer: Towards Autonomous Information Seeking Agency

    11 Jialong Wu, Baixuan Li, Runnan Fang, Wenbiao Yin, Liwen Zhang, Zhengwei Tao, Dingchu Zhang, Zekun Xi, Gang Fu, Yong Jiang, et al. Webdancer: Towards autonomous information seeking agency.arXiv preprint arXiv:2505.22648,

    work page internal anchor Pith review Pith/arXiv arXiv

  24. [24]

    Bapo: Stabilizing off-policy reinforcement learning for llms via balanced policy optimization with adaptive clipping.arXiv preprint arXiv:2510.18927,

    Zhiheng Xi, Xin Guo, Yang Nan, Enyu Zhou, Junrui Shen, Wenxiang Chen, Jiaqi Liu, Jixuan Huang, Zhihao Zhang, Honglin Guo, et al. Bapo: Stabilizing off-policy reinforcement learning for llms via balanced policy optimization with adaptive clipping.arXiv preprint arXiv:2510.18927,

    work page arXiv

  25. [25]

    MiMo-V2-Flash Technical Report

    Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, Chen Zhang, Chenhong He, Chiheng Lou, Fuli Luo, Gang Wang, et al. Mimo-v2-flash technical report.arXiv preprint arXiv:2601.02780,

    work page internal anchor Pith review Pith/arXiv arXiv

  26. [26]

    DCPO: Dynamic Clipping Policy Optimization

    Shihui Yang, Chengfeng Dou, Peidong Guo, Kai Lu, Qiang Ju, Fei Deng, and Rihui Xin. Dcpo: Dynamic clipping policy optimization.arXiv preprint arXiv:2509.02333,

    work page internal anchor Pith review Pith/arXiv arXiv

  27. [27]

    DAPO: An Open-Source LLM Reinforcement Learning System at Scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    work page internal anchor Pith review Pith/arXiv arXiv

  28. [28]

    Agenttuning: Enabling generalized agent abilities for llms

    Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for llms. InFindings of the Association for Computational Linguistics: ACL 2024, pages 3053–3077,

    work page 2024

  29. [29]

    The Landscape of Agentic Reinforcement Learning for LLMs: A Survey

    Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, et al. The landscape of agentic reinforcement learning for llms: A survey.arXiv preprint arXiv:2509.02547, 2025a. Hanchen Zhang, Xiao Liu, Bowen Lv, Xueqiao Sun, Bohao Jing, Iat Long Iong, Zhenyu Hou, Zehan Qi, Hanyu Lai, Yifan X...

    work page internal anchor Pith review Pith/arXiv arXiv

  30. [30]

    Divide-Fuse-Conquer: Eliciting "Aha Moments" in Multi-Scenario Games

    Xiaoqing Zhang, Huabin Zheng, Ang Lv, Yuhan Liu, Zirui Song, Xiuying Chen, Rui Yan, and Flood Sung. Divide-fuse-conquer: Eliciting" aha moments" in multi-scenario games.arXiv preprint arXiv:2505.16401, 2025c. Yuxiang Zhang, Shangxi Wu, Yuqi Yang, Jiangming Shu, Jinlin Xiao, Chao Kong, and Jitao Sang. o1-coder: an o1 replication for coding.arXiv preprint a...

    work page internal anchor Pith review Pith/arXiv arXiv

  31. [31]

    Demys- tifying group relative policy optimization: Its policy gradient is a u-statistic.arXiv preprint arXiv:2603.01162,

    Hongyi Zhou, Kai Ye, Erhan Xu, Jin Zhu, Ying Yang, Shijin Gong, and Chengchun Shi. Demys- tifying group relative policy optimization: Its policy gradient is a u-statistic.arXiv preprint arXiv:2603.01162,

    work page arXiv

  32. [32]

    Hephaestus: Improving Fundamental Agent Capabilities of Large Language Models through Continual Pre-Training

    Yuchen Zhuang, Jingfeng Yang, Haoming Jiang, Xin Liu, Kewei Cheng, Sanket Lokegaonkar, Yifan Gao, Qing Ping, Tianyi Liu, Binxuan Huang, et al. Hephaestus: Improving fundamental agent capa- bilities of large language models through continual pre-training.arXiv preprint arXiv:2502.06589,

    work page internal anchor Pith review Pith/arXiv arXiv

  33. [33]

    14 A.2 Hyperparameters

    12 Appendix Table of Contents A Implementation Details and Hyperparameters 14 A.1 Environments and Training Data . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 A.2 Hyperparameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 B Additional Results with RLOO-based Training Variant 14 C Analysis of Discrepancies with Repo...

    work page 2025