Pith. sign in

REVIEW 4 major objections 6 minor 14 cited by

The paper claims that policy-gradient magnitude is inherently coupled to entropy, and proposes an entropy-modulated advantage to re-calibrate it for long-horizon LLM agents.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

EMPG re-weights policy-gradient updates by step-level token entropy, amplifying confident correct actions and muting uncertain ones, and adds a future-clarity bonus.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection EMPG is a plausible empirical win for long-horizon LLM agent training, but the theory that motivates it is overreaching; referee it with a request for honest reframing and missing baselines. the 4 major comments →

arxiv 2509.09265 v1 pith:TIXCOYNH submitted 2025-09-11 cs.LG cs.CL

Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agents

Jiawei Wang , Jiacai Liu , Yuqian Fu , Yingru Li , Xintao Wang , Yuan Lin , Yu Yue , Lin Zhang
show 2 more authors
Yang Wang Ke Wang
This is my paper
classification cs.LG cs.CL
keywords entropy-modulated policy gradientslong-horizon agentscredit assignmentsparse rewardsLLM agentspolicy gradientuncertaintyreinforcement learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a core obstacle in training LLM-based agents on long-horizon tasks is an intrinsic coupling between policy-gradient magnitude and policy entropy: confident steps receive small updates and uncertain steps receive large, noisy ones. To fix this, it proposes Entropy-Modulated Policy Gradients (EMPG), which re-weights each step's advantage by the step's own uncertainty and adds a future-clarity bonus for actions that lead to more predictable states. The method is presented as a drop-in advantage modulation on top of existing policy-gradient algorithms. Experiments on WebShop, ALFWorld, and Deep Search report consistent gains over the underlying baselines, with the largest improvement being +8.1 points on ALFWorld average success rate. If the claim holds, it gives a cheap, PRM-free way to turn sparse outcome rewards into step-level learning signals.

Core claim

The paper identifies what it calls an entropy-gradient coupling: for a softmax policy, the expected squared norm of the score function with respect to the logits is 1 - exp(-H2(pi)), a monotonic function of the policy's Rényi-2 entropy. This means low-entropy (confident) actions produce small gradient norms and high-entropy (uncertain) actions produce large ones. EMPG counters this by scaling each step's advantage with a batch-normalized function g(H_t) = exp(-k H_norm,t)/mean, so confident correct steps are amplified, confident errors are penalized more strongly, and uncertain exploration is dampened. It also appends a future clarity bonus zeta * exp(-k' H_norm,t+1) to encourage transitions

What carries the argument

Proposition 1's identity E_a||∇_z log π(a|s)||^2 = 1 - exp(-H2(π)) links the score-function norm to Rényi-2 entropy; EMPG builds on it by using the modulated advantage A_mod(i,t) = A(i)·g(H_t) + ζ·f(H_{t+1}), where g is a batch-normalized exponential scaling and f is the future-clarity bonus. The identity provides the motivation; the modulated advantage is the mechanism that redistributes the learning signal.

Load-bearing premise

The paper extends the logit-space result (Proposition 1) to the parameter-space gradient without proof; the claim that confident steps get small updates and uncertain steps get large ones may not hold after the network Jacobian transforms the score function.

What would settle it

Run EMPG and a baseline on ALFWorld while recording per-step entropy and per-step gradient norm w.r.t. network parameters; if the average gradient norm of low-entropy steps is not smaller than that of high-entropy steps, the motivating coupling is absent and the method's gains would need another explanation.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • EMPG converts sparse, outcome-only rewards into step-level learning signals without training a process reward model.
  • It is compatible with existing policy-gradient backbones, so it can be dropped onto GRPO or DAPO without altering the underlying algorithm.
  • The reported stability improvements (e.g., flat KL loss late in training) suggest the scaling term acts as an implicit regularizer against policy collapse.
  • The out-of-domain gains on Deep Search (+3.9 points) imply the method teaches a generalizable posture toward uncertainty, not just task-specific memorization.
  • Because the method only needs token-level probabilities, it is cheap enough to scale to 32B-parameter agents.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The proof of the entropy-gradient coupling is given for the score function w.r.t. logits; whether the coupling survives the Jacobian to the network parameters is a gap the paper does not close. If it fails, an alternative justification for the empirical gains might lie in the implicit regularization the scaling provides.
  • The future clarity bonus can be seen as a local empowerment or information-gain objective; connecting EMPG to existing intrinsic-motivation theory could suggest principled ways to set k and ζ rather than tuning them.
  • The same entropy-modulated advantage shaping could be applied to embodied control or multi-agent settings where step-level uncertainty is available, but the coupling would need re-checking outside the softmax finite-vocabulary case.
  • A simple test: train the same tasks with the scaling term frozen at g=1; if gains vanish, the normalization itself is the active ingredient rather than the entropy modulation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Entropy-Modulated Policy Gradients (EMPG), a method for training LLM agents on long-horizon tasks with sparse, binary rewards. EMPG modulates the per-step policy gradient by (i) multiplying the trajectory-level advantage with a self-calibrating scaling factor g(H_t) that amplifies confident steps and attenuates uncertain ones (Eqs. 8–10), and (ii) adding a future-clarity bonus proportional to the confidence of the next step (Eq. 11). The method is motivated by Proposition 1, which states that for a softmax policy the expected squared norm of the logit-space score function equals 1 - exp(-H_2(π)), implying that high-entropy actions have larger score norms. Experiments on WebShop, ALFWorld, and Deep Search with Qwen2.5-1.5B/7B/32B show consistent improvements over GRPO and DAPO baselines, with the largest reported gain being +8.1 points on ALFWorld average success for Qwen2.5-1.5B with GRPO (Table 1). The paper also includes ablations on Deep Search (Table 2), learning-dynamics analyses, and an implementation appendix.

Significance. If the empirical results hold, EMPG is a practically useful, lightweight plug-in for policy-gradient LLM agent training: it requires only per-step token entropies, adds no learned value model, and yields consistent gains across three benchmarks, two RL backbones, and three model scales. The paper's strengths include reproducible experimental settings, ablations of both components, multiple training seeds, and detailed implementation pseudocode (Appendices C and E). However, the paper's central theoretical claim—that policy-gradient magnitude is inherently coupled with policy entropy in a way that motivates the proposed re-calibration—is only proven for the logit-space score function, not for the parameter-space gradients actually used in training. This gap, together with the mismatch between Shannon entropy used in the method and Rényi-2 entropy in the proposition, weakens the mechanistic interpretation of the empirical gains. The empirical contribution is plausible, but the theoretical framing needs substantial revision.

major comments (4)
  1. [Section 3.4, Eq. (5); Section 4.2, Eq. (8); Appendix B] Proposition 1 is proven for the score function with respect to logits z: E_a ||∇_z log π(a|s)||^2 = 1 - exp(-H_2(π)). The method, however, reweights the parameter-space gradient ∇_θ log π = J_z(θ) ∇_z log π. The squared parameter-space norm is (e_k - π)^T J_z^T J_z (e_k - π), so the singular-value spectrum of the network Jacobian can rescale, redistribute, or even reverse the entropy ordering. The text in Section 3.4 states that 'the magnitude of policy gradients is inherently coupled with the entropy' and uses this as the fundamental motivation for EMPG, but no proof or experiment establishes this coupling for ∇_θ log π. Appendix B explicitly admits (B.2.1) that a closed-form reweighted objective is 'non-trivial' and that the interpretation is 'sufficient to justify the update rule.' This is a load-bearing gap: if the parameter-space coupling is weak or absent, the stated mechanism behi
  2. [Section 4.1, Eq. (7); Proposition 1] The step-level uncertainty H_t used in Eqs. (7)–(11) is the average token-level Shannon entropy, while Proposition 1 is a statement about the Rényi-2 entropy of the action distribution. Shannon entropy and Rényi-2 entropy are not monotonically related for categorical distributions with varying support; a distribution with lower Shannon entropy can have higher Rényi-2 entropy and vice versa. The paper claims (Section 4.1) that a lower H_t corresponds to a 'lower-entropy state in the sense of Proposition 1,' which is not justified. This weakens the link between the theoretical identity and the implemented scaling. The authors should either use Rényi-2 entropy (or an estimator thereof) or provide empirical evidence that the monotonic relationship holds in the action distributions encountered during training.
  3. [Appendix B.2.2, Eq. (20)] The derivation of the intrinsic gradient applies the policy gradient theorem to J_intrinsic = E[Σ r_t^int(s_{t+1}; θ)], where the reward r_t^int depends on θ through the policy entropy H(π_θ(·|s_{t+1})). The standard score-function estimator for a θ-dependent reward includes an additional term ∇_θ r_t^int that is omitted in Eq. (20). If the entropy is intended to be treated as detached (stop-gradient), this must be stated explicitly; otherwise the 'rigorous theoretical derivation' in Appendix B is incomplete. As written, Eq. (23) is not the gradient of the composite objective J_EMPG(θ) defined in Eq. (16).
  4. [Section 5.2, Tables 1 and 2] The central empirical claim relies on tables reporting 'averaged over 3 random seeds' but without standard deviations, confidence intervals, or significance tests. Table 2 is particularly concerning: the ablations show +Gradient Scaling at 63.7 overall, +Future Bonus at 64.2, and full EMPG at 65.3 against a DAPO baseline of 62.0. With no variance estimates, it is difficult to assess whether the component-wise gains are meaningful or within noise. For a paper whose abstract claims 'substantial performance gains,' the authors should report per-seed results or at least standard deviations, especially for the headline +8.1 ALFWorld gain in Table 1.
minor comments (6)
  1. [Section 4.2, Eq. (9)] Equation (9) is typeset incorrectly: '1PNB' should be a fraction 1/(Σ_i T_i) or similar. Please fix the notation.
  2. [Algorithm 2 (Appendix E), line 47] The variable f_H is called 'future clarity bonus g'(H)' in the pseudocode comment, but Eq. (11) defines it as f(H). Rename for consistency.
  3. [Figure 1 and Section 5] The caption of Figure 1 mentions 'training dynamics on Musique,' but no Musique training curve is shown in the main text; only a KL-loss curve in Figure 2. Clarify or move the reference.
  4. [Section 5.3, Figure 3] The analysis of 'over 9,000 steps on ALFWorld' does not specify which model (1.5B or 7B) or which RL baseline produced the trajectories. Add this context for reproducibility.
  5. [Appendix A] The proof of Proposition 1 uses |V| for the number of actions, but the main text refers to tokens. Since the policy is over actions (reason-then-act steps), clarify that the proposition applies to the action distribution, not the token distribution.
  6. [General] Several references are to arXiv preprints from 2025; the paper would benefit from checking whether any have appeared in peer-reviewed venues, but this is not blocking.

Circularity Check

0 steps flagged

No significant circularity: Proposition 1 is proven in Appendix A, the citation to Li [19] is non-load-bearing, and all reported gains are external empirical comparisons.

full rationale

The central identity (Prop. 1) is derived from first principles in Appendix A: the lemma gives ||∇_z log π_k||² = 1−2π_k+Σ_j π_j², and taking the expectation yields E||∇_z log π||² = 1−exp(−H₂(π)). The parenthetical citation to Li [19] ('builds upon established relationships...') is not load-bearing because the full proof is included in the paper. The step-level uncertainty H_t (Eq. 7) is defined as average token Shannon entropy, while Prop. 1 is a Rényi-2 entropy identity over action logits; the paper asserts rather than proves that H_t 'corresponds' to the entropy in Prop. 1. This is an unsupported generalization, and the same gap appears in Section 3.4 / Appendix B where the logit-space identity is applied to parameter-space gradients ∇_θ log π without proof. However, this is a correctness/validity concern, not circularity: it does not define the target result in terms of itself, nor does it fit a parameter and rename it a prediction. Appendix B explicitly concedes that deriving a closed-form extrinsic objective is 'non-trivial' and that the interpretation is 'sufficient to justify the update rule,' so no derivation is disguised as a prediction. Reported gains come from benchmark comparisons against GRPO/DAPO on held-out tasks; hyperparameters ζ, k, k′ are empirically tuned, not presented as predictions. The only self-citation (Li [19]) is therefore non-load-bearing and does not raise the circularity score. The logit-to-parameter Jacobian gap and the Shannon-vs-Rényi entropy mismatch are flagged as omitted proofs / missing support, but they do not constitute circular reasoning.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

The paper introduces no new physical or structural entities; the 'future clarity' is a reward-shaping term, not an invented entity. The central claim rests on the axioms listed, particularly the unverified generalization of Proposition 1 from logit-space to parameter-space, and the ad hoc normalization assumptions.

free parameters (2)
  • ζ (future clarity weight) = 0.05 (WebShop/ALFWorld), 0.1 (Deep Search)
    The hyperparameter controlling the bonus for low-entropy next states; tuned per benchmark and central to the second term of Eq. 8.
  • k and k' (exponential scaling constants) = 1.0 for all experiments
    Set by hand to control how sharply entropy modulates the gradient; used in Eqs. 10 and 11.
axioms (5)
  • standard math Softmax policy and logit-space score function identity (Proposition 1)
    The proof in Appendix A is a direct computation: E||∇z log π||^2 = 1 - exp(-H2). It is used to motivate the method.
  • ad hoc to paper Policy gradient magnitude in parameter space is coupled to entropy in the same way as the logit score
    Section 3.4 generalizes Proposition 1 to the actual policy gradient without proof. Since ∇θ log π = J_z(θ) ∇z log π, the Jacobian can break the entropy coupling.
  • domain assumption Step-level average token entropy is a faithful proxy for decision confidence
    Section 4.1 defines H_t as the mean token-level entropy of a reason-then-act step and asserts it measures uncertainty; no calibration against true confidence is provided.
  • ad hoc to paper Batch normalization of entropies and scaling weights does not bias the gradient estimator
    Eqs. 10 and 12 use batch-wide statistics, making the weight depend on the entire batch rather than the single state; the paper does not analyze the statistical consequences.
  • ad hoc to paper The future clarity bonus behaves like a valid intrinsic reward under the policy gradient theorem
    Appendix B defines r_int = ζ exp(-k' H_norm) and applies the policy gradient theorem while ignoring ∇θ H_norm, and admits no closed-form objective exists for the extrinsic part.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agents." pith.science (2026). https://pith.science/paper/TIXCOYNH

@misc{pith2026250909265,
  author       = {Pith},
  title        = {Pith review of: Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TIXCOYNH}},
  note         = {Machine review of arXiv:2509.09265}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

In long-horizon tasks, recent agents based on Large Language Models (LLMs) face a significant challenge that sparse, outcome-based rewards make it difficult to assign credit to intermediate steps. Previous methods mainly focus on creating dense reward signals to guide learning, either through traditional reinforcement learning techniques like inverse reinforcement learning or by using Process Reward Models for step-by-step feedback. In this paper, we identify a fundamental problem in the learning dynamics of LLMs: the magnitude of policy gradients is inherently coupled with the entropy, which leads to inefficient small updates for confident correct actions and potentially destabilizes large updates for uncertain ones. To resolve this, we propose Entropy-Modulated Policy Gradients (EMPG), a framework that re-calibrates the learning signal based on step-wise uncertainty and the final task outcome. EMPG amplifies updates for confident correct actions, penalizes confident errors, and attenuates updates from uncertain steps to stabilize exploration. We further introduce a bonus term for future clarity that encourages agents to find more predictable solution paths. Through comprehensive experiments on three challenging agent tasks, WebShop, ALFWorld, and Deep Search, we demonstrate that EMPG achieves substantial performance gains and significantly outperforms strong policy gradient baselines. Project page is at https://empgseed-seed.github.io/

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 14 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learning

    cs.LG 2026-05 unverdicted novelty 7.0

    ResRL decouples shared semantics between positive and negative responses in LLM reinforcement learning via SVD-based projection residuals, outperforming baselines including NSR by up to 9.4% on math reasoning benchmarks.

  2. Post-Training Shifts Confidence: A Three-Stage Analysis of How SFT, RL, and OPD Shape CoT Calibration

    cs.CL 2026-07 conditional novelty 6.0

    Post-training changes where confidence is trustworthy in reasoning traces: OPD best gauges difficulty before thinking, SFT best supports early stopping, RL best selects final answers, and confidence should be read onl...

  3. STAPO: Selective Trajectory-Aware Policy Optimization for LLM Agent Training

    cs.AI 2026-07 conditional novelty 6.0

    Normalized entropy locates trajectory-neglect steps; STAPO selectively optimizes them with a trajectory-aware reward and trajectory-independent penalty, beating group-based RL baselines on agent tasks.

  4. Semantic Consistency Policy Optimization for Reinforcement Learning of LLM Agents

    cs.LG 2026-06 unverdicted novelty 6.0

    SCPO recovers step-level credit from successful siblings within rollout groups to reduce semantic inconsistency in group-based RL for LLM agents, matching or exceeding baselines on ALFWorld and WebShop.

  5. Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learning

    cs.AI 2026-05 unverdicted novelty 6.0

    Skill1 trains one policy to jointly evolve skill query generation, re-ranking, task solving, and distillation from a single task-success signal, with low-frequency trends crediting selection and high-frequency variati...

  6. T$^2$PO: Uncertainty-Guided Exploration Control for Stable Multi-Turn Agentic Reinforcement Learning

    cs.AI 2026-05 unverdicted novelty 6.0

    T²PO improves stability and performance in multi-turn agentic RL by using uncertainty dynamics at token and turn levels to guide exploration and avoid wasted rollouts.

  7. AEM: Adaptive Entropy Modulation for Multi-Turn Agentic Reinforcement Learning

    cs.AI 2026-05 unverdicted novelty 6.0

    AEM lifts entropy analysis to the response level and uses a derived uncertainty proxy to rescale advantages, enabling better exploration-exploitation balance and consistent gains over RL baselines on agent benchmarks.

  8. ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learning

    cs.LG 2026-05 unverdicted novelty 6.0

    ResRL boosts LLM reasoning by modulating negative gradients with SVD-based projection residuals from negative samples, outperforming NSR by 9.4% Avg@16 on math benchmarks while preserving diversity across 12 tasks.

  9. Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization

    cs.CL 2025-10 conditional novelty 6.0

    LLM attention maps reveal a preplan-and-anchor pattern, and reweighting RL credit toward the flagged tokens improves math/QA reasoning.

  10. Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimization

    cs.LG 2026-07 conditional novelty 5.0

    CPO uses the log-ratio of reference-guided to vanilla token probabilities as a correctness signal for per-token advantage shaping in RLVR, beating entropy-based methods on math and generalization benchmarks.

  11. Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learning

    cs.AI 2026-05 unverdicted novelty 5.0

    Skill1 trains a single RL policy to co-evolve skill selection, utilization, and distillation in language model agents from one task-outcome reward, using low-frequency trends to credit selection and high-frequency var...

  12. Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learning

    cs.AI 2026-05 unverdicted novelty 5.0

    Skill1 co-evolves skill selection, utilization, and distillation inside a single policy using only task-outcome reward, with low-frequency trends crediting selection and high-frequency variation crediting distillation...

  13. RoboAgent: Chaining Basic Capabilities for Embodied Task Planning

    cs.RO 2026-04 unverdicted novelty 5.0

    RoboAgent chains basic vision-language capabilities inside a single VLM via a scheduler and trains it in three stages (behavior cloning, DAgger, RL) to improve embodied task planning.

  14. AEM: Adaptive Entropy Modulation for Multi-Turn Agentic Reinforcement Learning

    cs.AI 2026-05 unverdicted novelty 4.0

    AEM adaptively modulates response-level entropy in agentic RL to improve credit assignment and exploration-exploitation balance, yielding gains on ALFWorld, WebShop, and SWE-bench.

Reference graph

Works this paper leans on

55 extracted references · 32 linked inside Pith · cited by 10 Pith papers

  1. [1]

    The unreasonable effectiveness of entropy minimization in llm reasoning

    Shivam Agarwal, Zimin Zhang, Lifan Yuan, Jiawei Han, and Hao Peng. The unreasonable effectiveness of entropy minimization in llm reasoning. arXiv preprint arXiv:2505.15134, 2025

    Pith/arXiv arXiv 2025

  2. [2]

    Open deep search: Democratizing search with open-source reasoning agents

    Salaheddin Alzubi, Creston Brooks, Purva Chiniya, Edoardo Contente, Chiara von Gerlach, Lucas Irwin, Yihan Jiang, Arda Kaz, Windsor Nguyen, Sewoong Oh, et al. Open deep search: Democratizing search with open-source reasoning agents. arXiv preprint arXiv:2503.20201, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    Unifying count-based exploration and intrinsic motivation

    Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016

    2016

  4. [4]

    Seed-grpo: Semantic entropy enhanced grpo for uncertainty-aware policy optimization

    Minghan Chen, Guikun Chen, Wenguan Wang, and Yi Yang. Seed-grpo: Semantic entropy enhanced grpo for uncertainty-aware policy optimization. arXiv preprint arXiv:2505.12346, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    Reasoning with exploration: An entropy perspective

    Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. arXiv preprint arXiv:2506.14758, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Mind2web: Towards a generalist agent for the web

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. In Advances in Neural Information Processing Systems, 2023

    2023

  7. [7]

    From novice to expert: Llm agent policy optimization via step-wise reinforcement learning

    Zhirui Deng, Zhicheng Dou, Yutao Zhu, Ji-Rong Wen, Ruibin Xiong, Mang Wang, and Weipeng Chen. From novice to expert: Llm agent policy optimization via step-wise reinforcement learning. arXiv preprint arXiv:2411.03817, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Group-in-group policy optimization for llm agent training

    Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for llm agent training. arXiv preprint arXiv:2505.10978, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    One-shot entropy minimization

    Zitian Gao, Lynx Chen, Joey Zhou, and Bryan Dai. One-shot entropy minimization. arXiv preprint arXiv:2505.20282, 2025

    Pith/arXiv arXiv 2025

  10. [10]

    Pasa: An llm agent for comprehensive academic paper search

    Yichen He, Guanhua Huang, Peiyuan Feng, Yuan Lin, Yuchen Zhang, Hang Li, et al. Pasa: An llm agent for comprehensive academic paper search. arXiv preprint arXiv:2501.10120, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. arXiv preprint arXiv:2011.01060, 2020

    Pith/arXiv arXiv 2011

  12. [12]

    Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    Pith/arXiv arXiv 2023

  13. [13]

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017

    Pith/arXiv arXiv 2017

  15. [15]

    Vineppo: Accurate credit assignment in rl for llm mathematical reasoning

    Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Accurate credit assignment in rl for llm mathematical reasoning. In The 4th Workshop on Mathematical Reasoning and AI at NeurIPS’24, 2024

    2024

  16. [16]

    Empowerment: A universal agent-centric measure of control

    Alexander S Klyubin, Daniel Polani, and Chrystopher L Nehaniv. Empowerment: A universal agent-centric measure of control. In IEEE Congress on Evolutionary Computation, 2005

    2005

  17. [17]

    Natural questions: a benchmark for question answering research

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 2019

    2019

  18. [18]

    Search-o1: Agentic search-enhanced large reasoning models

    Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-o1: Agentic search-enhanced large reasoning models. arXiv preprint arXiv:2501.05366, 2025

    Pith/arXiv arXiv 2025

  19. [19]

    Logit dynamics in softmax policy gradient methods

    Yingru Li. Logit dynamics in softmax policy gradient methods. arXiv preprint arXiv:2506.12912, 2025

    Pith/arXiv arXiv 2025

  20. [20]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations, 2023

    2023

  21. [21]

    Improving multi-step reasoning abilities of large language models with direct advantage policy optimization

    Jiacai Liu, Chaojie Wang, Chris Yuhao Liu, Liang Zeng, Rui Yan, Yiwen Sun, Yang Liu, and Yahui Zhou. Improving multi-step reasoning abilities of large language models with direct advantage policy optimization. arXiv preprint arXiv:2412.18279, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    Policy invariance under reward transformations: Theory and application to reward shaping

    Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In International Conference on Machine Learning, 1999. 12

    1999

  23. [23]

    Curiosity-driven exploration by self-supervised prediction

    Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning, 2017

    2017

  24. [24]

    On measures of entropy and information

    Alfréd Rényi. On measures of entropy and information. In Proceedings of the fourth Berkeley symposium on mathematical statistics and probability, volume 1: contributions to the theory of statistics, 1961

    1961

  25. [25]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  26. [26]

    ByteDance Seed, Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, Zhiqi Lin, Mingxuan Wang, Chengyi Wang, Xiangpeng Wei, Wenyuan Xu, et al. Seed1. 5-thinking: Advancing superb reasoning models with reinforcement learning. arXiv preprint arXiv:2504.13914, 2025

    arXiv 2025

  27. [27]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    Alfworld: Aligning text and embodied environments for interactive learning

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Cote, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, 2021

    2021

  30. [30]

    Entropy-guided sequence weighting for efficient exploration in rl-based llm fine-tuning

    Abdullah Vanlioglu. Entropy-guided sequence weighting for efficient exploration in rl-based llm fine-tuning. arXiv preprint arXiv:2503.22456, 2025

    Pith/arXiv arXiv 2025

  31. [31]

    Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning

    Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning. arXiv preprint arXiv:2506.01939, 2025

    Pith/arXiv arXiv 2025

  32. [32]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In Advances in neural information processing systems, 2022

    2022

  33. [33]

    Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution

    Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I Wang. Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution. arXiv preprint arXiv:2502.18449, 2025

    Pith/arXiv arXiv 2025

  34. [34]

    Webagent-r1: Training web agents via end-to-end multi-turn reinforcement learning

    Zhepei Wei, Wenlin Yao, Yao Liu, Weizhi Zhang, Qin Lu, Liang Qiu, Changlong Yu, Puyang Xu, Chao Zhang, Bing Yin, et al. Webagent-r1: Training web agents via end-to-end multi-turn reinforcement learning. arXiv preprint arXiv:2505.16421, 2025

    arXiv 2025

  35. [35]

    Webwalker: Benchmarking llms in web traversal

    Jialong Wu, Wenbiao Yin, Yong Jiang, Zhenglin Wang, Zekun Xi, Runnan Fang, Linhai Zhang, Yulan He, Deyu Zhou, Pengjun Xie, et al. Webwalker: Benchmarking llms in web traversal. arXiv preprint arXiv:2501.07572, 2025

    Pith/arXiv arXiv 2025

  36. [36]

    Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation

    An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, et al. Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation. arXiv preprint arXiv:2311.07562, 2023

    Pith/arXiv arXiv 2023

  37. [37]

    Hotpotqa: A dataset for diverse, explainable multi-hop question answering

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600, 2018

    Pith/arXiv arXiv 2018

  38. [38]

    Webshop: Towards scalable real-world web interaction with grounded language agents

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. In Advances in Neural Information Processing Systems, 2022

    2022

  39. [39]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023

    2023

  40. [40]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Honglin Yu, Weinan Dai, Yuxuan Song, Xiang Wei, Haodong Zhou, Jingjing Liu, ...

    Pith/arXiv arXiv 2025

  41. [41]

    Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges

    Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. arXiv preprint arXiv:2401.07339, 2024. 13

    Pith/arXiv arXiv 2024

  42. [42]

    Consistent paths lead to truth: Self-rewarding reinforcement learning for llm reasoning

    Kongcheng Zhang, Qi Yao, Shunyu Liu, Yingjie Wang, Baisheng Lai, Jieping Ye, Mingli Song, and Dacheng Tao. Consistent paths lead to truth: Self-rewarding reinforcement learning for llm reasoning. arXiv preprint arXiv:2506.08745, 2025

    Pith/arXiv arXiv 2025

  43. [43]

    Right question is already half the answer: Fully unsupervised llm reasoning incentivization

    Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, and Yatao Bian. Right question is already half the answer: Fully unsupervised llm reasoning incentivization. arXiv preprint arXiv:2504.05812, 2025

    Pith/arXiv arXiv 2025

  44. [44]

    Edge-grpo: Entropy-driven grpo with guided error correction for advantage diversity

    Xingjian Zhang, Siwei Wen, Wenjun Wu, and Lei Huang. Edge-grpo: Entropy-driven grpo with guided error correction for advantage diversity. arXiv preprint arXiv:2507.21848, 2025

    Pith/arXiv arXiv 2025

  45. [45]

    Siren’s song in the ai ocean: A survey on hallucination in large language models

    Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren’s song in the ai ocean: A survey on hallucination in large language models. Computational Linguistics, pages 1–45, 2025

    2025

  46. [46]

    Learning to reason without external rewards

    Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. Learning to reason without external rewards. arXiv preprint arXiv:2505.19590, 2025

    Pith/arXiv arXiv 2025

  47. [47]

    Deepresearcher: Scaling deep research via reinforcement learning in real-world environments

    Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments. arXiv preprint arXiv:2504.03160, 2025

    Pith/arXiv arXiv 2025

  48. [48]

    Maximum entropy inverse reinforcement learning

    Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In The Association for the Advancement of Artificial Intelligence, 2008

    2008

  49. [49]

    T−1X t=0 (∇θ logπ θ(at|st))r int t (st+1;θ) # (20) =E τi∼πθ

    Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, et al. Ttrl: Test-time reinforcement learning. arXiv preprint arXiv:2504.16084, 2025. 14 Appendix A Proof of Proposition 1 We aim to prove that Eak∼π ||∇z logπ k||2 = 1− P|V| j=1 π2 j . The proof requires the result for the gradient norm of...

    Pith/arXiv arXiv 2025

  50. [50]

    Direct Answer Filtering:We sampled 5 results per question using Doubao-Seed-1.6 (Thinking) [ 26]. We then filtered out all questions that could be answered directly (where at least one of the 5 results was correct) to ensure the agent learns to use its search tools rather than relying on memorized answers

  51. [51]

    stably all-correct

    Agent Workflow Filtering:We further filtered the dataset by sampling 8 results using a search workflow built on Doubao-Seed-1.6 (Thinking). We removed data points that were "stably all-correct" to focus the RL training on more challenging instances and improve training efficiency. The key hyperparameters for the RL training on the Deep Search task are det...

  52. [52]

    assistant

    Step-Level Entropy Collection:The function first iterates through the batch of trajectories to identify agent action steps (i.e., the “assistant” responses). For each step t, it computes the corresponding step-level entropy Ht by averaging the policy’s token-level entropies for that action

  53. [53]

    Modulation Component Calculation:All collected step entropies {Ht} are normalized across the batch using min-max scaling to produce {Hnorm,t} (as per Eq. 12). These normalized values are then used to compute the two key components of our method: the self-calibrating scaling factor g(Ht) (Eq. 10) and the future clarity bonus termg ′(Ht+1)(Eq. 11)

  54. [54]

    For each step, the advantage is scaled by g(Ht) and augmented by the future clarity bonus ζ·g ′(Ht+1), yielding the modulated advantageA mod as defined in our main formula (Eq

    Advantage Modulation:The function then applies these components to the original outcome-based advantage. For each step, the advantage is scaled by g(Ht) and augmented by the future clarity bonus ζ·g ′(Ht+1), yielding the modulated advantageA mod as defined in our main formula (Eq. 8)

  55. [55]

    assistant

    Final Normalization:Finally, to reduce variance and ensure stable training, the entire batch of resulting modulated advantages is normalized to have a mean of zero. This produces the final advantage Afinal (Eq. 13) that is used to compute the policy gradient. 19 0 20 40 60 80 100 120 140 RL Training Steps 0.0 0.2 0.4 0.6 0.8 1.0Success Rate GRPO (Baseline...

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.