REVIEW 4 major objections 6 minor 18 references
Rewarding LLM agents for predicting their next observation backfires catastrophically under GRPO: every trained agent collapses into a degenerate 'dark room' state — and the culprit is the standard-deviation term in advantage normalization.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 07:54 UTC pith:7C53S2RC
load-bearing objection Genuine and practically important failure mode, with a clean but conditional theory; the main causal claim needs an extra control before it's fully pinned. the 4 major comments →
The Dark Room in the Reward Channel: Dense Prediction Rewards Collapse GRPO-Trained LLM Agents -- and What Actually Works
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central discovery is that a potential-based prediction reward, bounded at the return level exactly as the shaping literature promises, is transformed by GRPO's standard-deviation normalization into an unbounded force that drives the policy into a degenerate absorbing 'dark room' state. The mechanism is captured by Proposition 1: in a group where every trajectory fails the task, the z-scored advantage is invariant to the shaping coefficient λ, so the bounded reward becomes full-scale pressure no matter how small λ is. Proposition 2 generalizes this into a variance-profile criterion: what z-scoring amplifies is a dense signal's within-group variance at saturation, so signals whose variance
What carries the argument
The argument is carried by GRPO's group advantage estimator, which normalizes each group's returns by subtracting the mean and dividing by the standard deviation. Proposition 1 (λ-invariance) shows that when a group contains only failed trajectories, all task returns are equal and the shaping term is the sole within-group variation; after z-scoring, the normalized advantage is independent of the shaping coefficient λ, so reducing the reward's size does not reduce the gradient pressure. Proposition 2 (variance-profile criterion) states that the amplifier only stops when the signal's within-group variance vanishes at saturation — meaning the safety of a dense reward is governed by its variance
Load-bearing premise
The collapse is blamed on the premise that within every all-fail training group, the only thing that differs between trajectories is the tiny prediction-shaping term; if other factors (invalid-action penalties, length differences, stochasticity) create comparable within-group variation, the λ-invariance proof and the causal story weaken.
What would settle it
During the first 30 training steps, record per-trajectory returns for groups where every episode failed and compute the within-group standard deviation; decompose it into the λ-scaled shaping component versus all other sources (penalties, length, noise). If the non-shaping component is not negligible, the invariance premise is empirically violated. Alternatively, run the identical GRPO setup with group size 16 or 32 so that all-fail groups become rare; if the collapse still occurs, the all-fail-group mechanism is not sufficient.
If this is right
- Boundedness at the return level does not make a dense shaping reward safe under GRPO; the advantage normalizer can amplify a λ-scaled term into full-scale gradients.
- Removing GRPO's std normalization (mean-only) is sufficient to rescue the policy from collapse, turning 0% into baseline parity (51.6%) and replicating the pathology-free regime.
- For sparse-success LLM agents, the auxiliary-loss channel is a safer path for dense prediction signals: identical signal content gains ~20 points over baseline, while every reward-channel variant is at best neutral.
- The variance-profile criterion provides a screening rule for dense rewards under group normalization: prefer signals whose within-group variance decays at mastery; avoid potential-difference signals whose per-step variance persists.
- Collapse is irreversible once it occurs (constant pressure and annealing both fail), so detection must be preventive; the joint precursor (entropy decline ∧ prediction saturation ∧ length pinning) warns 15–30 steps before the turn.
Where Pith is reading between the lines
- The λ-invariance mechanism should generalize beyond prediction rewards: any bounded, policy-dependent shaping term that retains within-group variance in all-fail groups should produce the same collapse under GRPO — a directly testable prediction.
- If the preregistered disjoint-vocabulary placebo also gains ~20 points, the apparent value of auxiliary prediction losses may be a regularization/compute effect rather than world-model information transfer, which would reframe many auxiliary-objective successes.
- A practical mitigation implied by the analysis but not yet run: increasing group size so that all-fail groups become rare should dilute the amplifier; this would both test the mechanism and potentially serve as a fix.
- An offline audit could apply the variance-profile criterion to existing dense rewards by computing within-group variance vs. training progress, flagging signals that keep variance at saturation before training runs.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies the effect of adding a potential-based next-observation prediction reward (a 'PS signal') to GRPO training of Qwen3-1.7B/4B/8B agents in ALFWorld. It reports that every std-normalized run collapses into a degenerate absorbing state: prediction accuracy -> 1.0, task success -> 0, episode length pinned at the horizon. A single-flag ablation (mean-only vs. std normalization) rescues the PS arm to baseline parity. The authors prove a parameter-free λ-invariance result for all-fail groups under the assumption that all task returns in a group are equal to a constant, and use it to propose a variance-profile criterion that predicts which dense signals are safe under std normalization. A signal-delivery matrix compares reward-channel vs. auxiliary-loss-channel delivery of the same signal, finding the loss channel gains ~20 points, with a shuffled-gold placebo matching the true-gold arm. The paper is explicitly a single-seed preprint with preregistered replications in progress.
Significance. If the reported collapse and rescue replicate, the paper identifies an important and practically relevant failure mode: return-level boundedness guarantees for potential-based shaping do not survive group-relative advantage normalization in sparse-success settings. The λ-invariance proposition (Proposition 1) is a clean parameter-free derivation, and the controlled channel matrix, placebo controls, and preregistration ledger are methodologically strong. The paper is unusually candid about its limitations. However, the current evidence base is single-seed, and a central assumption of Proposition 1 is contradicted by the experiment's own invalid-action penalty and early-termination configuration. These issues prevent the claims from being accepted as stated.
major comments (4)
- [§4.3, Eq. (3), and Appendix C] Proposition 1 assumes R_i = C + λs_i for all trajectories in an all-fail group. The experimental configuration includes an invalid-action penalty of 0.1 (Appendix C) and allows early termination. In any all-fail group with different numbers of invalid actions or different lengths, the non-shaping component of return is not constant, so σ_R = sqrt(λ²σ_s² + σ_other²) and the advantage in Eq. (3) is not λ-invariant. The paper asserts (§§4.2, 5) that the shaping term is the 'only within-group variation' in all-fail groups, but no measurement of σ_other is provided; since the penalty magnitude (0.1) equals λ, the std amplifier may be amplifying penalty/termination noise as much as shaping. The no-signal std baseline (49.5%, Table 2) shows the prediction signal is necessary for the collapse, but not that the Proposition-1 mechanism is the operative one. Please add an ablation with no invalid-a
- [§4.4 and Proposition 2] The collapse's terminal condition is 'prediction accuracy -> 1.0'. If the rewarded score Φ_t saturates to a constant, the per-step reward r_pred(t)=λ(Φ_t−Φ_{t−1}) becomes zero and the within-group variance σ_s would vanish, which by Proposition 2 should starve the amplifier and prevent the absorbing state. To explain the observed collapse, §4.4 asserts that 'potential-difference scoring keeps fluctuating per step even at prediction accuracy≈1.0', but the manuscript does not define how the logged prediction accuracy relates to the rewarded Φ_t. If Φ_t is a partial-credit F1-like score, this should be stated and the variance trajectory σ_s should be plotted; if Φ_t saturates to a constant, the mechanism is internally inconsistent. This is load-bearing for Proposition 2's retrodictive claims.
- [§9 and Table 2] All endpoints are single-seed last-6 validation means, with the text quoting ±8.4 pt single-point noise. Several load-bearing comparisons—the net-attribution ≈0 (§5), the shuffled-gold vs. true-gold loss-channel comparison (76.0 vs. 69.3, §7), and the mean-only vs. decoupled gap (51.6 vs. 31.2, §5)—are descriptively assessed without seed replication. The authors' disclosure is honest and the preregistration is exemplary, but the central empirical claims (three-scale collapse and std-removal rescue) need replication before the title-level conclusion can be accepted. I would ask for at least the collapse/rescue arms under seeds 42/96 to be included in the revision.
- [§5 and Fig. 3] The 'advantage-level dose–response' plots three points from three different algorithms (std full-scale, decoupled capped, mean-only) on a common x-axis. The x-axis values are not uniformly measured: the decoupled cap reports 0.066–0.077, while the mean-only arm's raw advantage magnitude is not reported, and the resulting ordering is non-monotonic (mean-only at ~0.1 gives 51.6%, decoupled at ~0.07 gives 31.2%). As presented this is a schematic comparison, not a dose–response curve; please report measured per-arm advantage magnitudes or relabel it accordingly.
minor comments (6)
- [Abstract and §1] The phrase 'every run' overstates the evidence; all runs are single-seed. Please qualify as 'every run in our single-seed trials'.
- [§9] The 8B baseline anomaly (32.8% vs. 49.5% at 4B) is unexplained. Consider reporting full validation curves for the baselines so readers can judge the scale ordering.
- [§3] Please specify whether validation episodes use the same group structure as training and whether the invalid-action penalty applies identically to all arms.
- [Table 2] The baseline row's 'std' normalization may confuse readers because std normalization alone does not collapse; a one-sentence explanation in the caption would help.
- [§4.7] The three false-alarm counterexamples are useful, but the reader should be told how these counterexamples were selected—exhaustively from all runs, or cherry-picked?
- [Appendix A] Please add dates and version identifiers for the SHA256-digest preregistration entries.
Circularity Check
No significant circularity: Proposition 1 is a parameter-free consequence of the GRPO estimator, and the empirical claims rest on ablations and preregistered predictions rather than on fitted inputs or self-citation.
full rationale
The paper's core derivation chain is self-contained. Proposition 1 is proven directly from the definition of the GRPO normalized advantage (Eq. 1) and the all-fail-group assumption R_i = C + λ s_i; it is algebra, not a fitted parameter or a restatement of the experimental outcome. Proposition 2 is the same formula in the saturation limit and is used to organize retrodictive and prospective predictions; the paper explicitly labels compatibility checks as 'not an independent test' and reports ledger outcomes, including falsified predictions (e.g., weaker-model-collapses-earlier). The causal localization ('0%→51.6%') is a single-factor ablation (std normalization on/off), with a no-signal mean-only control, which is an interventionist comparison rather than a circular definition. There are no load-bearing self-citations: the only self-identifying item is the Zenodo deposit of an earlier version, cited only as provenance, not as evidence. The concern that the invalid-action penalty and early termination could violate the constant-task-return premise is a robustness/correctness issue about the empirical scope of Proposition 1, not a circularity of the derivation itself. No fitted input is renamed as a prediction; no uniqueness theorem is imported from the authors' prior work; no ansatz is smuggled via citation. Thus the circularity score is 0.
Axiom & Free-Parameter Ledger
free parameters (4)
- shaping coefficient λ =
0.1
- group size n=4 =
4
- GRPO std floor epsilon =
1e-6
- validation window (last-6 means) =
6
axioms (3)
- domain assumption In all-fail groups, all task returns are equal to a constant C
- domain assumption The GRPO advantage estimator is exactly A_i = (R_i - mean(R))/std(R) + eps with group-level normalization
- standard math The potential-difference reward telescopes exactly as Eq. (2) states
invented entities (1)
-
The 'variance-profile criterion' as a safety rule for dense signals
no independent evidence
read the original abstract
Dense per-step supervision is an appealing remedy for sparse-reward, long-horizon LLM agents: reward the agent for predicting its next observation, and memory should follow. We show that under group-normalized RL (GRPO), this recipe does not merely fail -- it destroys the policy. Across Qwen3-1.7B/4B/8B on ALFWorld, a potential-based prediction reward drives every run into a degenerate absorbing state (prediction accuracy -> 1.0, task success -> 0,episode length pinned at the horizon): the "dark room" pathology, built automatically by the optimizer. A single-factor ablation localizes the cause -- removing only GRPO's std normalization turns the same reward from catastrophic (0%) into baseline parity -- and a two-line proposition explains why: in all-fail groups the z-scored advantage is invariant to the shaping coefficient, so bounded rewards become unbounded pressure and annealing cannot help. Our central insight generalizes this: what z-scoring amplifies is a dense signal's within-group variance while all-fail groups dominate, so signals whose variance decays by mastery are structurally amplifier-safe.This variance-profile criterion retrodicts our collapses, carries preregistered predictions for arms that had not yet run, and is consistent with published reward-channel successes (a compatibility check, not an independent test). Finally, a controlled signal-delivery matrix (identical signal, varying only the consumption mechanism) shows the reward channel is at best neutral while the auxiliary-loss channel gains ~20 points -- and a shuffled-gold placebo matches the true-gold arm, so the gap survives without correct labels. Endpoints are single-seed; seed replication and group-size controls are preregistered and in progress.
Figures
Reference graph
Works this paper leans on
-
[4]
Forbes, Leonardo Villalobos-Arias, Jianxun Xu, Colin M
Grant C. Forbes, Leonardo Villalobos-Arias, Jianxun Xu, Colin M. Potts, Arnav Jhala, and David L. Roberts. Potential-based intrinsic motivation: Preserving optimality with complex, non- Markovian shaping rewards. InarXiv preprint arXiv:2402.07411,
-
[6]
Jian Hu et al. REINFORCE++: Stabilizing critic-free policy optimization with global advantage normalization.arXiv preprint arXiv:2501.03262,
-
[7]
Yuki Ichihara et al. MO-GRPO: Mitigating reward hacking of group relative policy optimization on multi-objective problems.arXiv preprint arXiv:2509.22047,
-
[8]
Shih-Yang Liu et al. GDPO: Group reward-decoupled normalization policy optimization for multi- reward RL optimization.arXiv preprint arXiv:2601.05242,
-
[10]
Policy and world modeling co-training for language agents.arXiv preprint arXiv:2606.02388,
Ning Lu et al. Policy and world modeling co-training for language agents.arXiv preprint arXiv:2606.02388,
-
[12]
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. DeepSeekMath: Pushing the limits of math- ematical reasoning in open language models.arXiv preprint arXiv:2402.03300,
-
[13]
ECHO: Terminal agents learn world models for free.arXiv preprint arXiv:2605.24517,
Vaishnavi Shrivastava, Piero Kauffmann, Ahmed Awadallah, and Dimitris Papailiopoulos. ECHO: Terminal agents learn world models for free.arXiv preprint arXiv:2605.24517,
-
[14]
Claas V oelcker, Tyler Kastner, Igor Gilitschenski, and Amir-massoud Farahmand. When does self-prediction help? Understanding auxiliary tasks in reinforcement learning.arXiv preprint arXiv:2406.17718,
-
[15]
Yibin Wang, Zhimin Li, Yuhang Zang, Yujie Zhou, Jiazi Bu, Chunyu Wang, Qinglin Lu, Cheng Jin, and Jiaqi Wang. Pref-GRPO: Pairwise preference reward-based GRPO for stable text-to-image reinforcement learning.arXiv preprint arXiv:2508.20751, 2025d. Zihan Wang et al. RAGEN: Understanding self-evolution in LLM agents via multi-turn reinforce- ment learning.ar...
-
[16]
13 Preprint. Under review. Changyi Xiao et al. BNPO: Beta normalization policy optimization.arXiv preprint arXiv:2506.02864,
-
[17]
Reinforcement world model learning for LLM-based agents.arXiv preprint arXiv:2602.05842,
Xiao Yu et al. Reinforcement world model learning for LLM-based agents.arXiv preprint arXiv:2602.05842,
-
[18]
Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. ArCHer: Training lan- guage model agents via hierarchical multi-turn RL.arXiv preprint arXiv:2402.19446,
-
[2012]
Reward shaping to mitigate reward hacking in RLHF.arXiv preprint arXiv:2502.18770,
Jiayi Fu et al. Reward shaping to mitigate reward hacking in RLHF.arXiv preprint arXiv:2502.18770,
-
[2021]
Michael Bereket and Jure Leskovec. Uncalibrated reasoning: GRPO induces overconfidence for stochastic outcomes.arXiv preprint arXiv:2508.11800,
-
[2022]
Youssef Mroueh. Reinforcement learning with verifiable rewards: GRPO’s effective loss, dynamics, and success amplification.arXiv preprint arXiv:2503.06639,
-
[2024]
The entropy mechanism of reinforcement learning for reasoning language models
Ganqu Cui et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617, 2025a. Ganqu Cui et al. Process reinforcement through implicit rewards.arXiv preprint arXiv:2502.01456, 2025b. DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learn- ing.arXiv preprint arXi...
-
[2025]
Carson E. Denison et al. Sycophancy to subterfuge: Investigating reward-tampering in large lan- guage models.arXiv preprint arXiv:2406.10162,
-
[2026]
Understanding R1-Zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783,
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding R1-Zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783,
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.