Pith. sign in

REVIEW 2 major objections 6 minor 1 cited by

Replacing Gaussian actors with discretized categorical actors yields consistent gains in on-policy reinforcement learning for continuous control.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 06:13 UTC pith:ZYQ6RPKA

load-bearing objection Useful empirical result on categorical actors for PPO; SOTA claim needs missing discrete-actor baselines or softening. the 2 major comments →

arxiv 2601.23075 v2 pith:ZYQ6RPKA submitted 2026-01-30 cs.LG cs.RO

RN-D: Discretized Categorical Actors for On-Policy Reinforcement Learning

Yuexin Bian , Jie Feng , Tao Wang , Yijiang Li , Sicun Gao , Yuanyuan Shi This is my paper
classification cs.LG cs.RO
keywords discretized categorical actoron-policy reinforcement learningPPOcontinuous controlcross-entropy losspolicy gradient varianceresidual networksample efficiency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper's central claim is that the standard Gaussian actor is a limiting choice for on-policy reinforcement learning in continuous control. By discretizing each action dimension into a fixed number of bins and treating the policy as a categorical distribution, the actor update becomes a cross-entropy-like objective, which the authors show to have bounded gradient variance. Pairing this with a residual network that uses pre-layer normalization — the RN-D architecture — yields consistent gains in final performance and sample efficiency over standard PPO with Gaussian MLP actors, while the critic and PPO objective remain unchanged. The authors provide a variance identity (Proposition 4.1) and a loss-swap ablation to trace the gain to the categorical likelihood rather than to discretization alone. If correct, this establishes actor representation and architecture as a powerful, underused lever for stabilizing on-policy optimization.

Core claim

The central discovery is that actor parameterization and network architecture are first-class design choices in on-policy RL, not minor details. The paper shows that a factorized categorical policy over uniformly discretized action bins, combined with a residual feedforward network with pre-layer normalization, consistently outperforms standard Gaussian actors with MLP policies across locomotion and manipulation benchmarks. The key mechanism is the cross-entropy-like objective induced by the categorical likelihood, which has dramatically lower and more stable policy-gradient variance than the weighted squared-error objective of a Gaussian actor — especially as the Gaussian's standard deviati

What carries the argument

The central object is the factorized categorical actor over K uniformly spaced action bins per dimension, trained under the PPO clipped surrogate. The key identity is the gradient-variance bound in Proposition 4.1: for a categorical policy, E||g||^2 = R^2 Σ_i (1 - ||p_i||^2) ≤ mR^2(1 - 1/K), bounded above by the uniform distribution, whereas for a Gaussian policy E||g||^2 = R^2 Tr(Σ^{-1}), which grows without bound as the variance shrinks. This identity, together with the pre-layer-norm residual network, carries the argument: the categorical likelihood turns the actor update into a weighted cross-entropy loss over bins, which is stable and scalable.

Load-bearing premise

The 'state-of-the-art' claim rests on the choice of baselines and the fixed bin count K=41; if stronger discrete-actor baselines or a different bin count are used, the central claim may narrow.

What would settle it

Compare RN-D directly with the ordinal and unimodal discretized actor baselines cited in the related work under identical hyperparameters; if RN-D does not beat them, the 'state-of-the-art' claim is false. Alternatively, run the loss-swap ablation with a well-tuned Gaussian variance in the swapped actor; if the performance gap narrows, the categorical likelihood is not the necessary ingredient.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Replacing only the actor network with RN-D, keeping critic and PPO objective fixed, yields better final performance and 1.3–1.9× faster convergence across tasks.
  • The categorical likelihood, not discretization alone, drives the gains; the loss-swap ablation shows that a squared-error objective on discretized actions loses most of the improvement.
  • Wider RN-D actors converge faster without hurting final performance, suggesting that the categorical objective can exploit increased capacity.
  • The number of bins is a sensitive hyperparameter: performance degrades when K exceeds a few hundred, so moderate granularity (about 11–101 bins) is best.
  • Policy-gradient variance is much lower and more stable for categorical actors than Gaussian actors throughout training.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The variance bound suggests categorical actors should be most beneficial in the late phase of training, when Gaussian policies' covariance collapses; a testable prediction is that the gap widens as training progresses.
  • Since the actor update becomes a weighted cross-entropy loss, techniques from supervised classification — label smoothing, temperature scaling, or adaptive binning — could be transferred to on-policy RL and may further improve performance.
  • The paper's fixed bin count of 41 is a tuned value; an adaptive or learned action tokenizer might preserve precision at higher bin counts, which the authors themselves suggest as future work.
  • The categorical actor may be particularly well-suited for high-dimensional or vision-based control, where the CNN encoder plus discrete logits resembles classification heads; this connection is worth exploring beyond the tasks tested.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper proposes RN-D, a drop-in replacement for the Gaussian actor in on-policy PPO: each action dimension is discretized into K=41 bins and represented by a per-dimension categorical distribution, paired with a pre-layer-norm residual MLP (or CNN) backbone. The critic, PPO objective, and other training components are kept fixed. The authors report consistent gains over standard PPO with MLP-Gaussian actors across MuJoCo locomotion and ManiSkill state- and vision-based tasks, and claim state-of-the-art performance within on-policy RL. They also present a gradient-variance analysis (Proposition 4.1), a loss-swap ablation isolating the categorical likelihood, and scaling studies for actor width and bin count.

Significance. If the empirical claim is accepted, RN-D is a practically useful, simple modification: it improves final performance and sample efficiency over standard PPO with an MLP-Gaussian actor while leaving the critic and objective fixed. The paper includes useful ablations (regularized vs. plain MLP, discrete vs. Gaussian likelihood, loss-swap, width scaling, bin scaling) and releases code. Proposition 4.1 provides a clean, self-contained variance calculation for a toy estimator. However, the headline claim is broader than what is measured: the paper does not compare against the prior discrete/unimodal actor parameterizations it cites (Tang & Agrawal 2020; Zhu et al. 2024), nor against Beta/Bernoulli actors or PPO-CMA. The evidence supports a narrower claim of improvement over standard PPO with MLP-Gaussian actors, not state-of-the-art within on-policy RL.

major comments (2)
  1. [Abstract and §1; §6; Appendix C.1] The claim that RN-D 'achieves state-of-the-art results within on-policy RL' is not supported by the experimental scope. The baselines are MLP-C (PPO with Gaussian MLP actor), RN-C (Gaussian with the regularized network), MLP-D (plain-MLP categorical actor), and, in the appendix, TD3 and TD-MPC2 (off-policy / model-based). The paper explicitly cites prior on-policy actor parameterizations that are designed to improve over Gaussian PPO—Tang & Agrawal 2020 (ordinal/factorized categorical), Zhu et al. 2024 (unimodal discrete), Beta actors, Bernoulli actors, PPO-CMA—but does not evaluate any of them. Since RN-D changes both the representation (categorical bins) and the architecture (pre-LN residual MLP), the provided comparisons do not establish superiority over those methods. Please either add the missing on-policy baselines or revise the contribution claim to 'improves over standard PPO wit
  2. [§7 and Appendix C.4] The main results use K=41 bins throughout, but Appendix C.4 shows that performance is sensitive to bin count and degrades markedly for larger K on the tasks examined. The paper notes that 'discrete-policy performance is sensitive to bin granularity' and that performance drops when bins exceed roughly 100. This is a disclosed limitation, but it directly qualifies the 'consistent gains' and 'state-of-the-art' claims: the reported gains are for a particular K, and the method appears to need a moderate bin range to preserve the benefit. The manuscript should state this dependency in the main text and should not imply that the choice of K is immaterial. Also, the text in §7 says the sweep is 'from 11 to 1001' while Appendix C.4 says 'from 21 to 1001'; please correct the inconsistency.
minor comments (6)
  1. [Abstract] Grammar: 'yields consistent gains and achieve' should be 'achieves'.
  2. [§5, Relation to Prior Architectures] The reference is given as '(Lee et al., 2024; Lee et al.)' with an incomplete second citation; add the year or correct the citation.
  3. [§4.1, Proposition 4.1] The variance comparison is between the gradient with respect to the Gaussian mean and the gradient with respect to categorical logits. These are different parameter spaces, and the constant-return assumption makes it a motivational toy result rather than an optimization guarantee. This is fine, but the text should avoid implying that this proposition alone explains PPO training differences.
  4. [Appendix C.2] The loss-swap ablation is described qualitatively ('eliminates most of the performance improvement'/'closely matching the continuous Gaussian baseline') without reporting numeric scores or confidence intervals. Please provide the actual aggregate numbers or a table so the reader can judge the magnitude.
  5. [Appendix C.1] TD-MPC2 is run on an 'NVIDIA A200 GPU, which is typically faster,' while TD3 and the proposed method run on different hardware. This claim is not substantiated and the GPU names are unusual; please report exact hardware and clock times, or state that the comparison is not controlled.
  6. [§7 and Appendix C.1] The phrase 'state-of-the-art baselines' is used for TD3 and TD-MPC2, which are off-policy or model-based methods; for an on-policy SOTA claim, the relevant baselines are on-policy. Please clarify the comparison scope.

Circularity Check

0 steps flagged

No significant circularity: central empirical claim is benchmarked externally; only a minor non-load-bearing self-citation.

full rationale

The paper's derivation chain is not circular. The main claim (RN-D improves over standard PPO with Gaussian MLP actors) is established by direct head-to-head experiments against MLP-C, RN-C, and MLP-D under an identical critic, plus external comparisons to TD3 and TD-MPC2. No performance number is obtained by fitting a parameter and then 'predicting' a closely related quantity: the ablations (loss swap, actor width, bin count) manipulate one factor at a time and report measured returns. Proposition 4.1 is an independent mathematical identity (closed-form second moments of REINFORCE gradients); it is used only as motivation and is not fitted to the empirical results. The residual pre-LN architecture is explicitly borrowed from SimBa/BroNet with the disclaimer that 'our contribution does not lie in introducing a novel network module,' so no ansatz is smuggled in via self-citation. The only self-citation is Wang et al. (2025) in the Introduction/Related Work, co-authored by Tao Wang and Sicun Gao; it serves as background about value estimation and is not load-bearing for the RN-D result. The 'state-of-the-art within on-policy RL' wording in the Abstract/Introduction is broader than the evaluated baseline set (Section 6 compares only PPO variants plus off-policy TD3/TD-MPC2; prior discrete/unimodal actors such as Tang & Agrawal 2020 and Zhu et al. 2024 are cited in Related Work but not run), and Appendix C.4 shows sensitivity to bin count; these are comparison-scope/limitation concerns, not circular reductions. Under the stated rules, this is a normal non-circular empirical paper, with a minor self-citation that does not affect the central derivation.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

The paper introduces no new physical or mathematical entity. Its central dependencies are the hand-chosen bin count K=41, the residual-network configuration, and the assumption that the toy variance model and loss-swap ablation isolate the mechanisms claimed.

free parameters (2)
  • Number of action bins K = 41
    Chosen by hand for all main experiments; Appendix C.4 shows performance degrades as K grows and is sensitive to this value, so the central gains may depend on this choice.
  • Actor network width and block count (dh, N) = Gym: dh=256, N=2; ManiSkill-state: dh=128, N=2; ManiSkill-RGB: dh=512, N=1
    Architectural configuration selected by the authors and varied per benchmark family; Appendix C.3 shows width affects learning speed and sometimes final performance.
axioms (5)
  • domain assumption The PPO clipped surrogate objective with GAE is an adequate and representative on-policy optimization framework for the claim.
    The paper evaluates only PPO-style training and generalizes its conclusions to on-policy RL without testing other on-policy algorithms.
  • domain assumption A factorized categorical policy with conditional independence across action dimensions (Eq. 7) is a sufficient policy class for continuous control.
    All discrete-actor results rely on the factorized softmax structure, which discards cross-dimension correlations.
  • ad hoc to paper The one-step, constant-return variance model in Section 4.1 captures the optimization-relevant variance of real PPO training.
    The toy setting assumes constant return independent of the action, which does not hold in actual control; the paper uses it to motivate, not prove, the empirical advantage.
  • domain assumption Uniform discretization into K=41 bins preserves enough action precision for the benchmark tasks.
    The paper uses uniform binning and flags in C.4 that very fine binning degrades performance; the adequacy of 41 bins is an empirical assumption.
  • domain assumption The loss-swap ablation isolates the likelihood objective while holding the action representation fixed.
    The ablation requires that the rollout/sampling procedure and all other training details remain otherwise identical; the paper does not fully specify how actions are sampled for the loss-swapped Gaussian policy.

pith-pipeline@v1.3.0-alltime-deepseek · 16524 in / 13540 out tokens · 159517 ms · 2026-08-03T06:13:17.457557+00:00 · methodology

0 comments
read the original abstract

On-policy Reinforcement Learning (RL) remains a dominant paradigm for continuous control, yet standard implementations rely on Gaussian actors and relatively shallow MLP policies, often leading to brittle optimization when gradients are noisy, and policy updates must be conservative. In this paper, we revisit actor policy representation as a first-class design choice for on-policy RL. We study discretized categorical actors, which represent each action dimension as a distribution over discrete bins and induce a policy objective analogous to classification cross-entropy loss. Building on architectural advances from supervised learning, we further pair discretized categorical actors with regularized networks, yielding RN-D. Across diverse continuous-control benchmarks, we show that simply replacing the standard Gaussian actor with our proposed actor substantially improves performance, achieving state-of-the-art results within on-policy RL. We release our code at https://github.com/alwaysbyx/RND-RL.

Figures

Figures reproduced from arXiv: 2601.23075 by Jie Feng, Sicun Gao, Tao Wang, Yijiang Li, Yuanyuan Shi, Yuexin Bian.

Figure 1
Figure 1. Figure 1: Understanding Continuous Gaussian and Discrete cate￾gorical actor policy from two perspective. 4.1. Policy-gradient estimator and variance On-policy policy gradient. To analyze the stochastic gra￾dient structure underlying PPO, we consider the policy￾gradient term obtained by differentiating the loss terms in (2) with respect to θ: g(θ) ≜ E(st,at)∼πθold h wt(θ) Aˆ t ∇θ log πθ(at | st) i , (8) where wt(θ) d… view at source ↗
Figure 2
Figure 2. Figure 2: Proposed Regularized Network for Discrete action poli￾cies (RN-D). The actor consists of a feature extractor (MLP or CNN) followed by pre-LayerNorm residual MLP blocks, enabling stable optimization and improved scalability. The output layer produces categorical logits over K bins for each action dimension. categorical actors, we hypothesize that the classification￾style structure of the categorical update … view at source ↗
Figure 3
Figure 3. Figure 3: Aggregate learning curves across benchmarks. Each subplot reports the mean performance over tasks within a benchmark (MuJoCo locomotion: normalized return; ManiSkill: success rate) as a function of environment steps. Curves are averaged over 5 random seeds; shaded regions denote 95% stratified bootstrap confidence intervals. The red annotations indicate a sample-efficiency speedup. where LN(·) denotes laye… view at source ↗
Figure 4
Figure 4. Figure 4: The evolution of the policy-gradient variance over train￾ing (log scale). Component Analysis. We visualize the gradient signal￾to-noise ratio (SNR) and normalized return on Gym locomo￾tion tasks, where SNR is computed as the squared norm of the mean policy gradient divided by its variance across mini￾batches during training. Following prior work (Andrychow￾icz et al., 2020), we report the 95th-percentile p… view at source ↗
Figure 5
Figure 5. Figure 5: Component Analysis. (a) Gradient signal-to-noise ratio (SNR) on Gym locomotion tasks. Bars show the mean and error bars denote one standard deviation across tasks. (b) Average nor￾malized return. Higher SNR correlates with higher returns, with RN-D achieving the best performance. 7 [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Extended studies on sample efficiency, optimization objective, and scaling. 7. Extended Study Additional Comparisons with State-of-the-Art Meth￾ods. In Appendix C.1, we further compare our method to strong state-of-the-art baselines, TD3 (Fujimoto et al., 2018) and TD-MPC2 (Hansen et al., 2024), on Humanoid￾v4 and StackCube-v1. In [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Extended studies on sample efficiency, optimization objective, and scaling. C.1. Additional Comparisons with State-of-the-Art Methods To complement our main evaluation, we compare against two recent state-of-the-art continuous-control methods: TD3 (Fuji￾moto et al., 2018) and TD-MPC2 (Hansen et al., 2024). Since these methods differ in their interaction patterns and training pipelines, we report wall-clock… view at source ↗
Figure 8
Figure 8. Figure 8: Complete learning curves across all tasks. We report normalized return on MuJoCo benchmarks (top row) and success rate on ManiSkill manipulation benchmarks (remaining rows), including both state-based and RGB-based variants. Curves are averaged over 5 random seeds; shaded regions denote 95% stratified bootstrap confidence intervals. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Low-Rank Adaptation for Critic Learning in Off-Policy Reinforcement Learning

    cs.LG 2026-04 unverdicted novelty 5.0

    LoRA applied to critics in SAC and FastTD3 reduces critic loss and yields best or competitive policy performance on most evaluated tasks.

Reference graph

Works this paper leans on

6 extracted references · 4 linked inside Pith · cited by 1 Pith paper

  1. [1]

    mean).Let πc(a|s) =N(µ,Σ) with fixed diagonal Σ = Diag(σ2) and parameter µ∈R m

    Gaussian policy (gradient w.r.t. mean).Let πc(a|s) =N(µ,Σ) with fixed diagonal Σ = Diag(σ2) and parameter µ∈R m. The log-density is logπ c(a|s) =− 1 2 (a−µ) ⊤Σ−1(a−µ) +C, hence ∇µ logπ c(a|s) = Σ −1(a−µ).(17) Thereforeˆgµ =RΣ −1(a−µ)and E ∥ˆgµ∥2 2 |s =R 2 E (a−µ) ⊤Σ−2(a−µ)|s =R 2 Tr Σ−2 E (a−µ)(a−µ) ⊤ |s =R 2 Tr Σ−2Σ =R 2 Tr(Σ−1). For diagonalΣ = Diag(σ 2...

  2. [2]

    logits).Consider one action dimension i with logits zi ∈R K and softmax probabilities pi = softmax(zi)∈∆ K−1

    Categorical policy (gradient w.r.t. logits).Consider one action dimension i with logits zi ∈R K and softmax probabilities pi = softmax(zi)∈∆ K−1. Let ji ∼Cat(p i) and denote the sampled one-hot vector by eji ∈R K. For the log-probabilitylogπ d(ai ji |s) = logp i,ji, the softmax score w.r.t. logits is ∇zi logp i,ji =e ji −p i.(18) Hence the per-dimension R...

    2025

  3. [456]

    Lee, H., Lee, Y ., Seno, T., Kim, D., Stone, P., and Choo, J

    pmlr, 2015. Lee, H., Lee, Y ., Seno, T., Kim, D., Stone, P., and Choo, J. Hyperspherical normalization for scalable deep reinforce- ment learning. InForty-second International Conference on Machine Learning. 9 RN-D: Discretized Categorical Actors with Regularized Networks Lee, H., Hwang, D., Kim, D., Kim, H., Tai, J. J., Subrama- nian, K., Wurman, P. R., ...

    Pith/arXiv arXiv 2015

  4. [843]

    Cobbe, K

    PMLR, 2017. Cobbe, K. W., Hilton, J., Klimov, O., and Schulman, J. Phasic policy gradient. InInternational Conference on Machine Learning, pp. 2020–2027. PMLR, 2021. Farebrother, J., Orbay, J., Vuong, Q., Ta¨ıga, A. A., Cheb- otar, Y ., Xiao, T., Irpan, A., Levine, S., Castro, P. S., Faust, A., et al. Stop regressing: Training value func- tions via classi...

    Pith/arXiv arXiv 2017

  5. [1937]

    Mnih, V ., Badia, A

    PmLR, 2016a. Mnih, V ., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. Asyn- chronous methods for deep reinforcement learning. In Balcan, M. F. and Weinberger, K. Q. (eds.),Proceedings of The 33rd International Conference on Machine Learn- ing, volume 48 ofProceedings of Machine Learning Re- search, pp. 192...

    Pith/arXiv arXiv 1928

  6. [2025]

    Williams, R

    URL https://openreview.net/forum? id=Nq3oz7vn3j. Williams, R. J. Simple statistical gradient-following algo- rithms for connectionist reinforcement learning.Machine learning, 8(3):229–256, 1992. Wu, Y ., Mansimov, E., Grosse, R. B., Liao, S., and Ba, J. Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation.Ad-...

    Pith/arXiv arXiv 1992