Pith. sign in

REVIEW 4 major objections 3 minor 24 references

Average-Reward Soft Actor-Critic

T0 review · 4 major / 3 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An average-reward version of soft actor-critic learns continuing tasks without a discount factor and beats prior deep RL baselines on standard control benchmarks.

desk verdict Genuine theoretical results, but the paper's own citation of RVI-SAC contradicts its 'first deep ERAR actor-critic' claim, and the missing baseline undermines the empirical comparison. read the letter →

arxiv 2501.09080 v2 pith:SD7WNCF5 submitted 2025-01-15 cs.LG cs.AI

Jacob Adamczyk , Volodymyr Makarenko , Stas Tiomkin , Rahul V. Kulkarni This is my paper · ORCID
classification cs.LGcs.AI MSC 68T0568Q3290C40
keywords average-rewardreinforcementlearningentropyregularizationsoftactor-criticmaximumpolicydifferentialvaluefunctioncontinuouscontrolimprovementtheoremdeep
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper presents an average-reward soft actor-critic (ASAC), a deep actor-critic algorithm that maximizes the entropy-regularized long-run average reward instead of a discounted return. It closes a gap: discounted maximum-entropy methods have no analogue for continuing tasks, while existing deep average-reward methods are policy-gradient based and skip entropy regularization. The paper proves that the Boltzmann-style policy update used by soft actor-critic increases the entropy-regularized reward-rate in the average-reward setting, and demonstrates on continuous-control benchmarks that ASAC learns faster and reaches higher asymptotic performance than prior average-reward algorithms. If the claims hold, the average-reward objective becomes a practical alternative to discounting for ongoing tasks.

What carries the argument

The load-bearing object is the entropy-regularized average-reward objective, whose rate is the long-run mean of the reward minus the inverse-temperature-weighted KL divergence from the prior policy, together with its differential Q-function, which satisfies a Bellman backup with the rate subtracted. The policy improvement theorem is carried by the Boltzmann update that reweights the prior policy by the exponential of the differential Q-value; the proof uses the variational log-partition formula for this exponential weighting to show the rate gap is nonnegative. The algorithm then alternates policy evaluation of the differential Q-function with policy improvement, using twin Q-networks, Polyak-averaged targets, a trainable reward-rate fitted to replay-batch means, and a value-centering operation that pins the differential value at the origin.

What would settle it

Run ASAC to convergence on a communicating continuous-control task, then compare the learned reward-rate against a direct Monte Carlo estimate obtained by rolling out the final policy from many starting states; if the two differ systematically beyond standard error, the replay-batch estimator is biased and the critic updates are not consistent with the true objective.

Watch

Extended reading notes

Core claim

The central claim is that the soft policy improvement step of discounted SAC carries over to entropy-regularized average-reward MDPs, yielding the first deep entropy-regularized actor-critic algorithm for the average-reward setting. The paper proves that the gap between the reward-rates of two policies equals the expected advantage plus the KL term under the new policy's stationary distribution, and that setting the updated policy proportional to the prior times the exponential of the differential Q-function makes this gap nonnegative. The resulting algorithm, ASAC, evaluates the differential Q-function with a Bellman backup in which the reward rate is subtracted, updates the actor by minimizing a KL loss against the Boltzmann target, and maintains the rate as a trainable parameter fitted to the batch mean of the per-step regularized reward. On six MuJoCo locomotion tasks, ASAC outperforms the three existing deep average-reward baselines, and the appendix reports that it also outperforms discounted SAC. The paper additionally introduces an adaptive reset cost for terminating episodes and a value-centering shift to handle the non-uniqueness of the average-reward Bellman solution.

Load-bearing premise

The practical performance of ASAC rests on the unproven heuristic that the batch mean of the per-step regularized reward over off-policy replay data converges to the true entropy-regularized reward-rate; if that estimate is biased, the critic targets are wrong and learning can fail.

Editorial extensions

If this is right

  • ASAC provides a drop-in alternative to SAC for continuing tasks, removing the discount factor as a tunable hyperparameter while keeping the same default hyperparameters.
  • The policy improvement theorem extends monotonic average-reward improvement to entropy-regularized policies with an arbitrary prior policy, going beyond the uniform-prior MaxEnt case.
  • On the six MuJoCo tasks tested, ASAC achieves the best average-reward performance compared with ATRPO, APO, and ARO-DDPG, and also outperforms discounted SAC in the appendix comparison.
  • The adaptive reset cost makes terminating environments tractable in the average-reward framework without hand-picking a task-specific penalty.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the replay-batch estimator for the reward rate is biased under off-policy updates, ASAC's critic targets inherit that bias; a principled estimator, for example one derived from the differential value function itself, would remove the heuristic and likely improve stability.
  • The value-centering operation at the origin is an arbitrary choice; a learned or statistically motivated centering scheme could be more data-efficient and less sensitive to the choice of reference state.
  • Automated temperature tuning, in the spirit of later SAC variants, is a natural next step since the paper leaves the inverse temperature fixed.
  • The same rate-gap and Boltzmann-update proof structure should extend to other entropy-regularized and behavior-regularized policy updates, suggesting the theoretical result reaches beyond SAC.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes ASAC, an average-reward extension of the soft actor-critic (SAC) algorithm with entropy regularization. It introduces the entropy-regularized average-reward (ERAR) MDP objective, derives a Bellman backup equation, a rate-gap lemma, and an ERAR policy improvement theorem (Lemma 1 and Theorem 1), and presents an off-policy deep RL algorithm with a critic, an actor, and a trainable reward-rate parameter. Experiments on MuJoCo continuous-control tasks compare ASAC against ARO-DDPG, ATRPO, and APO, and report faster learning and higher asymptotic average reward. The paper also discusses an adaptive reset cost and a value-centering heuristic, and notes in Section 6 that it became aware of RVI-SAC (Hisaki & Ono, 2024) after submission.

Significance. If the theoretical results are correct, the paper makes a genuine contribution by extending soft policy improvement to the entropy-regularized average-reward setting; the proofs of Lemma 1 and Theorem 1 are concise and appear sound. The algorithm itself is a natural and potentially useful extension of SAC. However, the paper's central significance claims—being the first deep ERAR actor-critic and achieving superior performance—are not substantiated as written. The paper's own Section 6 acknowledges RVI-SAC, a deep off-policy average-reward SAC variant, and the experimental section omits this most relevant baseline. Furthermore, the reward-rate estimation and value-centering mechanisms are heuristics whose convergence and unbiasedness are not established, and these heuristics are load-bearing for the practical algorithm. The contribution is therefore best seen as an independent derivation of an ERAR actor-critic with heuristic stabilization, rather than the first such algorithm with demonstrated superiority.

major comments (4)
  1. [Abstract and Section 6] The abstract and Section 1 state that 'deep RL algorithms for the entropy-regularized average-reward objective have not been developed' and that the paper introduces an algorithm to address this gap. Section 6, however, states: 'After submission of this paper, we became aware of related work: RVI-SAC Hisaki & Ono (2024), which uses relative value iteration (RVI) to estimate the reward-rate.' This is internally inconsistent: if RVI-SAC is a deep entropy-regularized average-reward actor-critic, the 'first' claim is false. The authors need to either demonstrate a substantive algorithmic difference from RVI-SAC or revise the novelty claim to acknowledge concurrent work.
  2. [Section 5 Experiments] The experimental evaluation compares ASAC only with ARO-DDPG, ATRPO, and APO, and omits RVI-SAC, which is cited in Section 6 as the most closely related method. Given the paper's stated goal of validating 'superior performance for the average-reward criterion' against existing average-reward algorithms, the omission of the most relevant baseline makes the comparative claim unsupported. A direct comparison with RVI-SAC, on the same environments and with comparable hyperparameter tuning, is necessary before the empirical superiority claim can be accepted.
  3. [Section 4.2 and Algorithm 1] The reward-rate parameter θ is updated by minimizing (θ − θ̄)², where θ̄ is the batch mean of r − β⁻¹ log(π/π₀) computed from off-policy replay data (Algorithm 1, line 18). The paper provides no theoretical justification that this estimate converges to the true entropy-regularized reward-rate θπ, especially when the actor is changing and the data are off-policy. Since θ enters the critic target (Equation 12), a biased θ directly corrupts the critic update. Lemma 2 assumes that θπ is 'calculated (e.g. with direct rollouts)' and does not cover this estimation scheme; the gap needs to be addressed or the claim of convergence relaxed.
  4. [Section 4.2 and Algorithm 1] The value-centering heuristic, which subtracts Q(0,0) from all target Q-values (Algorithm 1, line 12), is introduced to address the shift-invariance of the average-reward Bellman equation. The paper states that this choice is 'arbitrary, but works well in practice,' but it does not analyze whether subtracting a fixed state-action value introduces bias under function approximation, nor whether the reference (0,0) is meaningful across the tested environments. Since the differential value function is defined only up to an additive constant, an analysis or ablation is needed to show that this operation does not distort learning.
minor comments (3)
  1. [Supplementary Materials, proof of Lemma 1] In the proof of the ERAR Rate Gap, the line 'E[...] Aπ(st, at) − 1/β log π(at|st)/π0(at|st)' uses π in the log term where the statement of the lemma uses π′; this is a typographical slip that should be corrected.
  2. [Section 4.1, Lemma 2] Lemma 2 refers to 'θπ of Equation (1)', but Equation (1) defines the unregularized reward-rate ρπ; the entropy-regularized rate θπ is defined in Equation (3). The reference should be to Equation (3).
  3. [Section 4.1, introductory paragraph] The statement 'π′ ∝ exp βQπ(s, a)' omits the prior policy π₀; for the non-uniform prior used in this paper, the improved policy is π′ ∝ π₀ exp(β Qπ), as given in Theorem 1.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ASAC's rate-gap lemma and policy-improvement theorem are derived from standard Bellman and variational identities, not from their conclusions.

full rationale

The paper's derivation chain is self-contained and non-circular. Lemma 1 (ERAR Rate Gap) is proved by substituting the Bellman backup equation (Proposition 1 / Eq. 6) and using the stationarity of the steady-state distribution d^{π'} (Eq. 17); the expression θ^{π'} - θ^{π} is not assumed but obtained by algebra from the definition of the entropy-regularized reward-rate (Eq. 3). Theorem 1 then defines π' as the Boltzmann policy over Q^π (Eq. 9/18), rewrites (1/β) log(π'/π0) using Eq. 19, and invokes the standard variational formula (Mitter & Newton 2000; Theodorou & Todorov 2012) to conclude θ^{π'} ≥ θ^{π}. The inequality follows from the fact that V^π is no larger than the soft-max supremum; this is not a restatement of the target result. Lemma 2 imports convergence of average-reward policy evaluation from Wan et al. (2021) after absorbing the finite KL cost into the reward, which is a legitimate reduction to an external result rather than circularity. The algorithm's update of θ via the batch mean (Algorithm 1, line 18) is an estimator of the same quantity defined in Eq. 3, and the value-centering trick (line 12) is acknowledged as an arbitrary heuristic; these are implementation choices, not fitted inputs disguised as predictions. Self-citations (Adamczyk et al. 2023, 2025; Arriojas et al. 2023) appear only as background context and are not load-bearing for the central claims. The admitted related work RVI-SAC and its omission from experiments raise a novelty/comparison-validity concern, not circularity. Therefore the paper's core derivation does not reduce to its own inputs.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard average-reward MDP assumptions and a cited convergence result. The algorithm introduces no new physical or mathematical entities. The main free parameters are the temperature β, the reset cost scale p0, and the learning rate for θ, all chosen by hand.

free parameters (3)
  • Inverse temperature beta = β=5 for most environments; β=20 for Swimmer and Humanoid
    Controls the strength of the entropy regularization relative to reward. Chosen by hand per environment, not automatically tuned, and the paper notes it is in line with the reward scale used in Haarnoja et al. (2018b).
  • Reset cost scale p0 = 10
    Scales the adaptive reset penalty used to convert episodic tasks into continuing tasks. Set to 10 for all environments without tuning.
  • Learning rate for reward-rate theta = 5 × 10^-3
    Used by the Adam optimizer to update the trainable reward-rate parameter θ in Algorithm 1. Listed in Section 9.1.
assumptions (5)
  • domain assumption The Markov chain induced by any stationary policy is communicating (Assumption 1).
    Required for the entropy-regularized reward-rate (Equation 3) to be a unique scalar independent of the initial state. Standard in average-reward MDP literature, but not always satisfied in practice.
  • domain assumption The reward function is bounded (Assumption 2).
    Ensures the infinite sums in Equations (2) and (5) are well-defined. Standard and generally true for the MuJoCo benchmarks.
  • standard math The variational formula (1/β) log E_{a~π0} e^{βQ(s,a)} = sup_π E_{a~π}[Q(s,a) - (1/β) log π/π0] holds.
    Used in the proof of Theorem 1 to show that the policy improvement gap is non-negative. Cited to Mitter & Newton (2000) and Theodorou & Todorov (2012).
  • standard math Average-reward policy evaluation converges for a fixed policy (Wan et al., 2021).
    Invoked in Lemma 2 to guarantee that the critic update converges to the entropy-regularized differential value. The paper absorbs the KL term into the reward and applies the external convergence result.
  • standard math The steady-state distribution dπ' of policy π' satisfies E_{s~dπ',a~π',s'~p} V(s') = E_{s~dπ'} V(s) for any value function V.
    Used in the proof of Lemma 1 to cancel the difference of value functions. Follows from the definition of the steady-state distribution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Average-Reward Soft Actor-Critic." pith.science (2026). https://pith.science/paper/SD7WNCF5

@misc{pith2026250109080,
  author       = {Pith},
  title        = {Pith review of: Average-Reward Soft Actor-Critic},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SD7WNCF5}},
  note         = {Machine review of arXiv:2501.09080}
}
read the original abstract

The average-reward formulation of reinforcement learning (RL) has drawn increased interest in recent years for its ability to solve temporally-extended problems without relying on discounting. Meanwhile, in the discounted setting, algorithms with entropy regularization have been developed, leading to improvements over deterministic methods. Despite the distinct benefits of these approaches, deep RL algorithms for the entropy-regularized average-reward objective have not been developed. While policy-gradient based approaches have recently been presented for the average-reward literature, the corresponding actor-critic framework remains less explored. In this paper, we introduce an average-reward soft actor-critic algorithm to address these gaps in the field. We validate our method by comparing with existing average-reward algorithms on standard RL benchmarks, achieving superior performance for the average-reward criterion.

Figures

Figures reproduced from arXiv: 2501.09080 by the authors.

Figure 1
Figure 1. The Swimmer-v5 environment, often not included in Mujoco bench￾marks (Franceschetti et al., 2022), is no￾toriously difficult for discounted methods to solve when the discount factor is not tuned over and set to its default value of γ = 0.99. Other discount-sensitive exam￾ples of environments have been discussed by Tessler & Mannor (2020). We find that after carefully tuning the discount factor, SAC can solve the tas… view at source ↗
Figure 2
Figure 2. Training curves on continuous control benchmarks. We compare our algorithm, average [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Comparison to SAC shows that our average-reward extension outperforms the original [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 11 canonical work pages

  1. [7]

    Image augmentation is all you need: Regularizing deep reinforcement learning from pixels

    Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004.13649,

    arXiv 2004

  2. [9]

    Stochastic first-order methods for average-reward Markov decision processes

    Tianjiao Li, Feiyang Wu, and Guanghui Lan. Stochastic first-order methods for average-reward markov decision processes. arXiv preprint arXiv:2205.05800,

    work page Pith review arXiv

  3. [10]

    Lillicrap, Jonathan J

    Reinforcement Learning Journal 2025 Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun (eds.), International Conference on Learning Representations,

    work page 2025

  4. [11]

    Discounted reinforcement learning is not an optimization problem

    Abhishek Naik, Roshan Shariff, Niko Yasui, Hengshuai Yao, and Richard S Sutton. Discounted reinforcement learning is not an optimization problem. arXiv preprint arXiv:1910.02140,

    arXiv 1910

  5. [13]

    Controllability-aware unsupervised skill discovery

    Seohong Park, Kimin Lee, Youngwoon Lee, and Pieter Abbeel. Controllability-aware unsupervised skill discovery. arXiv preprint arXiv:2302.05103,

    arXiv

  6. [17]

    Relative entropy and free energy dualities: Con- nections to path integral and kl control

    Evangelos A Theodorou and Emanuel Todorov. Relative entropy and free energy dualities: Con- nections to path integral and kl control. In 2012 IEEE 51st IEEE Conference on Decision and Control (CDC), pp. 1466–1473. IEEE,

    work page 2012

  7. [20]

    Behavior regularized offline reinforcement learning

    Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361,

    arXiv 1911

  8. [21]

    Efficient reinforcement learning with large language model priors

    Xue Yan, Yan Song, Xidong Feng, Mengyue Yang, Haifeng Zhang, Haitham Bou Ammar, and Jun Wang. Efficient reinforcement learning with large language model priors. arXiv preprint arXiv:2410.07927,

    arXiv

Show all 24 references
  1. [22]

    Finite sample analysis of average-reward TD learning and Q-learning

    Reinforcement Learning Journal 2025 Sheng Zhang, Zhe Zhang, and Siva Theja Maguluri. Finite sample analysis of average-reward TD learning and Q-learning. Advances in Neural Information Processing Systems , 34:1230–1242,

    2025

  2. [24]

    reward scale

    The scale for reset penalties is chosen as p0 = 10 (see pseudocode below). In all environments (for SAC and ASAC) we use β = 5, except for Swimmer and Humanoid, for which we use β = 20 . Note that this is in line with the “reward scale” used in (Haarnoja et al., 2018b). We fou...

    2025

  3. [1999]

    Reward tweaking: Maximizing the total reward while planning for short horizons

    Chen Tessler and Shie Mannor. Reward tweaking: Maximizing the total reward while planning for short horizons. arXiv preprint arXiv:2002.03327,

    2002 arXiv

  4. [2003]

    Dense dynamics-aware reward synthesis: Integrating prior experience with demonstrations

    Cevahir Koprulu, Po-han Li, Tianyu Qiu, Ruihan Zhao, Tyler Westenbroek, David Fridovich-Keil, Sandeep Chinchali, and Ufuk Topcu. Dense dynamics-aware reward synthesis: Integrating prior experience with demonstrations. arXiv preprint arXiv:2412.01114,

  5. [2009]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems , pp. 5026–5033. IEEE,

    2012

  6. [2010]

    ∞X k=1 r(st+k, at+k) − 1 β log π(at+k|st+k) π0(at+k|st+k) − θπ # , Qπ(st+1, at+1) = r(st+1, at+1) − θπ + E p,π

    Average-Reward Soft Actor-Critic Supplementary Materials The following content was not necessarily subject to peer review. 8 Proofs Lemma 1 (ERAR Backup Equation) . Let an ERAR MDP be given with reward function r(s, a), fixed evaluation policy π and prior policy π0. Then the d...

    2017

  7. [2016]

    Proximal policy optimization algorithms

    Average-Reward Soft Actor-Critic John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    arXiv

  8. [2017]

    Composable deep reinforcement learning for robotic manipulation

    Tuomas Haarnoja, Vitchyr Pong, Aurick Zhou, Murtaza Dalal, Pieter Abbeel, and Sergey Levine. Composable deep reinforcement learning for robotic manipulation. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 6244–6251. IEEE, 2018a. Tuomas Haarnoja, A...

    2018 arXiv

  9. [2018]

    RVI-SAC: average reward off-policy deep reinforcement learning

    Yukinari Hisaki and Isao Ono. RVI-SAC: average reward off-policy deep reinforcement learning. arXiv preprint arXiv:2408.01972,

    arXiv

  10. [2019]

    A unified view of entropy-regularized Markov decision processes

    Gergely Neu, Anders Jonsson, and Vicenç Gómez. A unified view of entropy-regularized Markov decision processes. arXiv preprint arXiv:1705.07798,

    arXiv

  11. [2020]

    Reinforcement learning and control as probabilistic inference: Tutorial and review

    Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:1805.00909,

    arXiv

  12. [2021]

    Dueling network architectures for deep reinforcement learning

    Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pp. 1995–2003. PMLR,

    1995

  13. [2022]

    Making reinforcement learning work on swimmer

    Average-Reward Soft Actor-Critic Maël Franceschetti, Coline Lacoux, Ryan Ohouens, Antonin Raffin, and Olivier Sigaud. Making reinforcement learning work on swimmer. arXiv preprint arXiv:2208.07587,

    arXiv

  14. [2023]

    Prioritized experience replay.arXiv preprint arXiv:1511.05952,

    Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay.arXiv preprint arXiv:1511.05952,

    arXiv

  15. [2024]

    The dependence of effective planning horizon on model accuracy

    Nan Jiang, Alex Kulesza, Satinder Singh, and Richard Lewis. The dependence of effective planning horizon on model accuracy. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, pp. 1181–1189,

    2015

  16. [2025]

    What matters in on-policy reinforcement learning? a large-scale empirical study

    Marcin Andrychowicz, Anton Raichuk, Piotr Sta ´nczyk, Manu Orsini, Sertan Girgin, Raphael Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, et al. What matters in on-policy reinforcement learning? a large-scale empirical study. arXiv preprint arXi...

    2006 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.