{"id":"8fdf46f5-c818-4892-9e22-aaddb2348332","arxiv_id":"2501.09080","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ASAC extends soft actor-critic to the entropy-regularized average-reward setting with a policy improvement theorem, but its claimed novelty is undermined by the earlier RVI-SAC algorithm.","lead":"This paper introduces ASAC, an entropy-regularized soft actor-critic algorithm for average-reward reinforcement learning, and claims it outperforms existing average-reward methods on continuous control benchmarks. The work is relevant because it extends the popular SAC algorithm to the average-reward setting, which avoids discount-factor tuning, but its novelty claim is undercut by prior work RVI-SAC.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central 'first deep ERAR actor-critic, superior performance' claim is contradicted by the paper's own citation of RVI-SAC, which is omitted from the experiments.","rationale":"The reader's formal weakest_assumption is the off-policy estimate of the reward-rate theta, which is a genuine secondary concern: the batch mean in Algorithm 1, line 18, is not an unbiased estimate of theta under the stationary distribution of the current policy, and biased targets could shift the learned differential values. However, the decisive issue for the paper's central claim is the false novelty assertion and the omitted RVI-SAC baseline. RVI-SAC is cited in Section 6 but absent from Section 5's comparisons, so the abstract's 'first' and 'superior' claims cannot both hold as stated. Since the reader's overall verdict is already REJECT and my concern reinforces that conclusion, the verdict is unchanged. The theoretical rate-gap analysis is not the problem; the paper could be repositioned as a concurrent or independent instantiation of ERAR SAC, but in its current form the headline claims are unsupported.","tokens_in":15963,"tokens_out":5180,"duration_ms":52205,"concrete_test":"Run the same benchmark protocol reported in Figure 2 (six MuJoCo v5 environments, 20 seeds, identical evaluation-return metric) with RVI-SAC (Hisaki & Ono, 2024) as an additional baseline, using the published RVI-SAC hyperparameters. If RVI-SAC matches or outperforms ASAC on any environment, the 'superior performance' claim is qualified; regardless, confirming that RVI-SAC implements the same entropy-regularized average-reward objective falsifies the 'first' claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise for the abstract's central claims is that no deep entropy-regularized average-reward actor-critic algorithm existed before ASAC. The paper itself undermines this in Section 6: 'After submission of this paper, we became aware of related work: RVI-SAC Hisaki & Ono (2024), which uses relative value iteration (RVI) to estimate the reward-rate.' RVI-SAC is a deep off-policy SAC extension for the average-reward criterion with entropy regularization, so the abstract's assertion that 'deep RL algorithms for the entropy-regularized average-reward objective have not been developed' is factually false. Moreover, the experiments in Section 5 compare only ARO-DDPG, ATRPO, and APO; RVI-SAC is not included, even though it is the most relevant baseline. As a consequence, the empirical claim of 'superior performance for the average-reward criterion' is not established against the existing method it most directly extends. This is not a question of internal consistency of Theorem 1 or Lemma 1, which appear sound; it is a failure of the paper's central novelty and comparative claims.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ASAC, an average-reward extension of the soft actor-critic (SAC) algorithm with entropy regularization. It introduces the entropy-regularized average-reward (ERAR) MDP objective, derives a Bellman backup equation, a rate-gap lemma, and an ERAR policy improvement theorem (Lemma 1 and Theorem 1), and presents an off-policy deep RL algorithm with a critic, an actor, and a trainable reward-rate parameter. Experiments on MuJoCo continuous-control tasks compare ASAC against ARO-DDPG, ATRPO, and APO, and report faster learning and higher asymptotic average reward. The paper also discusses an adaptive reset cost and a value-centering heuristic, and notes in Section 6 that it became aware of RVI-SAC (Hisaki & Ono, 2024) after submission.","tokens_in":16104,"tokens_out":5675,"duration_ms":54711,"significance":"If the theoretical results are correct, the paper makes a genuine contribution by extending soft policy improvement to the entropy-regularized average-reward setting; the proofs of Lemma 1 and Theorem 1 are concise and appear sound. The algorithm itself is a natural and potentially useful extension of SAC. However, the paper's central significance claims—being the first deep ERAR actor-critic and achieving superior performance—are not substantiated as written. The paper's own Section 6 acknowledges RVI-SAC, a deep off-policy average-reward SAC variant, and the experimental section omits this most relevant baseline. Furthermore, the reward-rate estimation and value-centering mechanisms are heuristics whose convergence and unbiasedness are not established, and these heuristics are load-bearing for the practical algorithm. The contribution is therefore best seen as an independent derivation of an ERAR actor-critic with heuristic stabilization, rather than the first such algorithm with demonstrated superiority.","major_comments":[{"comment":"The abstract and Section 1 state that 'deep RL algorithms for the entropy-regularized average-reward objective have not been developed' and that the paper introduces an algorithm to address this gap. Section 6, however, states: 'After submission of this paper, we became aware of related work: RVI-SAC Hisaki & Ono (2024), which uses relative value iteration (RVI) to estimate the reward-rate.' This is internally inconsistent: if RVI-SAC is a deep entropy-regularized average-reward actor-critic, the 'first' claim is false. The authors need to either demonstrate a substantive algorithmic difference from RVI-SAC or revise the novelty claim to acknowledge concurrent work.","section":"Abstract and Section 6"},{"comment":"The experimental evaluation compares ASAC only with ARO-DDPG, ATRPO, and APO, and omits RVI-SAC, which is cited in Section 6 as the most closely related method. Given the paper's stated goal of validating 'superior performance for the average-reward criterion' against existing average-reward algorithms, the omission of the most relevant baseline makes the comparative claim unsupported. A direct comparison with RVI-SAC, on the same environments and with comparable hyperparameter tuning, is necessary before the empirical superiority claim can be accepted.","section":"Section 5 Experiments"},{"comment":"The reward-rate parameter θ is updated by minimizing (θ − θ̄)², where θ̄ is the batch mean of r − β⁻¹ log(π/π₀) computed from off-policy replay data (Algorithm 1, line 18). The paper provides no theoretical justification that this estimate converges to the true entropy-regularized reward-rate θπ, especially when the actor is changing and the data are off-policy. Since θ enters the critic target (Equation 12), a biased θ directly corrupts the critic update. Lemma 2 assumes that θπ is 'calculated (e.g. with direct rollouts)' and does not cover this estimation scheme; the gap needs to be addressed or the claim of convergence relaxed.","section":"Section 4.2 and Algorithm 1"},{"comment":"The value-centering heuristic, which subtracts Q(0,0) from all target Q-values (Algorithm 1, line 12), is introduced to address the shift-invariance of the average-reward Bellman equation. The paper states that this choice is 'arbitrary, but works well in practice,' but it does not analyze whether subtracting a fixed state-action value introduces bias under function approximation, nor whether the reference (0,0) is meaningful across the tested environments. Since the differential value function is defined only up to an additive constant, an analysis or ablation is needed to show that this operation does not distort learning.","section":"Section 4.2 and Algorithm 1"}],"minor_comments":[{"comment":"In the proof of the ERAR Rate Gap, the line 'E[...] Aπ(st, at) − 1/β log π(at|st)/π0(at|st)' uses π in the log term where the statement of the lemma uses π′; this is a typographical slip that should be corrected.","section":"Supplementary Materials, proof of Lemma 1"},{"comment":"Lemma 2 refers to 'θπ of Equation (1)', but Equation (1) defines the unregularized reward-rate ρπ; the entropy-regularized rate θπ is defined in Equation (3). The reference should be to Equation (3).","section":"Section 4.1, Lemma 2"},{"comment":"The statement 'π′ ∝ exp βQπ(s, a)' omits the prior policy π₀; for the non-uniform prior used in this paper, the improved policy is π′ ∝ π₀ exp(β Qπ), as given in Theorem 1.","section":"Section 4.1, introductory paragraph"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central novelty and empirical claims are undermined by the authors' own acknowledgment of RVI-SAC. This is a scope and fit issue: for a journal publication, the related-work claim must be corrected and the most relevant baseline must be included in the experiments. The theoretical core is sound, so I see a viable path to revision, but the currently stated claims cannot stand as written."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. The theory is real: Lemma 1's rate-gap identity and Theorem 1's policy improvement are correct and not in prior work. The central empirical claim is not supported: the paper's own Section 6 acknowledges RVI-SAC, a deep entropy-regularized average-reward actor-critic, but the experiments never compare against it, and the abstract still says no such algorithms existed. That is a load-bearing omission.\n\nWhat's new: the ERAR rate-gap lemma and the policy improvement theorem are genuine extensions of the discounted SAC results to the average-reward setting. The proofs are straightforward given the Bellman equation and the standard variational formula, but they are clean and the paper does them correctly. The adaptation of SAC to average reward is also a natural engineering contribution — value centering by subtracting Q(0,0) and the adaptive reset cost are sensible heuristics, clearly explained and easy to replicate.\n\nWhere it falls down: the novelty claim. The abstract and Discussion state that 'deep RL algorithms for the entropy-regularized average-reward objective have not been developed.' The paper itself cites RVI-SAC (Hisaki & Ono, 2024) in Section 6 and calls it 'related work' that uses RVI to estimate the reward rate. That is exactly a deep off-policy SAC extension for average reward with entropy regularization. So the paper's own reference list contradicts its headline. The experiments compare only ARO-DDPG, ATRPO, and APO. RVI-SAC is the one baseline that matters, and it is absent. You cannot claim superior performance against the method you are extending when you choose not to run it.\n\nIs the omission fatal? For the empirical claim, yes. The theory stands on its own, and the algorithm may well be competitive, but the paper doesn't show it. The heuristic updates for θ and the centering trick are not theoretically justified, but that is a common situation in this literature; the real problem is the missing comparison.\n\nWho is this for? A researcher working on average-reward deep RL who wants the rate-gap lemma and the policy improvement theorem in one place. The experimental section should be treated as preliminary.\n\nMy recommendation: send it to review, but with a strong request that the authors include RVI-SAC, rerun the comparisons, and correct the novelty claims in the abstract and discussion. The paper is not desk-rejectable — the theory is solid and the algorithm is likely useful — but it is not acceptable in this form.","headline":"Genuine theoretical results, but the paper's own citation of RVI-SAC contradicts its 'first deep ERAR actor-critic' claim, and the missing baseline undermines the empirical comparison.","tokens_in":16711,"tokens_out":1944,"would_cite":true,"duration_ms":19382,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T05","68Q32","90C40"],"pacs":[],"model":"deepseek-v4-flash","headline":"An average-reward version of soft actor-critic learns continuing tasks without a discount factor and beats prior deep RL baselines on standard control benchmarks.","keywords":["average-reward reinforcement learning","entropy regularization","soft actor-critic","maximum entropy policy","differential value function","continuous control","policy improvement theorem","deep reinforcement learning"],"falsifier":"Run ASAC to convergence on a communicating continuous-control task, then compare the learned reward-rate against a direct Monte Carlo estimate obtained by rolling out the final policy from many starting states; if the two differ systematically beyond standard error, the replay-batch estimator is biased and the critic updates are not consistent with the true objective.","tokens_in":15702,"feed_emoji":"🤖","tokens_out":5153,"duration_ms":48731,"temperature":0.7,"pith_summary":"This paper presents an average-reward soft actor-critic (ASAC), a deep actor-critic algorithm that maximizes the entropy-regularized long-run average reward instead of a discounted return. It closes a gap: discounted maximum-entropy methods have no analogue for continuing tasks, while existing deep average-reward methods are policy-gradient based and skip entropy regularization. The paper proves that the Boltzmann-style policy update used by soft actor-critic increases the entropy-regularized reward-rate in the average-reward setting, and demonstrates on continuous-control benchmarks that ASAC learns faster and reaches higher asymptotic performance than prior average-reward algorithms. If the claims hold, the average-reward objective becomes a practical alternative to discounting for ongoing tasks.","feed_headline":"Average-reward soft actor-critic beats prior baselines without discounting.","feed_subtitle":"It is the first entropy-regularized deep actor-critic for average reward, and it wins on six MuJoCo tasks.","key_machinery":"The load-bearing object is the entropy-regularized average-reward objective, whose rate is the long-run mean of the reward minus the inverse-temperature-weighted KL divergence from the prior policy, together with its differential Q-function, which satisfies a Bellman backup with the rate subtracted. The policy improvement theorem is carried by the Boltzmann update that reweights the prior policy by the exponential of the differential Q-value; the proof uses the variational log-partition formula for this exponential weighting to show the rate gap is nonnegative. The algorithm then alternates policy evaluation of the differential Q-function with policy improvement, using twin Q-networks, Polyak-averaged targets, a trainable reward-rate fitted to replay-batch means, and a value-centering operation that pins the differential value at the origin.","core_discovery":"The central claim is that the soft policy improvement step of discounted SAC carries over to entropy-regularized average-reward MDPs, yielding the first deep entropy-regularized actor-critic algorithm for the average-reward setting. The paper proves that the gap between the reward-rates of two policies equals the expected advantage plus the KL term under the new policy's stationary distribution, and that setting the updated policy proportional to the prior times the exponential of the differential Q-function makes this gap nonnegative. The resulting algorithm, ASAC, evaluates the differential Q-function with a Bellman backup in which the reward rate is subtracted, updates the actor by minimizing a KL loss against the Boltzmann target, and maintains the rate as a trainable parameter fitted to the batch mean of the per-step regularized reward. On six MuJoCo locomotion tasks, ASAC outperforms the three existing deep average-reward baselines, and the appendix reports that it also outperforms discounted SAC. The paper additionally introduces an adaptive reset cost for terminating episodes and a value-centering shift to handle the non-uniqueness of the average-reward Bellman solution.","pith_inferences":["If the replay-batch estimator for the reward rate is biased under off-policy updates, ASAC's critic targets inherit that bias; a principled estimator, for example one derived from the differential value function itself, would remove the heuristic and likely improve stability.","The value-centering operation at the origin is an arbitrary choice; a learned or statistically motivated centering scheme could be more data-efficient and less sensitive to the choice of reference state.","Automated temperature tuning, in the spirit of later SAC variants, is a natural next step since the paper leaves the inverse temperature fixed.","The same rate-gap and Boltzmann-update proof structure should extend to other entropy-regularized and behavior-regularized policy updates, suggesting the theoretical result reaches beyond SAC."],"forward_implications":["ASAC provides a drop-in alternative to SAC for continuing tasks, removing the discount factor as a tunable hyperparameter while keeping the same default hyperparameters.","The policy improvement theorem extends monotonic average-reward improvement to entropy-regularized policies with an arbitrary prior policy, going beyond the uniform-prior MaxEnt case.","On the six MuJoCo tasks tested, ASAC achieves the best average-reward performance compared with ATRPO, APO, and ARO-DDPG, and also outperforms discounted SAC in the appendix comparison.","The adaptive reset cost makes terminating environments tractable in the average-reward framework without hand-picking a task-specific penalty."],"supporting_citations":[{"why":"Supplies the soft actor-critic algorithm and the soft policy improvement lemma that ASAC generalizes to the average-reward setting.","marker":"Haarnoja et al. (2018b)"},{"why":"Provides the ATRPO baseline and the unregularized average-reward policy improvement result that Theorem 1 extends.","marker":"Zhang & Ross (2021)"},{"why":"Provides the APO baseline and the discussion of value drift that motivates value-function centering.","marker":"Ma et al. (2021)"},{"why":"Provides the ARO-DDPG baseline and the double Q-learning technique used in ASAC's critic.","marker":"Saxena et al. (2023)"},{"why":"Gives the convergence results for average-reward policy evaluation that Lemma 2 relies on.","marker":"Wan et al. (2021)"},{"why":"Supplies the variational formula used in the proof of Theorem 1 to bound the rate gap below by zero.","marker":"Mitter & Newton (2000)"},{"why":"Provides the same variational identity in the KL-control context, used to show Boltzmann updates improve the rate.","marker":"Theodorou & Todorov (2012)"},{"why":"Concurrently derives the entropy-regularized average-reward Bellman backup in the inverse RL setting.","marker":"Wu et al. (2024)"},{"why":"Motivates using twin critics and the minimum Q-value to reduce estimation bias in actor-critic methods.","marker":"Fujimoto et al. (2018)"}],"fun_headline_variants":["No discounting needed: entropy-regularized average-reward AC wins","First entropy-regularized avg-reward actor-critic, wins MuJoCo","Average-reward SAC: first to beat discounted methods on MuJoCo","No discounting, still beats everything: average-reward actor-critic","Avg-reward soft actor-critic: skips discounting, tops MuJoCo"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The practical performance of ASAC rests on the unproven heuristic that the batch mean of the per-step regularized reward over off-policy replay data converges to the true entropy-regularized reward-rate; if that estimate is biased, the critic targets are wrong and learning can fail.","fun_headline_variants_meta":{"raw":{"variants":["No discounting needed: entropy-regularized average-reward AC wins","First entropy-regularized avg-reward actor-critic, wins MuJoCo","Average-reward SAC: first to beat discounted methods on MuJoCo","No discounting, still beats everything: average-reward actor-critic","Avg-reward soft actor-critic: skips discounting, tops MuJoCo"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000959,"raw_usage":{"total_tokens":4061,"prompt_tokens":897,"completion_tokens":3164,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":3063}},"tokens_in":513,"tokens_out":3164,"duration_ms":20622,"temperature":1.0,"reasoning_tokens":3063,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:10:18.416568+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ASAC to convergence on a communicating continuous-control task, then compare the learned reward-rate against a direct Monte Carlo estimate obtained by rolling out the final policy from many starting states; if the two differ systematically beyond standard error, the replay-batch estimator is biased and the critic updates are not consistent with the true objective.","supporting_citations":[{"cited_title":"Relative entropy and free energy dualities: Con- nections to path integral and kl control","cited_arxiv_id":null,"evidence_quote":"Provides the same variational identity in the KL-control context, used to show Boltzmann updates improve the rate."}],"review_version":1}