REVIEW 3 major objections 6 minor 31 references
An Empirical Study of Deep Reinforcement Learning in Continuing Tasks
T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Subtracting an estimate of the reward rate before computing the TD error improves or maintains every deep RL algorithm tested on continuing tasks, fully removes the damage from reward offsets, and reduces the harm of large discount factors.
desk verdict Useful empirical study of continuing-task RL with reusable testbeds; the reward-centering headline is inflated by best-hyperparameter selection and needs a re-analysis before the 'improves or maintains all' claim can stand. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism that carries the argument is TD-based reward centering, a modification of the temporal-difference update rule. Given a transition $(S, R, S')$, the usual value update uses $\delta = R + \gamma V(S') - V(S)$; reward centering replaces this with $\delta_{\mathrm{RC}} = R - \bar{R} + \gamma V(S') - V(S)$, where the scalar $\bar{R}$ is a biased estimate of the average reward rate under the current policy, updated by $\bar{R} \leftarrow \bar{R} + \eta \alpha \delta_{\mathrm{RC}}$. The motivation comes from a Laurent-series decomposition of the discounted value function, which separates a state-independent term $r(\pi)/(1-\gamma)$ from the state-dependent differences that actually drive policy improvement; reward centering subtracts an estimate of that offset so the function approximator's capacity is not wasted on a huge constant. In the paper, this device is applied to DQN, DDPG, TD3, SAC, and PPO by swapping their TD errors for centered versions, and it is the reason every tested algorithm improves or holds steady across the testbeds.
What would settle it
In a continuing task with all rewards shifted by a large constant (say +100), a TD-based reward-centered DQN or SAC should produce, after subtracting the offset, the same steady reward rate as the uncentered algorithm on the unshifted task; if it does not, the paper's claim that centering 'completely removes the detrimental effect' of a common reward offset is refuted.
Extended reading notes
Core claim
On continuing tasks, standard deep RL algorithms degrade sharply: without resets they often get trapped in suboptimal regions or wander into never-seen states; with resets they can underperform their episodic cousins in the continuing metric; and large discount factors or large shared reward offsets reliably hurt all of them. The paper's principal discovery is that a scalar correction — TD-based reward centering — fixes much of this. Instead of updating values with the raw TD error $\delta = R + \gamma V(S') - V(S)$, the agent uses $\delta_{\mathrm{RC}} = R - \bar{R} + \gamma V(S') - V(S)$, where $\bar{R}$ is an estimate of the reward rate updated by the same centered error, $\bar{R} \leftarrow \bar{R} + \eta \alpha \delta_{\mathrm{RC}}$. Across fifteen Mujoco-based and six Atari-based continuing testbeds, every tested algorithm either improves or stays the same when this modification is applied; a $+100$ (or $-100$) common reward offset, which normally causes catastrophic failure, leaves the centered algorithms' steady reward rates essentially unchanged; and the loss from raising the discount factor from $0.99$ to $0.999$ shrinks substantially, though it does not vanish in the larger tasks. The authors present this as the first large-scale confirmation that reward centering, previously demonstrated with Q-learning on small tasks, transfers to modern actor-critic and policy-gradient algorithms.
Load-bearing premise
The load-bearing premise is that the no-reset Mujoco testbeds are weakly communicating — from almost every state a skilled agent can reach any other state — so poor performance without resets reflects weak learning rather than an impossible task.
Editorial extensions
If this is right
- Deploying a TD-based reward-centered version of DQN, DDPG, TD3, SAC, or PPO on a continuing task with any constant reward offset should recover, to within noise, the same steady reward rate as on the unshifted task.
- Tuning the discount factor matters less when reward centering is on: increasing it from 0.99 to 0.999 causes much smaller performance drops, though the drop does not disappear in the harder Mujoco testbeds.
- Reward centering is the safest of the three variants tested: the moving-average version helps off-policy algorithms somewhat despite its theoretical unsoundness, while the reference-state version helps in some tasks and hurts in others.
- In tasks without resets, today's deep RL algorithms benefit strongly from even rare random resets, because resets both pull agents out of suboptimal traps and confine exploration to a manageable region of state space; the paper shows the latter effect explicitly via a wrapped-angle Swimmer testbed.
- Policies trained in the continuing formulation with predefined resets can beat policies trained in the episodic formulation when both are evaluated in the continuing task, by trading slightly more frequent resets for higher reward rates, and reset cost acts as a solution parameter rather than just a problem parameter.
Reading between the lines
- A natural extension the paper leaves open is offline RL: the same centered TD error could be applied when learning from a fixed dataset, where no reward rate estimate from online interaction is available.
- The observation that predefined resets both unstick agents and shrink the explored state space suggests an algorithmic substitute for resets: any mechanism that keeps the visited-state distribution compact, such as state-space wrapping or exploration regularization, might recover part of the benefit without changing the task.
- The paper's claim that reset cost acts as a solution parameter implies that continuing-task benchmarks should report results across a range of reset costs, since a single default value could either hide or exaggerate an algorithm's strengths.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents an empirical study of deep reinforcement learning algorithms in continuing tasks. It constructs continuing-task testbeds from Mujoco and Atari environments under three reset scenarios (no resets, predefined resets, agent-controlled resets) and evaluates DDPG, TD3, SAC, PPO, and DQN. The authors report that algorithms perform worse without resets, that reset costs act as solution parameters, and that large discount factors and reward offsets degrade performance. They then evaluate TD-based reward centering, originally proposed by Naik et al. (2024), and claim that it improves or maintains performance of all tested algorithms in all testbeds, eliminates sensitivity to reward offsets, and mitigates the harm of large discount factors, while also comparing it against moving-average and reference-state-based reward centering.
Significance. If the claims hold, this would be a valuable large-scale empirical validation of a simple drop-in modification for deep RL in continuing tasks, extending prior results from small tabular/linear tasks to Mujoco and Atari. The study is extensive: ten seeds per condition, multiple discount factors, three reset scenarios, and a publicly released implementation. It also provides useful comparative data on two alternative reward-centering approaches and an honest limitations section. The main obstacle is that the central 'improves or maintains all' claim currently rests on an asymmetric hyperparameter-selection protocol that can inflate the apparent benefit of reward centering; this needs to be addressed before the headline conclusion can be accepted.
major comments (3)
- [Section 3, Table 16] The central claim that TD-based reward centering 'improves or maintains the performance of all of the tested algorithms in all testbeds' is not supported by the reported selection protocol. The paper states that 'the reported value is the best result across all tested hyperparameter settings for both reward-centered and baseline algorithms,' but the grids are not symmetric: Section A.4 gives reward-centered algorithms five beta values (3e-2, 1e-2, 3e-3, 1e-3, 3e-4) and three discount factors (0.99, 0.999, 1.0), about 15 settings, whereas baseline algorithms in Section A.2 vary only the discount factor over 0.99 and 0.999, two settings. Taking the maximum over many noisy runs for the treated condition and over fewer runs for the control inflates the apparent improvement even under the null, and the Welch t-tests computed on the selected settings are not valid because they ignore the selection step and no multiple-comparison correction is applied. The claim should be re-established with fixed hyperparameters, or with a nested selection protocol, or by reporting the distribution of improvement across all settings.
- [Section 3 and Table 16] The abstract's claim that reward centering 'completely removes the detrimental effect caused by a large common reward offset' is stronger than the data in Table 16. For example, in the Swimmer row, centered PPO with a +100 offset shows a percentage improvement of -19.01% relative to the no-offset condition, and several other centered entries are negative as well. Since the offset has already been subtracted in the metric, a negative value means the centered algorithm still performs worse under the offset. The paper should either soften the claim to 'largely removes' or provide significance tests showing these residual differences are not statistically meaningful.
- [Section 2.1] The no-reset testbeds are justified by the weakly communicating property, but this property is asserted rather than verified. For the Swimmer testbed, the unbounded front-tip angle can drift without limit, as the paper's own Figure 2 shows; from such drifted states, the practical reachability of high-reward states is questionable. If weak communicability fails for a specific environment, the no-reset results in Tables 1 and 4 may reflect an unsolvable task rather than an exploration failure. The authors should provide evidence for the property (e.g., reachability checks or a control experiment with reset-based initialization) or qualify the interpretation of the no-reset results. This concern does not affect the predefined-reset and Atari comparisons.
minor comments (6)
- [Section 3] The sentence 'we illustrate this approach using use TD(0)' contains a duplicated verb, and several figure captions say 'Shading area standards for one standard error' instead of 'stands for'.
- [Throughout] The paper inconsistently spells 'Mujoco' and 'MuJoCo', and 'Relu' should be 'ReLU'.
- [Table 16] Table 16 is very dense; the column structure with paired Y/N and -100/+100 offset columns is hard to read. Consider splitting into two tables or using explicit subheaders for each algorithm and offset.
- [Abstract and Section 3] The claim that TD-based reward centering 'outperforms two other reward-centering approaches' is only true on average; Tables 17-18 contain counterexamples such as SAC in Swimmer with the RVI-based approach and PPO in BeamRider with the moving-average approach. Please qualify the statement.
- [Appendix A.4] Equation numbers (1)-(3) are used for DQN's update, but the subsequent TD-error definitions for DDPG, TD3, SAC, and PPO are not numbered, which makes it harder to refer to them in the text.
- [References] The reference list includes two entries for Sharma et al. with identical titles (2021 and 2022); please disambiguate them, for example by noting the venue or adding a footnote.
Circularity Check
Reward-centering comparison is partly an artifact of unequal hyperparameter search budgets; the underlying empirical study is otherwise independent.
-
fitted input called prediction
[Section 3 (Tables 5-6) and Appendix A.4; baseline hyperparameters in Section A.2]
"The reported value is the best result across all tested hyperparameter settings for both reward-centered and baseline algorithms. ... The tested β values in our experiments are 3e−2, 1e−2, 3e−3, 1e−3, 3e−4. The tested discount factors in our experiments are 0.99, 0.999, and 1.0."
Reward centering is evaluated over 5 β values and 3 discount factors (about 15 configurations), while the baseline tables in Section A.2 list only discount factors 0.99 and 0.999 (2 configurations). Because the reported improvement is the maximum over each method's grid, the RC maximum is selected from many more noisy runs than the baseline maximum. Under the null hypothesis of no true effect, the expected best-of-15 exceeds best-of-2, so part of the reported 'improves or maintains all testbeds' result is forced by the unequal search budget rather than by the method. The subsequent Welch t-tests are computed after this best-setting selection and no multiple-comparison correction is applied, so the central claim is not a fixed-hyperparameter prediction.
full rationale
The paper's main empirical contribution—applying TD-based reward centering to DDPG, TD3, SAC, PPO and DQN on new continuing-task testbeds—is genuinely new and does not reduce by construction to the Naik et al. (2024) result. The self-citation to Naik et al. for the convergence of reward centering is real but not load-bearing for the empirical claim, since the paper's evidence is its own tables and learning curves. However, the headline universal conclusion is weakened by the best-hyperparameter selection: reward centering receives 5 β values and 3 discount factors, while baselines receive only 2 discount factors, so 'best result across all tested hyperparameter settings' inflates the apparent benefit. This is a selection-on-the-evaluation-metric issue rather than a definitional equivalence, so it is scored as partial circularity (4) rather than as a fully forced result.
Assumptions & free parameters
free parameters (5)
- Per algorithm-task hyperparameter selection =
Best among tested discount factors {0.99, 0.999} (plus 1.0 with reward centering), learning rates, and exploration…
- TD reward-centering step size beta =
Best among {3e-2, 1e-2, 3e-3, 1e-3, 3e-4}
- Moving-average reward-centering rate beta =
Best among {0.99, 0.999, 0.9999}
- Reset-dimension exploration noise and SAC target entropy =
Noise standard deviations {0.05, 0.005, 0.0005, 0.00005}; entropy targets {-|A|, -|A|-3, -|A|-6, -|A|-9}
- Reference-state batch I for RVI-based reward centering =
Batch size, fixed set sampled from replay buffer before first update
assumptions (5)
- domain assumption Average-reward criterion r(pi, s0) is well defined and used as evaluation metric.
- domain assumption No-reset Mujoco testbeds are weakly communicating.
- domain assumption Each policy induces a Markov chain with a single recurrent class, enabling the Laurent series decomposition of v_pi.
- standard math TD-based reward centering converges as stated in Theorem 1 of Naik et al. (2024).
- domain assumption Discounted optimal policies approach average-reward optimality as gamma goes to 1.
Cite this review
Pith. "Pith review of An Empirical Study of Deep Reinforcement Learning in Continuing Tasks." pith.science (2026). https://pith.science/paper/AZ6XIVV6
@misc{pith2026250106937,
author = {Pith},
title = {Pith review of: An Empirical Study of Deep Reinforcement Learning in Continuing Tasks},
year = {2026},
howpublished = {\url{https://pith.science/paper/AZ6XIVV6}},
note = {Machine review of arXiv:2501.06937}
}
read the original abstract
In reinforcement learning (RL), continuing tasks refer to tasks where the agent-environment interaction is ongoing and can not be broken down into episodes. These tasks are suitable when environment resets are unavailable, agent-controlled, or predefined but where all rewards-including those beyond resets-are critical. These scenarios frequently occur in real-world applications and can not be modeled by episodic tasks. While modern deep RL algorithms have been extensively studied and well understood in episodic tasks, their behavior in continuing tasks remains underexplored. To address this gap, we provide an empirical study of several well-known deep RL algorithms using a suite of continuing task testbeds based on Mujoco and Atari environments, highlighting several key insights concerning continuing tasks. Using these testbeds, we also investigate the effectiveness of a method for improving temporal-difference-based RL algorithms in continuing tasks by centering rewards, as introduced by Naik et al. (2024). While their work primarily focused on this method in conjunction with Q-learning, our results extend their findings by demonstrating that this method is effective across a broader range of algorithms, scales to larger tasks, and outperforms two other reward-centering approaches.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Abounadi, J., Bertsekas, D., and Borkar, V. S. (2001). Learning algorithms for Markov decision processes with average cost. SIAM Journal on Control and Optimization , 40(3):681--698
work page 2001
-
[2]
G., Naddaf, Y., Veness, J., and Bowling, M
Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. (2013). The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research , 47:253--279
work page 2013
-
[3]
Brockman, G. (2016). Openai gym. arXiv preprint arXiv:1606.01540
arXiv 2016
-
[4]
Devraj, A. M. and Meyn, S. P. (2021). Q-learning with uniformly bounded variance. IEEE Transactions on Automatic Control , 67(11):5948--5963
work page 2021
-
[5]
Eysenbach, B., Gu, S., Ibarz, J., and Levine, S. (2017). Leave no trace: Learning to reset for safe and autonomous reinforcement learning. arXiv preprint arXiv:1711.06782
arXiv 2017
-
[6]
Fujimoto, S., Hoof, H., and Meger, D. (2018). Addressing function approximation error in actor-critic methods. In International conference on machine learning , pages 1587--1596. PMLR
2018
-
[7]
Grand-Cl \'e ment, J. and Petrik, M. (2024). Reducing blackwell and average optimality to discounted mdps via the blackwell discount factor. Advances in Neural Information Processing Systems , 36
work page 2024
-
[8]
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. (2018). Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905
arXiv 2018
Show all 31 references
-
[9]
and Ono, I
Hisaki, Y. and Ono, I. (2024). Rvi-sac: Average reward off-policy deep reinforcement learning. arXiv preprint arXiv:2408.01972
2024 arXiv
-
[10]
Lillicrap, T. (2015). Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971
2015 arXiv
-
[11]
Ma, X., Tang, X., Xia, L., Yang, J., and Zhao, Q. (2021). Average-reward reinforcement learning with trust region methods. arXiv preprint arXiv:2106.03442
2021 arXiv
-
[12]
A., Veness, J., Bellemare, M
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015). Human-level control through deep reinforcement learning. nature , 518(7540):529--533
2015
-
[13]
Naik, A., Wan, Y., Tomar, M., and Sutton, R. S. (2024). Reward centering. arXiv preprint arXiv:2405.09999
2024 arXiv
-
[14]
A., Saparov, A., and Mitchell, T
Platanios, E. A., Saparov, A., and Mitchell, T. (2020). Jelly bean world: A testbed for never-ending learning. arXiv preprint arXiv:2002.06306
2020 arXiv
-
[15]
Puterman, M. L. (2014). Markov Decision Processes: Discrete Stochastic Dynamic Programming . John Wiley & Sons
2014
-
[16]
Saxena, N., Khastagir, S., Kolathaya, S., and Bhatnagar, S. (2023). Off-policy average reward actor-critic with deterministic policy search. In International Conference on Machine Learning , pages 30130--30203. PMLR
2023
-
[17]
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347
2017 arXiv
-
[18]
Sharma, A., Xu, K., Sardana, N., Gupta, A., Hausman, K., Levine, S., and Finn, C. (2021). Autonomous reinforcement learning: Formalism and benchmarking. In International Conference on Learning Representations
2021
-
[19]
Sharma, A., Xu, K., Sardana, N., Gupta, A., Hausman, K., Levine, S., and Finn, C. (2022). Autonomous reinforcement learning: Formalism and benchmarking. In International Conference on Learning Representations
2022
-
[20]
Sutton, R. S. (2018). Reinforcement learning: An introduction. A Bradford Book
2018
-
[21]
Todorov, E., Erez, T., and Tassa, Y. (2012). Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 5026--5033. IEEE
2012
-
[22]
U., De Cola, G., Deleu, T., Goul \ a o, M., Kallinteris, A., Krimmel, M., KG, A., et al
Towers, M., Kwiatkowski, A., Terry, J., Balis, J. U., De Cola, G., Deleu, T., Goul \ a o, M., Kallinteris, A., Krimmel, M., KG, A., et al. (2024). Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032
2024 arXiv
-
[23]
Wan, Y., Naik, A., and Sutton, R. S. (2021). Learning and planning in average-reward Markov decision processes. In Proceedings of the 38th International Conference on Machine Learning , volume 139, pages 10653--10662
2021
-
[24]
and Ross, K
Zhang, Y. and Ross, K. W. (2021). On-policy deep reinforcement learning for the average-reward criterion. In International Conference on Machine Learning , pages 12535--12545. PMLR
2021
-
[25]
Zhao, R., Abbas, Z., Szepesv \'a ri, D., Naik, A., Holland, Z., Tanner, B., and White, A. (2022). Csuite: Continuing environments for reinforcement learning. Github: google-deepmind/csuite
2022
-
[26]
Zhu, H., Yu, J., Gupta, A., Shah, D., Hartikainen, K., Singh, A., Kumar, V., and Levine, S. (2020). The ingredients of real-world robotic reinforcement learning. arXiv preprint arXiv:2004.12570
2020 arXiv
-
[27]
Zhu, Z., Braz, R. d. S., Bhandari, J., Jiang, D., Wan, Y., Efroni, Y., Wang, L., Xu, R., Guo, H., Nikulkov, A., et al. (2023). Pearl: A production-ready reinforcement learning agent. arXiv preprint arXiv:2312.03814
2023 arXiv
-
[28]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
-
[29]
@esa (Ref
\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
-
[30]
\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
-
[31]
yF࡞g5. t]k e_kx kϣ], [#>3,>Mj <3oseӡ؎O 7v 10 ΓZ Snc ay xط<Xت֬2/̡ Z̄峖G?Y[x=S c _ZSFX3#v)7n֎a | t 6IͰ|q֎Ğb /eev .Jj 6 4^ D[OVY L |axj ?#+#ڱ 44 . T c 7 W O &# O` !ң ծ [bY ncMj .0^?
@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.