{"id":"95ea83df-97fb-438b-ad29-b7144d0bf841","arxiv_id":"2501.06937","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An empirical study shows deep RL algorithms struggle in continuing tasks without resets and that TD-based reward centering improves their performance across larger MuJoCo and Atari testbeds.","lead":"This paper builds continuing-task versions of MuJoCo and Atari benchmarks and tests five deep RL algorithms under no resets, predefined resets, and agent-controlled resets. It reports that resets help learning and that a reward-centering method from earlier work improves most algorithms when rewards are shifted or discount factors are large.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Best-hyperparameter selection in Section 3 biases the reward-centering comparison; the universal 'improves or maintains' claim is not supported until the comparison is repeated under fixed hyperparameters.","rationale":"In good faith, the paper's central contribution is empirical evidence that TD-based reward centering is a robust drop-in improvement for deep RL in continuing tasks. The evidence in Tables 5-6 is generated by selecting the best hyperparameter setting separately for RC and baseline, which biases the comparison in favor of RC because it searches over more configurations. If the fixed-hyperparameter rerun shows many negative or insignificant cells, the universal claim should be weakened to 'can improve when tuned,' and the offset/discount conclusions would need re-examination. The weak-communication issue raised by the reader is legitimate but secondary: it affects only the no-reset Mujoco testbeds and does not explain the RC results in predefined-reset and Atari testbeds. I therefore keep the reader's CONDITIONAL verdict, but the primary condition should be a fixed-hyperparameter rerun with proper multiple-comparison correction, rather than only a verification of weak communication.","tokens_in":29560,"tokens_out":15312,"duration_ms":162614,"concrete_test":"Re-run Tables 5 and 6 with a pre-registered fixed-hyperparameter rule: use the same discount factor and the same algorithm defaults for RC and baseline (e.g., gamma=0.99, beta=1e-2), with no per-method best-of-grid selection, and apply Holm-Bonferroni correction across task-algorithm pairs. Compute the median and worst-case improvement over the full RC hyperparameter grid at each fixed baseline setting. The central claim survives only if a majority of fixed-settings comparisons are statistically significant positive or non-negative and the worst-case is not a large degradation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 states that 'the reported value is the best result across all tested hyperparameter settings for both reward-centered and baseline algorithms.' Section A.4 gives reward centering five beta values (3e-2, 1e-2, 3e-3, 1e-3, 3e-4) and discount factors 0.99, 0.999, and 1.0, i.e., roughly 15 RC settings, whereas the baseline tables only vary the discount factor over 0.99 and 0.999, i.e., 2 settings. Reporting the maximum over many noisy runs for RC and the maximum over fewer runs for the baseline inflates the apparent benefit even under the null hypothesis of no true effect. The Welch t-tests reported after selecting the best setting are not valid, and no multiple-comparison correction is applied. This threatens every entry in Tables 5 and 6, including the Atari and offset/discount results, and therefore the central claim that reward centering improves or maintains all tested algorithms in all testbeds. This is more load-bearing than the weak-communication concern, which only affects the no-reset testbeds.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical study of deep reinforcement learning algorithms in continuing tasks. It constructs continuing-task testbeds from Mujoco and Atari environments under three reset scenarios (no resets, predefined resets, agent-controlled resets) and evaluates DDPG, TD3, SAC, PPO, and DQN. The authors report that algorithms perform worse without resets, that reset costs act as solution parameters, and that large discount factors and reward offsets degrade performance. They then evaluate TD-based reward centering, originally proposed by Naik et al. (2024), and claim that it improves or maintains performance of all tested algorithms in all testbeds, eliminates sensitivity to reward offsets, and mitigates the harm of large discount factors, while also comparing it against moving-average and reference-state-based reward centering.","tokens_in":29828,"tokens_out":6287,"duration_ms":57752,"significance":"If the claims hold, this would be a valuable large-scale empirical validation of a simple drop-in modification for deep RL in continuing tasks, extending prior results from small tabular/linear tasks to Mujoco and Atari. The study is extensive: ten seeds per condition, multiple discount factors, three reset scenarios, and a publicly released implementation. It also provides useful comparative data on two alternative reward-centering approaches and an honest limitations section. The main obstacle is that the central 'improves or maintains all' claim currently rests on an asymmetric hyperparameter-selection protocol that can inflate the apparent benefit of reward centering; this needs to be addressed before the headline conclusion can be accepted.","major_comments":[{"comment":"The central claim that TD-based reward centering 'improves or maintains the performance of all of the tested algorithms in all testbeds' is not supported by the reported selection protocol. The paper states that 'the reported value is the best result across all tested hyperparameter settings for both reward-centered and baseline algorithms,' but the grids are not symmetric: Section A.4 gives reward-centered algorithms five beta values (3e-2, 1e-2, 3e-3, 1e-3, 3e-4) and three discount factors (0.99, 0.999, 1.0), about 15 settings, whereas baseline algorithms in Section A.2 vary only the discount factor over 0.99 and 0.999, two settings. Taking the maximum over many noisy runs for the treated condition and over fewer runs for the control inflates the apparent improvement even under the null, and the Welch t-tests computed on the selected settings are not valid because they ignore the selection step and no multiple-comparison correction is applied. The claim should be re-established with fixed hyperparameters, or with a nested selection protocol, or by reporting the distribution of improvement across all settings.","section":"Section 3, Table 16"},{"comment":"The abstract's claim that reward centering 'completely removes the detrimental effect caused by a large common reward offset' is stronger than the data in Table 16. For example, in the Swimmer row, centered PPO with a +100 offset shows a percentage improvement of -19.01% relative to the no-offset condition, and several other centered entries are negative as well. Since the offset has already been subtracted in the metric, a negative value means the centered algorithm still performs worse under the offset. The paper should either soften the claim to 'largely removes' or provide significance tests showing these residual differences are not statistically meaningful.","section":"Section 3 and Table 16"},{"comment":"The no-reset testbeds are justified by the weakly communicating property, but this property is asserted rather than verified. For the Swimmer testbed, the unbounded front-tip angle can drift without limit, as the paper's own Figure 2 shows; from such drifted states, the practical reachability of high-reward states is questionable. If weak communicability fails for a specific environment, the no-reset results in Tables 1 and 4 may reflect an unsolvable task rather than an exploration failure. The authors should provide evidence for the property (e.g., reachability checks or a control experiment with reset-based initialization) or qualify the interpretation of the no-reset results. This concern does not affect the predefined-reset and Atari comparisons.","section":"Section 2.1"}],"minor_comments":[{"comment":"The sentence 'we illustrate this approach using use TD(0)' contains a duplicated verb, and several figure captions say 'Shading area standards for one standard error' instead of 'stands for'.","section":"Section 3"},{"comment":"The paper inconsistently spells 'Mujoco' and 'MuJoCo', and 'Relu' should be 'ReLU'.","section":"Throughout"},{"comment":"Table 16 is very dense; the column structure with paired Y/N and -100/+100 offset columns is hard to read. Consider splitting into two tables or using explicit subheaders for each algorithm and offset.","section":"Table 16"},{"comment":"The claim that TD-based reward centering 'outperforms two other reward-centering approaches' is only true on average; Tables 17-18 contain counterexamples such as SAC in Swimmer with the RVI-based approach and PPO in BeamRider with the moving-average approach. Please qualify the statement.","section":"Abstract and Section 3"},{"comment":"Equation numbers (1)-(3) are used for DQN's update, but the subsequent TD-error definitions for DDPG, TD3, SAC, and PPO are not numbered, which makes it harder to refer to them in the text.","section":"Appendix A.4"},{"comment":"The reference list includes two entries for Sharma et al. with identical titles (2021 and 2022); please disambiguate them, for example by noting the venue or adding a footnote.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The method under study originates from Naik et al. (2024), which shares an author with the current manuscript. This is not a reason for rejection, but the editors may wish to confirm that the contribution is clearly positioned as an independent empirical extension. The primary technical issue is the asymmetric hyperparameter selection in Section 3; a reanalysis or a revised claim would be needed before the paper can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely useful empirical paper. The testbed suite (no resets, predefined resets, agent-controlled resets; Mujoco and Atari; code released) fills a real gap, and the Section 2 findings about resets—especially the state-space analysis showing why resets help exploration in Swimmer, and the observation that reset cost acts as a solution parameter—are solid and worth citing. The extension of TD-based reward centering to DDPG, TD3, SAC, PPO, and DQN at larger scale than Naik et al. is a reasonable next step, and the comparison with moving-average and reference-state baselines is informative.\n\nThe soft spot is exactly what the stress-test flags. In Section 3 the reported improvement is the best result across all tested hyperparameter settings for both RC and baseline, but RC gets roughly 15 settings (five beta values times three discount factors) while the baseline gets two discount factors. Taking the max over more noisy runs inflates the apparent gain even under the null. The Welch t-tests are then run on the selected setting, which makes them invalid, and there is no multiple-comparison correction. So the central claim that reward centering 'improves or maintains the performance of all of the tested algorithms in all testbeds' is not supported as stated. Some entries are large enough that the effect probably survives, but the tables as presented overstate the evidence.\n\nThe weak-communication premise for the no-reset Mujoco testbeds is stated but not verified. That is a less serious issue because it only affects interpretation of the no-reset results, not the RC comparison, and the authors do acknowledge the assumption.\n\nThe self-citation is not a problem here: the empirical tests are new and the theory is correctly attributed; the paper does not pretend to derive the method.\n\nBottom line: a serious referee should engage with this, but Section 3 needs a re-analysis—either fixed hyperparameters for both sides, or a proper multiple-comparison correction, or reporting the full distribution of results. As is, the headline claim should be softened. I would send it to review and ask for that revision.","headline":"Useful empirical study of continuing-task RL with reusable testbeds; the reward-centering headline is inflated by best-hyperparameter selection and needs a re-analysis before the 'improves or maintains all' claim can stand.","tokens_in":30347,"tokens_out":2875,"would_cite":true,"duration_ms":26320,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Subtracting an estimate of the reward rate before computing the TD error improves or maintains every deep RL algorithm tested on continuing tasks, fully removes the damage from reward offsets, and reduces the harm of large discount factors.","keywords":["continuing tasks","reward centering","temporal-difference learning","average-reward criterion","deep reinforcement learning","Mujoco testbeds","Atari testbeds","reset scenarios"],"falsifier":"In a continuing task with all rewards shifted by a large constant (say +100), a TD-based reward-centered DQN or SAC should produce, after subtracting the offset, the same steady reward rate as the uncentered algorithm on the unshifted task; if it does not, the paper's claim that centering 'completely removes the detrimental effect' of a common reward offset is refuted.","tokens_in":29379,"feed_emoji":"🤖","tokens_out":13522,"duration_ms":102113,"temperature":0.7,"pith_summary":"Deep reinforcement learning is usually tested in episodic tasks that reset; this paper studies what happens when the interaction never resets, as in many real-world control problems. It introduces continuing-task testbeds built from Mujoco and Atari environments under three reset regimes — no resets, predefined resets, and agent-controlled resets — and reports how DDPG, TD3, SAC, PPO, and DQN behave under each. The paper's central claim is that temporal-difference-based reward centering — subtracting a learned estimate of the average reward rate from every reward before computing the TD error — improves or maintains the performance of all of these algorithms on all of the testbeds. It further claims that this method completely cancels the catastrophic effect of a large common offset added to all rewards, and largely undoes the harm of a large discount factor, at a scale larger than the original study. If correct, reward centering is a simple modification that makes existing deep RL algorithms far more usable in settings where resets are unavailable or expensive.","feed_headline":"Reward centering helps all five deep RL algorithms in continuing tasks","feed_subtitle":"Reward centering keeps DQN, SAC, PPO, DDPG, TD3 from collapsing in no-reset tasks, and wipes out reward-offset damage.","key_machinery":"The mechanism that carries the argument is TD-based reward centering, a modification of the temporal-difference update rule. Given a transition $(S, R, S')$, the usual value update uses $\\delta = R + \\gamma V(S') - V(S)$; reward centering replaces this with $\\delta_{\\mathrm{RC}} = R - \\bar{R} + \\gamma V(S') - V(S)$, where the scalar $\\bar{R}$ is a biased estimate of the average reward rate under the current policy, updated by $\\bar{R} \\leftarrow \\bar{R} + \\eta \\alpha \\delta_{\\mathrm{RC}}$. The motivation comes from a Laurent-series decomposition of the discounted value function, which separates a state-independent term $r(\\pi)/(1-\\gamma)$ from the state-dependent differences that actually drive policy improvement; reward centering subtracts an estimate of that offset so the function approximator's capacity is not wasted on a huge constant. In the paper, this device is applied to DQN, DDPG, TD3, SAC, and PPO by swapping their TD errors for centered versions, and it is the reason every tested algorithm improves or holds steady across the testbeds.","core_discovery":"On continuing tasks, standard deep RL algorithms degrade sharply: without resets they often get trapped in suboptimal regions or wander into never-seen states; with resets they can underperform their episodic cousins in the continuing metric; and large discount factors or large shared reward offsets reliably hurt all of them. The paper's principal discovery is that a scalar correction — TD-based reward centering — fixes much of this. Instead of updating values with the raw TD error $\\delta = R + \\gamma V(S') - V(S)$, the agent uses $\\delta_{\\mathrm{RC}} = R - \\bar{R} + \\gamma V(S') - V(S)$, where $\\bar{R}$ is an estimate of the reward rate updated by the same centered error, $\\bar{R} \\leftarrow \\bar{R} + \\eta \\alpha \\delta_{\\mathrm{RC}}$. Across fifteen Mujoco-based and six Atari-based continuing testbeds, every tested algorithm either improves or stays the same when this modification is applied; a $+100$ (or $-100$) common reward offset, which normally causes catastrophic failure, leaves the centered algorithms' steady reward rates essentially unchanged; and the loss from raising the discount factor from $0.99$ to $0.999$ shrinks substantially, though it does not vanish in the larger tasks. The authors present this as the first large-scale confirmation that reward centering, previously demonstrated with Q-learning on small tasks, transfers to modern actor-critic and policy-gradient algorithms.","pith_inferences":["A natural extension the paper leaves open is offline RL: the same centered TD error could be applied when learning from a fixed dataset, where no reward rate estimate from online interaction is available.","The observation that predefined resets both unstick agents and shrink the explored state space suggests an algorithmic substitute for resets: any mechanism that keeps the visited-state distribution compact, such as state-space wrapping or exploration regularization, might recover part of the benefit without changing the task.","The paper's claim that reset cost acts as a solution parameter implies that continuing-task benchmarks should report results across a range of reset costs, since a single default value could either hide or exaggerate an algorithm's strengths."],"forward_implications":["Deploying a TD-based reward-centered version of DQN, DDPG, TD3, SAC, or PPO on a continuing task with any constant reward offset should recover, to within noise, the same steady reward rate as on the unshifted task.","Tuning the discount factor matters less when reward centering is on: increasing it from 0.99 to 0.999 causes much smaller performance drops, though the drop does not disappear in the harder Mujoco testbeds.","Reward centering is the safest of the three variants tested: the moving-average version helps off-policy algorithms somewhat despite its theoretical unsoundness, while the reference-state version helps in some tasks and hurts in others.","In tasks without resets, today's deep RL algorithms benefit strongly from even rare random resets, because resets both pull agents out of suboptimal traps and confine exploration to a manageable region of state space; the paper shows the latter effect explicitly via a wrapped-angle Swimmer testbed.","Policies trained in the continuing formulation with predefined resets can beat policies trained in the episodic formulation when both are evaluated in the continuing task, by trading slightly more frequent resets for higher reward rates, and reset cost acts as a solution parameter rather than just a problem parameter."],"supporting_citations":[{"why":"Supplies the reward-centering idea and the prior small-scale DQN result this paper extends to more algorithms and larger testbeds.","marker":"Naik et al. (2024)"},{"why":"Provides the average-reward TD method from which the TD-based reward-rate estimate is derived; the convergence argument follows its proof steps.","marker":"Wan et al. (2021)"},{"why":"Gives the Laurent-series decomposition that separates the state-independent offset from value differences, motivating centering; also frames the weakly communicating property used in testbed design.","marker":"Puterman (2014)"},{"why":"Their relative Q-learning is the source of the reference-state reward-centering baseline that the paper compares against.","marker":"Devraj and Meyn (2021)"},{"why":"The TD(0) error formalism that reward centering rewrites into the centered error $\\delta_{\\mathrm{RC}}$.","marker":"Sutton (2018)"}],"fun_headline_variants":["Reward centering rescues deep RL in no-reset tasks","One scalar fix keeps deep RL stable without resets","Reward centering collapses the offset damage in continuing RL","Deep RL thrives in continuing tasks with reward centering","Reward centering outperforms other fixes in continuing RL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the no-reset Mujoco testbeds are weakly communicating — from almost every state a skilled agent can reach any other state — so poor performance without resets reflects weak learning rather than an impossible task.","fun_headline_variants_meta":{"raw":{"variants":["Reward centering rescues deep RL in no-reset tasks","One scalar fix keeps deep RL stable without resets","Reward centering collapses the offset damage in continuing RL","Deep RL thrives in continuing tasks with reward centering","Reward centering outperforms other fixes in continuing RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000226,"raw_usage":{"total_tokens":1526,"prompt_tokens":1064,"completion_tokens":462,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":680,"completion_tokens_details":{"reasoning_tokens":381}},"tokens_in":680,"tokens_out":462,"duration_ms":4660,"temperature":1.0,"reasoning_tokens":381,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:49:40.506135+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"In a continuing task with all rewards shifted by a large constant (say +100), a TD-based reward-centered DQN or SAC should produce, after subtracting the offset, the same steady reward rate as the uncentered algorithm on the unshifted task; if it does not, the paper's claim that centering 'completely removes the detrimental effect' of a common reward offset is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the average-reward TD method from which the TD-based reward-rate estimate is derived; the convergence argument follows its proof steps."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the Laurent-series decomposition that separates the state-independent offset from value differences, motivating centering; also frames the weakly communicating property used in testbed design."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Their relative Q-learning is the source of the reference-state reward-centering baseline that the paper compares against."}],"review_version":1}