{"id":"fb2805a6-5bde-49a2-8f2d-31d4ab24cb36","arxiv_id":"2501.00989","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Using the agent's current value estimate as a potential-based shaping function converges in tabular RL and accelerates DQN on Atari.","lead":"This paper proposes a reward-shaping rule that uses the agent's own running estimate of state values as the shaping signal. In Atari games the rule speeds up DQN training relative to no shaping, and the authors prove convergence for the tabular case.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Atari experiments use η up to 10 with γ=0.99, while Theorem 2 proves convergence only for η<0.005; the reported speedups lie entirely outside the proven contraction regime, so the theory does not support the headline empirical result.","rationale":"The reader identified the online-network/target-network mismatch as the weakest assumption; I agree that is a genuine gap, but the more elementary disconnect is the η regime. This is not an internal inconsistency—the paper candidly flags exceeding the bound and the contradiction with Proposition 5—but it is load-bearing because every empirical success is produced outside the theory's scope. The conditional verdict is still appropriate: the paper should be accepted only if the experimental claims are either restricted to heuristic observations or supported by a proof covering the actual η values / a demonstration of speedup in the proven range. This is why I recommend UNCHANGED relative to the reader's CONDITIONAL verdict.","tokens_in":12976,"tokens_out":10386,"duration_ms":100861,"concrete_test":"Run the tabular gridworld and a small Atari subset with η fixed inside the proven interval, e.g. η=0.005 for γ=0.99 and η=0.1 for γ=0.8, using the same five-seed protocol. If no consistent speedup appears, the empirical advantage is outside the theoretical regime and the paper must either prove an extended contraction bound or explicitly restrict its convergence claims.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Theorem 2's contraction guarantee is η∈(−1,(1−γ)/(1+γ)), which for the Atari discount γ=0.99 is η<0.005. The experiments evaluate η∈{0.5,1,2,3,5,10}, call η≈2 optimal, and report speedups at these values; every nonzero η tested is 100–2000× the proven upper bound. The same mismatch appears in the tabular study: with γ=0.8 the bound is 0.111, and the speedups are shown for η roughly three times larger. The manuscript acknowledges that the tabular bound can be exceeded and compares this to 'edge of stability,' but that is an analogy, not a proof. Consequently the Atari results cannot be attributed to Theorem 2: for η=2, γ=0.99 the proof's constant is |η|+γ|1+η|=4.97, so the shaped Bellman operator is not shown to be contractive, and no convergence or policy-invariance guarantee covers the experiments. A user who selects η inside the proven interval obtains a shaping signal roughly 400× smaller than the value the paper finds best, so the practical claim depends on an unproven extrapolation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BSRS (Bootstrapped Reward Shaping), a potential-based reward-shaping method in which the potential is the agent's current state-value estimate: Phi(n)(s) = eta * max_a Q(n)(s,a). The central theoretical result, Theorem 2, shows in a synchronous tabular setting that the shaped Bellman operator is a contraction for eta in (-1, (1-gamma)/(1+gamma)), with fixed point Q_infinity(s,a) = Q0(s,a) - (eta/(1+eta)) V0(s), so the greedy policy is unchanged. The paper also derives TD(0) and SARSA(0) equivalence interpretations (Propositions 4 and 5), and reports experiments on a tabular gridworld, the Atari suite with DQN, and Pendulum with TD3, claiming that BSRS improves training speed while requiring only a single-line code change.","tokens_in":13229,"tokens_out":7891,"duration_ms":73319,"significance":"If the claims hold, BSRS is an attractive minimal-intervention shaping method: it removes the need for task-specific potential design and reduces the shaping search to a scalar eta. The tabular contraction proof is clear, the fixed-point calculation is transparent, and the paper releases code. The explicit contraction bound is a genuine strength. The significance is limited, however, by a systematic mismatch between the proven regime and the experiments: all positive eta values used in the Atari study lie far outside Theorem 2's contraction range, and the deep RL implementation computes the potential with the online network despite Proposition 5 assuming a stop-gradient/target network. These gaps must be addressed before the empirical speedups can be presented as consequences of the theory.","major_comments":[{"comment":"Theorem 2 guarantees contraction only for eta in (-1, (1-gamma)/(1+gamma)). With the Atari discount gamma=0.99 this is eta < 0.005, yet all positive eta values tested in Figures 1 and 2 (0.5, 1, 2, 3, 5, 10) lie far outside this range, and eta approximately 2 is reported as optimal. The same mismatch appears in the tabular study: for gamma=0.8 the proven bound is eta < 0.111, while Figure 3 shows speedups at eta roughly three times larger. Consequently, the reported speedups are not consequences of Theorem 2; no convergence or policy-invariance guarantee covers them. The comparison to the 'edge of stability' is an analogy, not a proof. The authors should either run experiments inside the proven eta range or provide a separate analysis that covers the larger eta values actually used.","section":"Experiments (Atari and tabular), Figures 1-3; Theorem 2"},{"comment":"Proposition 5 explicitly assumes that V(s) is computed with a stop-gradient or target network (see Eq. 19). The experimental section states that using the online network to compute Phi gives better performance and that this contradicts Proposition 5. Therefore the regularized-SARSA equivalence in Proposition 5 does not hold for the deep RL experiments, and the Atari/TD3 speedups cannot be attributed to that theoretical result. To make the theory support the deep RL results, the experiments should use the target/stop-gradient network for Phi, or the paper should prove an analogous result for the online-network case.","section":"Function Approximation for Phi; Proposition 5"},{"comment":"Theorem 2 is proved for synchronous tabular value iteration on bounded value functions. The DQN and TD3 experiments use neural function approximation, minibatch stochastic gradient updates, replay buffers, and exploration policies; no argument is given that the contraction constant or fixed-point formula transfers to this setting. The 'Interpretation' paragraphs provide intuition, not convergence guarantees. The abstract and introduction state that the paper 'provides convergence proofs' and 'shows improved training speed' without clearly separating the proven tabular statements from the heuristic deep RL claims. This separation should be made explicit wherever the theory is invoked to explain the deep RL gains.","section":"Scope of Theorem 2 and deep RL transfer"}],"minor_comments":[{"comment":"The theorem statement writes Phi_infinity(s) = V0(s)/(1+eta), but the proof and the definition Phi(n)(s)=eta max_a Q(n)(s,a) give Phi_infinity(s) = eta*V0(s)/(1+eta). Please correct this typo in Eq. (8).","section":"Theorem 2, Eq. (8)"},{"comment":"The author list contains a spacing artifact ('V olodymyr'); this should be corrected in the camera-ready version.","section":"Title page"},{"comment":"Figure 1 selects the best nonzero eta per environment, which is an oracle selection. The text should state clearly that this per-environment tuning is not evidence for a single scalar hyperparameter; the aggregate curve in Figure 2 is the more direct support for the fixed-eta claim.","section":"Figure 1"},{"comment":"The proof of Proposition 4 is only a pointer to another paper's technique. Since Proposition 4 is stated as a result, the derivation should be included or the statement should be labeled as a direct corollary of that technique.","section":"Proposition 4"},{"comment":"The proof of Remark 3 is hard to follow because V(n)(s') is used both for the max of the inner backup and for the updated state-value function. Please disambiguate these two objects.","section":"Proof of Remark 3"},{"comment":"The claim that self-shaping has a negative impact in only 6/40 environments should be accompanied by error bars or a statistical significance test, since each environment uses five seeds.","section":"Atari reporting"}],"recommendation":"major_revision","confidential_remarks":"The paper's tabular theory is sound, and the method is simple enough to be practically interesting even if the deep RL gains are heuristic. The main issue for the editor is that the theory and experiments are misaligned: the proven eta range excludes the experimentally used values, and the deep RL implementation uses the online network contrary to Proposition 5. A revision that clearly separates proven results from heuristics and either extends the analysis or narrows the claims would make this acceptable. I see no evidence of misconduct or problematic citation practices."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the tabular theory is the real contribution, and it is in good shape—the contraction range, the fixed point, and the non-equivalence to any static potential are new and correctly argued. But the headline empirical claim is not backed by the theory: in the Atari experiments γ=0.99, so Theorem 2's contraction bound is η<0.005, and every nonzero η tested is at least 100 times larger. The authors concede this and gesture at 'edge of stability,' but that is an analogy, not a proof. On top of that, the Atari evaluation picks the best η per environment from the test set, so the reported 45%→60% median gain is an upper bound over hyperparameters, not a fixed-configuration result. The same mismatch shows up in the tabular gridworld: with γ=0.8, the bound is η<0.111, and the speedup curve peaks roughly three times higher.\n\nWhat is genuinely good: the idea of bootstrapping the potential from the agent's current value estimate is a natural but previously unformalized step from Ng et al. and Devlin-Kudenko; the proof that no static potential reproduces the BSRS updates closes a real gap; and the paper is transparent about the mismatch between Proposition 5 (which assumes a stop-gradient or target network) and the online-network potential that actually works. The tabular verification is consistent with the theory, and the code is public.\n\nThe soft spots are exactly what I would send back to the authors. First, separate the theoretical claims from the heuristic deep-RL findings; right now a reader could reasonably conclude BSRS is 'proven' in Atari, which is not true. Second, run a fixed-η evaluation where η is selected on a validation split, or at least report all η curves with error bars and confidence intervals. Third, either extend the convergence analysis to the online-potential setting or explicitly label the Atari and TD3 results as outside the theory. The Pendulum result is a nice bonus, but it is a single easy environment; Hopper and Humanoid in the appendix are honestly reported as insignificant.\n\nFor whom: anyone working on reward shaping or on sample-efficient value-based RL will find the theoretical piece useful. This is a solid workshop-to-conference paper, not a field-changer. A serious editor should send it to referees, with the expectation of a major revision focused on the evaluation protocol.","headline":"The tabular theory is sound but the headline Atari results live outside its guarantees; the paper deserves revision, not dismissal.","tokens_in":13774,"tokens_out":2690,"would_cite":true,"duration_ms":24847,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Bootstrapped reward shaping uses the agent's current value estimate as the shaping potential, giving a convergent, policy-preserving way to densify rewards without hand-crafted potentials.","keywords":["reward shaping","potential-based reward shaping","bootstrap","value function","reinforcement learning","sample efficiency","Atari","deep Q-learning"],"falsifier":"Run a deep Q-learning agent on a fixed Atari game under BSRS, computing the potential from a frozen target network instead of the fast-changing online network, while keeping all other settings identical. If the documented speedup disappears or reverses, then the contraction-based tabular theory is not what drives the deep-learning gain; the gain would instead come from the online potential's rapidly changing values acting as a transient regularizer or exploration signal.","tokens_in":12753,"feed_emoji":"🎮","tokens_out":6968,"duration_ms":56707,"temperature":0.7,"pith_summary":"This paper introduces bootstrapped reward shaping (BSRS), a method that removes the main obstacle in potential-based reward shaping: choosing the potential function. Instead of hand-designing a potential from domain knowledge, BSRS uses the agent's current estimate of the optimal state-value function, $V^{(n)}(s)$, as the potential. The paper proves that in tabular Markov decision processes the resulting shaped Bellman operator is a contraction for shape-scale values $\\eta \\in (-1, (1-\\gamma)/(1+\\gamma))$, so the learning process converges and preserves the optimal policy. The convergence limit is shown explicitly: $Q_\\infty(s,a)=Q_0(s,a) - \\frac{\\eta}{1+\\eta}V_0(s)$ with potential $\\Phi_\\infty(s)=\\frac{\\eta}{1+\\eta}V_0(s)$. Empirically, applying the one-line BSRS change to deep Q-learning speeds training on the Atari suite and improves a continuous-control TD3 agent, suggesting that a single scalar hyperparameter can replace task-specific reward design.","feed_headline":"Self-shaped rewards lift Atari aggregate score from 45% to 60%","feed_subtitle":"One scalar parameter replaces task-specific reward design, with a tabular convergence proof.","key_machinery":"The central object is the self-shaped Bellman operator $T$, formed by substituting the current value-function estimate into the potential-based shaping reward. The contraction inequality $\\lVert TU - TW \\rVert_\\infty \\le (\\lvert\\eta\\rvert + \\gamma\\lvert 1+\\eta\\rvert) \\lVert U-W\\rVert_\\infty$ determines the admissible range of the shape-scale $\\eta$; Banach's fixed-point theorem then guarantees convergence, and the limit is found by solving a self-consistent equation for $\\Phi_\\infty$. A second mechanism is the rescaling view: BSRS with parameter $\\eta$ produces the same updates as an unshaped problem with learning rate $\\alpha(1+\\eta)$, reward $r/(1+\\eta)$, and, when the potential uses a stop-gradient, an $\\ell^2$ penalty on the advantage function. This rescaling gives intuition for why the method changes training speed without changing the optimal policy.","core_discovery":"The paper's central claim is that bootstrapping the shaping potential from the agent's own value function is convergent and policy-preserving, not merely a heuristic. The key result, Theorem 2, states that for any $\\eta$ in the interval $(-1, (1-\\gamma)/(1+\\gamma))$, the operator $TQ(s,a) = r(s,a) - \\eta \\max_a Q(s,a) + \\gamma(1+\\eta)\\mathbb{E}_{s'\\sim p}\\max_{a'}Q(s',a')$ is a contraction in the supremum norm. At the fixed point, the shaped values become $Q_\\infty = Q_0 - \\frac{\\eta}{1+\\eta}V_0$ and the shaping potential settles at $\\Phi_\\infty = \\frac{\\eta}{1+\\eta}V_0$; because the correction depends only on the state, the greedy policy is identical to the optimal policy of the original MDP. The paper also shows that BSRS is not equivalent to any static potential (Remark 3) and that, in the TD(0) and SARSA(0) settings, the same updates can be viewed as an unshaped problem with a rescaled learning rate and reward, plus an advantage-regularization term in the SARSA case. These results are offered as theoretical grounding for the observed deep-RL speedups.","pith_inferences":["If the contraction theorem transfers to deep RL, BSRS effectively acts as an adaptive regularizer on the advantage function; this could be tested by measuring whether the advantage term's norm decreases during BSRS training compared with baseline DQN.","The paper reports that values of $\\eta$ beyond the proven range still improve convergence, an 'edge of stability' phenomenon; a direct test would be to compute the spectral radius of the shaped operator at those $\\eta$ values and see whether the dynamics stay bounded in an equilibrium sense.","A state-dependent shape scale $\\eta(s)$, which the paper mentions as future work, could be warm-started from visitation counts or TD errors and would turn BSRS into a built-in exploration mechanism.","If the online-versus-target network comparison holds across more environments, it would suggest that frequent potential updates (even noisy ones) aid exploration more than stability aids credit assignment, a claim separable from the paper's main convergence result."],"forward_implications":["In tabular MDPs, BSRS converges for any shape scale $\\eta \\in (-1, (1-\\gamma)/(1+\\gamma))$, reaching a known limiting value function and leaving the optimal policy unchanged.","Practitioners can apply BSRS as a one-line change to existing value-based algorithms and tune only the scalar $\\eta$ instead of designing a task-specific potential function.","On the Atari suite, BSRS improves the aggregate median human-normalized score from about 45 percent to 60 percent, with positive effects in most of the 40 games.","In the continuous-action setting, using the actor's sampled action to approximate the max in the potential gives TD3 agents faster, more robust learning on a pendulum task.","Because BSRS is not equivalent to any static potential, its benefit cannot be reproduced by choosing a fixed shaping function; the dynamic dependence on the agent's value estimate is essential."],"supporting_citations":[{"why":"Supplies the PBRS theorem that a potential difference added to rewards preserves the optimal policy; BSRS builds on exactly this shaping form.","marker":"Ng, Harada, and Russell 1999"},{"why":"Establishes that static PBRS is equivalent to Q-value initialization, the baseline that BSRS is shown to generalize beyond.","marker":"Wiewiora 2003"},{"why":"Provides the proof technique and rescaling interpretation used to show BSRS equals an unshaped problem with rescaled learning rate and reward.","marker":"Amit, Meir, and Ciosek 2020"},{"why":"Defines the Atari Arcade Learning Environment that the main deep-RL experiments are run on.","marker":"Bellemare et al. 2013"},{"why":"Provides the DQN algorithm and hyperparameters that BSRS modifies with its single-line reward change.","marker":"Mnih et al. 2015"},{"why":"Supplies the Stable-Baselines3 implementations of DQN and TD3 used for the Atari and continuous-control experiments.","marker":"Raffin et al. 2021"},{"why":"Defines dynamic PBRS with a time-dependent potential; the paper distinguishes its own training-step-varying potential from this setting.","marker":"Devlin and Kudenko 2012"},{"why":"Studies using the optimal value function as a potential in meta-learning; the paper positions BSRS as the single-task, bootstrap alternative to that idea.","marker":"Zou et al. 2021"}],"fun_headline_variants":["Agent sculpts its own rewards to speed learning","Reward shaping goes bootstrapped: value-driven boost","Self-shaped rewards cut Atari grind, proven convergent","Value-driven shaping speeds Atari training, with proof"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The convergence theorem assumes synchronous tabular value iteration, but the deep-learning experiments compute the shaping potential from the online network rather than a target network, which the paper's own Proposition 5 does not cover; the Atari speedup therefore depends on the unproven assumption that the tabular contraction result transfers to nonlinear function approximation with stochastic gradient updates.","fun_headline_variants_meta":{"raw":{"variants":["Agent sculpts its own rewards to speed learning","Reward shaping goes bootstrapped: value-driven boost","Self-shaped rewards cut Atari grind, proven convergent","Value-driven shaping speeds Atari training, with proof"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000139,"raw_usage":{"total_tokens":1160,"prompt_tokens":949,"completion_tokens":211,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":148}},"tokens_in":565,"tokens_out":211,"duration_ms":2816,"temperature":1.0,"reasoning_tokens":148,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:38:35.361439+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a deep Q-learning agent on a fixed Atari game under BSRS, computing the potential from a frozen target network instead of the fast-changing online network, while keeping all other settings identical. If the documented speedup disappears or reverses, then the contraction-based tabular theory is not what drives the deep-learning gain; the gain would instead come from the online potential's rapidly changing values acting as a transient regularizer or exploration signal.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes that static PBRS is equivalent to Q-value initialization, the baseline that BSRS is shown to generalize beyond."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the proof technique and rescaling interpretation used to show BSRS equals an unshaped problem with rescaled learning rate and reward."},{"cited_title":"M.; and Kudenko, D","cited_arxiv_id":null,"evidence_quote":"Defines dynamic PBRS with a time-dependent potential; the paper distinguishes its own training-step-varying potential from this setting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Studies using the optimal value function as a potential in meta-learning; the paper positions BSRS as the single-task, bootstrap alternative to that idea."}],"review_version":1}