Bootstrapped Reward Shaping

Jacob Adamczyk; Rahul V. Kulkarni; Stas Tiomkin; Volodymyr Makarenko

REVIEW 3 major objections 6 minor 36 references

Bootstrapped Reward Shaping

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Bootstrapped reward shaping uses the agent's current value estimate as the shaping potential, giving a convergent, policy-preserving way to densify rewards without hand-crafted potentials.

desk verdict The tabular theory is sound but the headline Atari results live outside its guarantees; the paper deserves revision, not dismissal. read the letter →

arxiv 2501.00989 v2 pith:RRYNIMFS submitted 2025-01-02 cs.LG cs.AI

Jacob Adamczyk , Volodymyr Makarenko , Stas Tiomkin , Rahul V. Kulkarni This is my paper · ORCID

classification cs.LGcs.AI

keywords rewardshapingpotential-basedbootstrapvaluefunctionreinforcementlearningsampleefficiencyAtarideepQ-learning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces bootstrapped reward shaping (BSRS), a method that removes the main obstacle in potential-based reward shaping: choosing the potential function. Instead of hand-designing a potential from domain knowledge, BSRS uses the agent's current estimate of the optimal state-value function, $V^{(n)}(s)$, as the potential. The paper proves that in tabular Markov decision processes the resulting shaped Bellman operator is a contraction for shape-scale values $\eta \in (-1, (1-\gamma)/(1+\gamma))$, so the learning process converges and preserves the optimal policy. The convergence limit is shown explicitly: $Q_\infty(s,a)=Q_0(s,a) - \frac{\eta}{1+\eta}V_0(s)$ with potential $\Phi_\infty(s)=\frac{\eta}{1+\eta}V_0(s)$. Empirically, applying the one-line BSRS change to deep Q-learning speeds training on the Atari suite and improves a continuous-control TD3 agent, suggesting that a single scalar hyperparameter can replace task-specific reward design.

What carries the argument

The central object is the self-shaped Bellman operator $T$, formed by substituting the current value-function estimate into the potential-based shaping reward. The contraction inequality $\lVert TU - TW \rVert_\infty \le (\lvert\eta\rvert + \gamma\lvert 1+\eta\rvert) \lVert U-W\rVert_\infty$ determines the admissible range of the shape-scale $\eta$; Banach's fixed-point theorem then guarantees convergence, and the limit is found by solving a self-consistent equation for $\Phi_\infty$. A second mechanism is the rescaling view: BSRS with parameter $\eta$ produces the same updates as an unshaped problem with learning rate $\alpha(1+\eta)$, reward $r/(1+\eta)$, and, when the potential uses a stop-gradient, an $\ell^2$ penalty on the advantage function. This rescaling gives intuition for why the method changes training speed without changing the optimal policy.

What would settle it

Run a deep Q-learning agent on a fixed Atari game under BSRS, computing the potential from a frozen target network instead of the fast-changing online network, while keeping all other settings identical. If the documented speedup disappears or reverses, then the contraction-based tabular theory is not what drives the deep-learning gain; the gain would instead come from the online potential's rapidly changing values acting as a transient regularizer or exploration signal.

Watch

Extended reading notes

Core claim

The paper's central claim is that bootstrapping the shaping potential from the agent's own value function is convergent and policy-preserving, not merely a heuristic. The key result, Theorem 2, states that for any $\eta$ in the interval $(-1, (1-\gamma)/(1+\gamma))$, the operator $TQ(s,a) = r(s,a) - \eta \max_a Q(s,a) + \gamma(1+\eta)\mathbb{E}_{s'\sim p}\max_{a'}Q(s',a')$ is a contraction in the supremum norm. At the fixed point, the shaped values become $Q_\infty = Q_0 - \frac{\eta}{1+\eta}V_0$ and the shaping potential settles at $\Phi_\infty = \frac{\eta}{1+\eta}V_0$; because the correction depends only on the state, the greedy policy is identical to the optimal policy of the original MDP. The paper also shows that BSRS is not equivalent to any static potential (Remark 3) and that, in the TD(0) and SARSA(0) settings, the same updates can be viewed as an unshaped problem with a rescaled learning rate and reward, plus an advantage-regularization term in the SARSA case. These results are offered as theoretical grounding for the observed deep-RL speedups.

Load-bearing premise

The convergence theorem assumes synchronous tabular value iteration, but the deep-learning experiments compute the shaping potential from the online network rather than a target network, which the paper's own Proposition 5 does not cover; the Atari speedup therefore depends on the unproven assumption that the tabular contraction result transfers to nonlinear function approximation with stochastic gradient updates.

Editorial extensions

If this is right

In tabular MDPs, BSRS converges for any shape scale $\eta \in (-1, (1-\gamma)/(1+\gamma))$, reaching a known limiting value function and leaving the optimal policy unchanged.
Practitioners can apply BSRS as a one-line change to existing value-based algorithms and tune only the scalar $\eta$ instead of designing a task-specific potential function.
On the Atari suite, BSRS improves the aggregate median human-normalized score from about 45 percent to 60 percent, with positive effects in most of the 40 games.
In the continuous-action setting, using the actor's sampled action to approximate the max in the potential gives TD3 agents faster, more robust learning on a pendulum task.
Because BSRS is not equivalent to any static potential, its benefit cannot be reproduced by choosing a fixed shaping function; the dynamic dependence on the agent's value estimate is essential.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the contraction theorem transfers to deep RL, BSRS effectively acts as an adaptive regularizer on the advantage function; this could be tested by measuring whether the advantage term's norm decreases during BSRS training compared with baseline DQN.
The paper reports that values of $\eta$ beyond the proven range still improve convergence, an 'edge of stability' phenomenon; a direct test would be to compute the spectral radius of the shaped operator at those $\eta$ values and see whether the dynamics stay bounded in an equilibrium sense.
A state-dependent shape scale $\eta(s)$, which the paper mentions as future work, could be warm-started from visitation counts or TD errors and would turn BSRS into a built-in exploration mechanism.
If the online-versus-target network comparison holds across more environments, it would suggest that frequent potential updates (even noisy ones) aid exploration more than stability aids credit assignment, a claim separable from the paper's main convergence result.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the tabular theory is the real contribution, and it is in good shape—the contraction range, the fixed point, and the non-equivalence to any static potential are new and correctly argued. But the headline empirical claim is not backed by the theory: in the Atari experiments γ=0.99, so Theorem 2's contraction bound is η<0.005, and every nonzero η tested is at least 100 times larger. The authors concede this and gesture at 'edge of stability,' but that is an analogy, not a proof. On top of that, the Atari evaluation picks the best η per environment from the test set, so the reported 45%→60% median gain is an upper bound over hyperparameters, not a fixed-configuration result. The same mismatch shows up in the tabular gridworld: with γ=0.8, the bound is η<0.111, and the speedup curve peaks roughly three times higher.

What is genuinely good: the idea of bootstrapping the potential from the agent's current value estimate is a natural but previously unformalized step from Ng et al. and Devlin-Kudenko; the proof that no static potential reproduces the BSRS updates closes a real gap; and the paper is transparent about the mismatch between Proposition 5 (which assumes a stop-gradient or target network) and the online-network potential that actually works. The tabular verification is consistent with the theory, and the code is public.

The soft spots are exactly what I would send back to the authors. First, separate the theoretical claims from the heuristic deep-RL findings; right now a reader could reasonably conclude BSRS is 'proven' in Atari, which is not true. Second, run a fixed-η evaluation where η is selected on a validation split, or at least report all η curves with error bars and confidence intervals. Third, either extend the convergence analysis to the online-potential setting or explicitly label the Atari and TD3 results as outside the theory. The Pendulum result is a nice bonus, but it is a single easy environment; Hopper and Humanoid in the appendix are honestly reported as insignificant.

For whom: anyone working on reward shaping or on sample-efficient value-based RL will find the theoretical piece useful. This is a solid workshop-to-conference paper, not a field-changer. A serious editor should send it to referees, with the expectation of a major revision focused on the evaluation protocol.

Referee Report

3 major / 6 minor

Summary. The paper proposes BSRS (Bootstrapped Reward Shaping), a potential-based reward-shaping method in which the potential is the agent's current state-value estimate: Phi(n)(s) = eta * max_a Q(n)(s,a). The central theoretical result, Theorem 2, shows in a synchronous tabular setting that the shaped Bellman operator is a contraction for eta in (-1, (1-gamma)/(1+gamma)), with fixed point Q_infinity(s,a) = Q0(s,a) - (eta/(1+eta)) V0(s), so the greedy policy is unchanged. The paper also derives TD(0) and SARSA(0) equivalence interpretations (Propositions 4 and 5), and reports experiments on a tabular gridworld, the Atari suite with DQN, and Pendulum with TD3, claiming that BSRS improves training speed while requiring only a single-line code change.

Significance. If the claims hold, BSRS is an attractive minimal-intervention shaping method: it removes the need for task-specific potential design and reduces the shaping search to a scalar eta. The tabular contraction proof is clear, the fixed-point calculation is transparent, and the paper releases code. The explicit contraction bound is a genuine strength. The significance is limited, however, by a systematic mismatch between the proven regime and the experiments: all positive eta values used in the Atari study lie far outside Theorem 2's contraction range, and the deep RL implementation computes the potential with the online network despite Proposition 5 assuming a stop-gradient/target network. These gaps must be addressed before the empirical speedups can be presented as consequences of the theory.

major comments (3)

[Experiments (Atari and tabular), Figures 1-3; Theorem 2] Theorem 2 guarantees contraction only for eta in (-1, (1-gamma)/(1+gamma)). With the Atari discount gamma=0.99 this is eta < 0.005, yet all positive eta values tested in Figures 1 and 2 (0.5, 1, 2, 3, 5, 10) lie far outside this range, and eta approximately 2 is reported as optimal. The same mismatch appears in the tabular study: for gamma=0.8 the proven bound is eta < 0.111, while Figure 3 shows speedups at eta roughly three times larger. Consequently, the reported speedups are not consequences of Theorem 2; no convergence or policy-invariance guarantee covers them. The comparison to the 'edge of stability' is an analogy, not a proof. The authors should either run experiments inside the proven eta range or provide a separate analysis that covers the larger eta values actually used.
[Function Approximation for Phi; Proposition 5] Proposition 5 explicitly assumes that V(s) is computed with a stop-gradient or target network (see Eq. 19). The experimental section states that using the online network to compute Phi gives better performance and that this contradicts Proposition 5. Therefore the regularized-SARSA equivalence in Proposition 5 does not hold for the deep RL experiments, and the Atari/TD3 speedups cannot be attributed to that theoretical result. To make the theory support the deep RL results, the experiments should use the target/stop-gradient network for Phi, or the paper should prove an analogous result for the online-network case.
[Scope of Theorem 2 and deep RL transfer] Theorem 2 is proved for synchronous tabular value iteration on bounded value functions. The DQN and TD3 experiments use neural function approximation, minibatch stochastic gradient updates, replay buffers, and exploration policies; no argument is given that the contraction constant or fixed-point formula transfers to this setting. The 'Interpretation' paragraphs provide intuition, not convergence guarantees. The abstract and introduction state that the paper 'provides convergence proofs' and 'shows improved training speed' without clearly separating the proven tabular statements from the heuristic deep RL claims. This separation should be made explicit wherever the theory is invoked to explain the deep RL gains.

minor comments (6)

[Theorem 2, Eq. (8)] The theorem statement writes Phi_infinity(s) = V0(s)/(1+eta), but the proof and the definition Phi(n)(s)=eta max_a Q(n)(s,a) give Phi_infinity(s) = eta*V0(s)/(1+eta). Please correct this typo in Eq. (8).
[Title page] The author list contains a spacing artifact ('V olodymyr'); this should be corrected in the camera-ready version.
[Figure 1] Figure 1 selects the best nonzero eta per environment, which is an oracle selection. The text should state clearly that this per-environment tuning is not evidence for a single scalar hyperparameter; the aggregate curve in Figure 2 is the more direct support for the fixed-eta claim.
[Proposition 4] The proof of Proposition 4 is only a pointer to another paper's technique. Since Proposition 4 is stated as a result, the derivation should be included or the statement should be labeled as a direct corollary of that technique.
[Proof of Remark 3] The proof of Remark 3 is hard to follow because V(n)(s') is used both for the max of the inner backup and for the updated state-value function. Please disambiguate these two objects.
[Atari reporting] The claim that self-shaping has a negative impact in only 6/40 environments should be accompanied by error bars or a statistical significance test, since each environment uses five seeds.

Circularity Check

0 steps flagged · score 1.0 of 10

No load-bearing circularity: BSRS's self-referential potential is the algorithm itself, and its convergence is established by a standard contraction argument against the externally given MDP.

full rationale

The paper's central derivation (Theorem 2) is not circular. The potential Phi^(n)(s)=eta max_a Q^(n)(s,a) is defined from the agent's current value estimate, making the shaped Bellman operator self-referential; however, the proof does not assume the conclusion. It verifies that ||TU - TW||_inf <= (|eta|+gamma|1+eta|)||U - W||_inf and invokes Banach's fixed point theorem, then solves the fixed-point equation self-consistently. The fixed point Q_inf(s,a)=Q_0(s,a)-(eta/(1+eta))V_0(s) is derived, not assumed. Propositions 4 and 5 are explicit algebraic equivalences between BSRS updates and rescaled learning-rate/reward (plus a regularization term in the SARSA case); these are reductions that lower novelty, and they are described honestly as interpretations rather than presented as predictions. The experiments tune eta and report the best result, which is hyperparameter selection, not a fitted parameter renamed as a prediction. The only self-citations are contextual (prior PBRS work) or a non-central remark that the entropy-regularized extension follows from Theorem 1 of Adamczyk et al. 2023a; none is load-bearing for the main convergence claim or the Atari speedup. The paper also explicitly notes that its deep-RL experiments use the online network for Phi, contradicting Proposition 5's assumption, and that the tabular experiments exceed the proven eta range; these are acknowledged extrapolations and correctness risks, not circular reasoning. Overall the derivation chain is self-contained against an externally given MDP, so no circular step is exhibited.

Assumptions & free parameters 1 free parameters · 4 assumptions · 1 invented entities

The central claim rests on one tuned scalar, eta, and on standard MDP and contraction assumptions. The only nonstandard load-bearing choice is using the online network for the potential in deep RL, which the paper admits falls outside its own Proposition 5. The BSRS potential itself is a new mathematical construction whose evidence is internal to the paper.

free parameters (1)

shape-scale eta = tested over {0, 0.5, 1, 2, 3, 5, 10}; aggregate optimum near eta=2 for Atari, best per-environment values selected ex…
The magnitude of the BSRS potential is a scalar hyperparameter chosen by the authors. The headline improvements depend on this choice, and the per-environment best is selected after seeing test performance.

assumptions (4)

standard math Potential-based reward shaping preserves the optimal policy for any bounded potential (Ng, Harada, and Russell 1999, Theorem 1).
Used as the foundation for the shaped Bellman equation in the Theory section.
standard math Banach's fixed point theorem applies to the shaped Bellman operator in the sup norm.
Invoked in the proof of Theorem 2 to guarantee a unique fixed point under the contraction condition.
domain assumption The MDP has bounded rewards and discount factor gamma < 1, and the state-value function is exactly computed as max_a Q in the tabular setting.
The convergence proof relies on standard MDP assumptions stated in the Background section.
ad hoc to paper In the deep RL experiments, using the online network rather than the target network to compute the potential is a heuristic that improves performance, and this contradicts Proposition 5.
The paper states this choice contradicts the theoretical equivalence and yet uses it for the headline Atari results, so the deep-RL speedup is not derived from the theory.

invented entities (1)

Bootstrapped dynamic potential Phi(n)(s) = eta * max_a Q(n)(s,a)
purpose: Provides the potential-based shaping reward gamma * Phi(s') - Phi(s) using the agent's current value estimate instead of the unknown V-star.
The construction is supported only by the paper's own convergence proof and experiments. There is no external dataset or independent benchmark that validates the potential as a general object beyond this work.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Bootstrapped Reward Shaping." pith.science (2026). https://pith.science/paper/RRYNIMFS

@misc{pith2026250100989,
  author       = {Pith},
  title        = {Pith review of: Bootstrapped Reward Shaping},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RRYNIMFS}},
  note         = {Machine review of arXiv:2501.00989}
}

read the original abstract

In reinforcement learning, especially in sparse-reward domains, many environment steps are required to observe reward information. In order to increase the frequency of such observations, "potential-based reward shaping" (PBRS) has been proposed as a method of providing a more dense reward signal while leaving the optimal policy invariant. However, the required "potential function" must be carefully designed with task-dependent knowledge to not deter training performance. In this work, we propose a "bootstrapped" method of reward shaping, termed BSRS, in which the agent's current estimate of the state-value function acts as the potential function for PBRS. We provide convergence proofs for the tabular setting, give insights into training dynamics for deep RL, and show that the proposed method improves training speed in the Atari suite.

Figures

Figures reproduced from arXiv: 2501.00989 by the authors.

**Figure 2.** Learning curves for 10M steps in the Atari suite. We take the median human-normalized score over all [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 4.** TD3 with BSRS is tested on a continuous control [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

Figures from the paper (3 more)

**Figure 3.** Figure 3: Upon solving the self-shaped Bellman equation in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png]

**Figure 5.** Figure 5: We also test using the target network to calculate [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

**Figure 6.** Figure 6: We also test BSRS on TD3 for more complex continuous action tasks. The performance is slightly better (at least for [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 24 canonical work pages

[1]

Adamczyk, J.; Arriojas, A.; Tiomkin, S.; and Kulkarni, R. V. 2023 a . Utilizing Prior Solutions for Reward Shaping and Composition in Entropy-Regularized Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 37(6): 6658--6665

2023
[2]

Adamczyk, J.; Makarenko, V.; Arriojas, A.; Tiomkin, S.; and Kulkarni, R. V. 2023 b . Bounding the optimal value function in compositional reinforcement learning. In Evans, R. J.; and Shpitser, I., eds., Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, volume 216 of Proceedings of Machine Learning Research, 22--32. PMLR

2023
[3]

Adamczyk, J.; Makarenko, V.; Tiomkin, S.; and Kulkarni, R. V. 2024. Boosting Soft Q-Learning by Bounding. Reinforcement Learning Journal, 5: 2373--2399

2024
[4]

T.; Suarez, F.; and Zhang, Y

Ahn, K.; Bubeck, S.; Chewi, S.; Lee, Y. T.; Suarez, F.; and Zhang, Y. 2024. Learning threshold neurons via edge of stability. Advances in Neural Information Processing Systems, 36

work page 2024
[5]

Amit, R.; Meir, R.; and Ciosek, K. 2020. Discount factor as a regularizer in reinforcement learning. In International conference on machine learning, 269--278. PMLR

work page 2020
[6]

G.; Naddaf, Y.; Veness, J.; and Bowling, M

Bellemare, M. G.; Naddaf, Y.; Veness, J.; and Bowling, M. 2013. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253--279

2013
[7]

Cao, H.; Cohen, S.; and Szpruch, . 2021. Identifiability in inverse reinforcement learning. Advances in Neural Information Processing Systems, 34: 12362--12373

work page 2021
[8]

Centa, M.; and Preux, P. 2023. Soft action priors: towards robust policy transfer. Proceedings of the AAAI Conference on Artificial Intelligence, 37(6): 6953--6961

work page 2023

Show all 36 references

[9]

Z.; and Talwalkar, A

Cohen, J.; Kaur, S.; Li, Y.; Kolter, J. Z.; and Talwalkar, A. 2021. Gradient descent on neural networks typically occurs at the edge of stability. In International Conference on Learning Representations

2021
[10]

H.; Klyne, H.; Zhang, E.; Laidlaw, C.; Tambe, M.; and Doshi-Velez, F

Cooke, L. H.; Klyne, H.; Zhang, E.; Laidlaw, C.; Tambe, M.; and Doshi-Velez, F. 2023. Toward computationally efficient inverse reinforcement learning via reward shaping. arXiv preprint arXiv:2312.09983

2023 arXiv
[11]

M.; and Kudenko, D

Devlin, S. M.; and Kudenko, D. 2012. Dynamic potential-based reward shaping. In 11th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2012), 433--440. IFAAMAS

2012
[12]

Fujimoto, S.; Hoof, H.; and Meger, D. 2018. Addressing function approximation error in actor-critic methods. In International conference on machine learning, 1587--1596. PMLR

2018
[13]

Gao, Y.; and Toni, F. 2015. Potential based reward shaping for hierarchical reinforcement learning. In Twenty-Fourth International Joint Conference on Artificial Intelligence

2015
[14]

Gleave, A.; Dennis, M.; Legg, S.; Russell, S.; and Leike, J. 2021. Quantifying differences in reward functions. International Conference on Learning Representations

2021
[15]

Grze \'s , M.; and Kudenko, D. 2010. Online learning of shaping rewards in reinforcement learning. Neural networks, 23(4): 541--550

2010
[16]

Gupta, A.; Pacchiano, A.; Zhai, Y.; Kakade, S.; and Levine, S. 2022. Unpacking reward shaping: Understanding the benefits of reward engineering on sample complexity. Advances in Neural Information Processing Systems, 35: 15281--15295

2022
[17]

Haarnoja, T.; Zhou, A.; Hartikainen, K.; Tucker, G.; Ha, S.; Tan, J.; Kumar, V.; Zhu, H.; Gupta, A.; Abbeel, P.; et al. 2018. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905

2018 arXiv
[18]

Hu, Y.; Wang, W.; Jia, H.; Wang, Y.; Chen, Y.; Hao, J.; Wu, F.; and Fan, C. 2020. Learning to utilize shaping rewards: A new approach of reward shaping. Advances in Neural Information Processing Systems, 33: 15931--15941

2020
[19]

Jenner, E.; Skalse, J. M. V.; and Gleave, A. 2022. A general framework for reward function distances. In NeurIPS ML Safety Workshop

2022
[20]

Jenner, E.; van Hoof, H.; and Gleave, A. 2022. Calculus on MDPs: Potential Shaping as a Gradient. arXiv preprint arXiv:2208.09570

2022 arXiv
[21]

Jiang, Y.; Bharadwaj, S.; Wu, B.; Shah, R.; Topcu, U.; and Stone, P. 2021. Temporal-logic-based reward shaping for continuing reinforcement learning tasks. In Proceedings of the AAAI Conference on artificial Intelligence, volume 35, 7995--8003

2021
[22]

V.; Sima, K.; and Leong, T.-Y

Ma, H.; Luo, Z.; Vo, T. V.; Sima, K.; and Leong, T.-Y. 2024. Highly efficient self-adaptive reward shaping for reinforcement learning. arXiv preprint arXiv:2408.03029

2024 arXiv
[23]

Mataric, M. J. 1994. Reward Functions for Accelerated Learning. In Cohen, W. W.; and Hirsh, H., eds., Machine Learning Proceedings 1994, 181--189. San Francisco (CA): Morgan Kaufmann. ISBN 978-1-55860-335-6

1994
[24]

A.; Veness, J.; Bellemare, M

Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A. A.; Veness, J.; Bellemare, M. G.; Graves, A.; Riedmiller, M.; Fidjeland, A. K.; Ostrovski, G.; et al. 2015. Human-level control through deep reinforcement learning. nature, 518(7540): 529--533

2015
[25]

Naik, A.; Wan, Y.; Tomar, M.; and Sutton, R. S. 2024. Reward Centering. Reinforcement Learning Journal, 4: 1995--2016

2024
[26]

Y.; Harada, D.; and Russell, S

Ng, A. Y.; Harada, D.; and Russell, S. 1999. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the 16th International Conference on Machine Learning, volume 99, 278--287

1999
[27]

Raffin, A.; Hill, A.; Gleave, A.; Kanervisto, A.; Ernestus, M.; and Dormann, N. 2021. Stable-Baselines3: Reliable Reinforcement Learning Implementations. Journal of Machine Learning Research, 22(268): 1--8

2021
[28]

Randl v, J.; and Alstr m, P. 1998. Learning to Drive a Bicycle Using Reinforcement Learning and Shaping. In Shavlik, J. W., ed., Proceedings of the Fifteenth International Conference on Machine Learning (ICML 1998), Madison, Wisconsin, USA, July 24-27, 1998 , 463--471. Morgan Kaufmann

1998
[29]

Skalse, J.; Howe, N.; Krasheninnikov, D.; and Krueger, D. 2022. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35: 9460--9471

2022
[30]

Wang, Y.; Yang, M.; Dong, R.; Sun, B.; Liu, F.; et al. 2024. Efficient potential-based exploration in reinforcement learning using inverse dynamic bisimulation metric. Advances in Neural Information Processing Systems, 36

2024
[31]

Wiewiora, E. 2003. Potential-based shaping and Q-value initialization are equivalent. Journal of Artificial Intelligence Research, 19: 205--208

2003
[32]

Wulfe, B.; Balakrishna, A.; Ellis, L.; Mercat, J.; McAllister, R.; and Gaidon, A. 2022. Dynamics-aware comparison of learned reward functions. International Conference on Learning Representations

2022
[33]

Zhang, B.; Rajan, R.; Pineda, L.; Lambert, N.; Biedenkapp, A.; Chua, K.; Hutter, F.; and Calandra, R. 2021. On the importance of hyperparameter optimization for model-based reinforcement learning. In International Conference on Artificial Intelligence and Statistics, 4015--4023. PMLR

2021
[34]

Zou, H.; Ren, T.; Yan, D.; Su, H.; and Zhu, J. 2021. Learning task-distribution reward shaping with meta-learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 11210--11218

2021
[35]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...
[36]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Adamczyk, J.; Arriojas, A.; Tiomkin, S.; and Kulkarni, R. V. 2023 a . Utilizing Prior Solutions for Reward Shaping and Composition in Entropy-Regularized Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 37(6): 6658--6665

2023

[2] [2]

Adamczyk, J.; Makarenko, V.; Arriojas, A.; Tiomkin, S.; and Kulkarni, R. V. 2023 b . Bounding the optimal value function in compositional reinforcement learning. In Evans, R. J.; and Shpitser, I., eds., Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, volume 216 of Proceedings of Machine Learning Research, 22--32. PMLR

2023

[3] [3]

Adamczyk, J.; Makarenko, V.; Tiomkin, S.; and Kulkarni, R. V. 2024. Boosting Soft Q-Learning by Bounding. Reinforcement Learning Journal, 5: 2373--2399

2024

[4] [4]

T.; Suarez, F.; and Zhang, Y

Ahn, K.; Bubeck, S.; Chewi, S.; Lee, Y. T.; Suarez, F.; and Zhang, Y. 2024. Learning threshold neurons via edge of stability. Advances in Neural Information Processing Systems, 36

work page 2024

[5] [5]

Amit, R.; Meir, R.; and Ciosek, K. 2020. Discount factor as a regularizer in reinforcement learning. In International conference on machine learning, 269--278. PMLR

work page 2020

[6] [6]

G.; Naddaf, Y.; Veness, J.; and Bowling, M

Bellemare, M. G.; Naddaf, Y.; Veness, J.; and Bowling, M. 2013. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253--279

2013

[7] [7]

Cao, H.; Cohen, S.; and Szpruch, . 2021. Identifiability in inverse reinforcement learning. Advances in Neural Information Processing Systems, 34: 12362--12373

work page 2021

[8] [8]

Centa, M.; and Preux, P. 2023. Soft action priors: towards robust policy transfer. Proceedings of the AAAI Conference on Artificial Intelligence, 37(6): 6953--6961

work page 2023