Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Improving the Effectiveness of Potential-Based Reward Shaping in Reinforcement Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Potential-based reward shaping can be made reliably effective by adding one constant bias to the potential, chosen from the initial Q-values and the step reward, without changing the optimal policy.

desk verdict Small, correct bias formula for PBRS with honest tabular validation; deep RL part is exploratory and overclaims, but the core idea is worth refereeing. read the letter →

arxiv 2502.01307 v1 pith:ZNMFHG6I submitted 2025-02-03 cs.LG

Henrik Müller , Daniel Kudenko This is my paper · ORCID
classification cs.LG
keywords potential-basedrewardshapingQ-valueinitializationsampleefficiencyreinforcementlearningpolicyinvariancesparserewardsdeepQ-network
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the usefulness of potential-based reward shaping is silently controlled by two quantities usually treated as irrelevant: the initial Q-values and the constant external reward per step. When those do not match the shaped rewards, the shaping signal can incentivize the wrong transitions and slow learning. The paper's central proposal is to add a constant bias to the potential function, choosing the bias so that the shaped rewards are measured relative to $(1-\gamma)Q_{\mathrm{init}}$ and the step reward. With the right bias, the same potential function encodes the same state preferences but produces correct positive and negative incentives from the first updates, improving sample efficiency in sparse-reward tasks. The claim is verified in tabular Gridworlds and in DQN experiments on Cart Pole and Mountain Car.

What carries the argument

The load-bearing device is the first-update simplification of Q-learning: before any update, $Q(s_t,a_t)$ and $\max_a Q(s_{t+1},a)$ both equal a single constant $Q_{\mathrm{init}}$, so the sign of the update is determined by whether the shaped reward $R'(s,a,s')$ is above or below $(1-\gamma)Q_{\mathrm{init}}$. The paper converts the requirement 'reward greater than this threshold for good transitions, less for bad ones' into inequalities, and then shows that shifting the potential by $b/(\gamma-1)$ shifts every non-terminal shaped reward by $b$. Choosing $b=(1-\gamma)Q_{\mathrm{init}} - r_\infty$ cancels the threshold and the constant step reward simultaneously, so the inequalities reduce to pure comparisons of potential values. An exponential variant $e^{\Phi(s)}$ is introduced to handle small potential differences, and the same bias logic removes the dependence on the previous potential value.

What would settle it

Run tabular Q-learning in a Gridworld with known $Q_{\mathrm{init}}$ and $r_\infty$, and compare the bias $b=(1-\gamma)Q_{\mathrm{init}} - r_\infty$ against the same bias perturbed by a small amount; if the perturbed bias converges to the optimal policy within the same training budget as often as the predicted one, the claim that this specific bias drives the sample-efficiency gain is refuted. In the deep RL setting, estimate the distribution of initial Q-values for unvisited states from freshly initialized networks, set the bias from that distribution, and check across network seeds whether the improvement over no shaping is stable; a seed where the bias makes learning slower than no shaping would show that the single-constant-$Q_{\mathrm{init}}$ assumption is load-bearing.

Watch

Extended reading notes

Core claim

The central discovery is that a simple linear shift of the potential function, $\Phi_b(s) = \Phi(s) + b/(\gamma-1)$, separates the shaping signal from the accidental baseline set by $Q_{\mathrm{init}}$ and $r_\infty$. Setting $b = (1-\gamma)Q_{\mathrm{init}} - r_\infty$ makes the condition for a first update to raise or lower a Q-value depend only on the potential difference, so transitions toward higher potential are incentivized and transitions away are disincentivized from the very first visit. This removes the need to tune initial Q-values to make shaping effective, which matters in deep reinforcement learning where initialization cannot be freely adjusted. The paper also proves a limitation: no continuous potential function can assign correct positive and negative shaping rewards for all small potential changes, and scaling alone cannot fix the baseline mismatch in goal-directed MDPs.

Load-bearing premise

The bias prescription rests on the assumption that, at the time of the first update, every relevant Q-value equals one known constant $Q_{\mathrm{init}}$ and that the update is indeed the first visit to that state–action pair; under function approximation or once states are revisited, that equality no longer holds and the promised improvement is not guaranteed.

Editorial extensions

If this is right

  • In tabular Q-learning with known $Q_{\mathrm{init}}$ and step reward, setting the bias once makes every first visit to a state–action pair carry the correct incentive without changing the potential's encoded preferences.
  • Because the shifted shaping is still potential-based, policy invariance is preserved: only the speed of learning changes, not the optimal policy.
  • For goal-directed MDPs, scaling the potential cannot compensate for a mismatch between external rewards and initial Q-values; the only available correction is the shift.
  • In deep RL, the same bias logic transfers to DQN despite unvisited states having non-constant Q-values, as shown for Cart Pole and Mountain Car.
  • Continuous potential functions inevitably mislabel small potential improvements as negative or positive shaping rewards; exponential potentials with the correct bias reduce the range of these mislabels.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not pursue is to make the bias time-dependent, since in deep RL the Q baseline for unvisited states drifts during training; updating $b$ online could preserve the first-update benefit throughout learning.
  • Read alongside the equivalence between reward shifts and Q-value initialization, the bias formula gives a principled way to choose optimistic versus pessimistic starts without changing the network initialization.
  • Because the shift does not correct terminal transitions, short-episode and truncation-heavy tasks may need a supplementary terminal-aware shaping term; the paper notes the failure mode but does not propose that fix.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper analyzes how potential-based reward shaping interacts with initial Q-values and the constant external reward in sparse-reward MDPs. It derives a set of inequalities (Eqs. (6)--(10)) that characterize when the first TD update increases or decreases an action's Q-value, and proposes shifting the potential by a constant b, with the recommended choice b = (1−γ)Q_init − r∞, so that the first-update condition no longer depends on r∞ or Q_init. The paper also argues that continuous potential functions cannot simultaneously assign correct positive/negative shaping rewards for all small potential differences, and it advocates exponentially growing potentials. Experiments on Gridworld with tabular Q-learning confirm the predicted bias values for the goal-directed and on-step reward functions. Experiments with DQN on Cart Pole and Mountain Car use a constant bias but do not compute b from measured initial Q-values; the authors nevertheless conclude that the approach also holds for function approximation in deep RL.

Significance. If the central claim is accepted, the paper gives a simple, parameter-free prescription for removing two nuisance dependencies in potential-based reward shaping, which is a widely used technique. The derivations in Section 5 are algebraically correct under the stated first-update assumption, and the Gridworld experiments cleanly validate the predicted bias values across three different initializations and two reward functions. The paper also correctly highlights the previously underappreciated coupling between shaping rewards and initial Q-values, and it usefully extends the requirements of Grzes and Kudenko. However, the deep-RL portion does not actually test the derived formula, and the first-update assumption is not satisfied under replay-based function approximation; the paper's own Section 6.2.2 admits this. The practical scope of the contribution is therefore narrower than the title and conclusion suggest, though the tabular results are solid and the theoretical framework is a worthwhile contribution.

major comments (3)
  1. [§5, Eqs. (4)–(5), §6.2.2] The first-update simplification Q(s_t,a_t) = max_a Q(s_{t+1},a) = Q_init is valid only when the transition is the first update for both the current state–action pair and the maximizing action in the successor state; after any replay or revisit, the equality fails. The bias b = (1−γ)Q_init − r∞ is derived from this equality, so in deep RL with a replay buffer and a neural-network Q-function, the cancellation in Eq. (5) does not obtain. Section 6.2.2 concedes that Q-values for unvisited states change during training, yet Section 7 concludes that the approach “also holds for function approximation in deep RL.” Since the deep-RL experiments never measure Q_init or enforce the first-update condition, this generalization is not established by the paper.
  2. [§6.2.2, §6.2.3, Fig. 4] The deep-RL experiments do not test the proposed bias formula. For Cart Pole (r∞ = +1) and Mountain Car (r∞ = −1), the authors probe a small set of arbitrary bias values (e.g., b = −1, 0, 1) and explicitly state that they do not match the bias to the initial Q-values. The observed improvement shows only that some constant shift can help in these environments; it does not validate b = (1−γ)Q_init − r∞ or the claimed independence from Q_init. To support the central claim in the deep-RL setting, the experiments should either compute b from an empirically determined Q_init (for example, the distribution of network outputs for randomly sampled states) and show that the predicted value works, or the conclusions should be restricted to the tabular setting.
  3. [§5.2, §6.2.1] The requirement that the potential be zero at terminal states appears to be violated by the experimental potential functions as described. For Gridworld, Φ(s) = −d(s,s_g) normalized to [0,1] gives Φ(goal) = 1 unless the terminal value is explicitly reset to zero; the same issue affects the normalized angle and velocity potentials in Cart Pole and Mountain Car, where terminal states generally have nonzero potentials. Since §5.2 motivates the bias shift by excluding terminal states from the shift, the experiments must state explicitly how terminal potentials are handled. Without that, the policy-invariance guarantee is not applicable, and the observed effects may reflect a changed objective rather than the proposed first-update mechanism.
minor comments (5)
  1. [§6.2.1] The bias is written as “b = −R + (1−γ)Q_init,” but the reward notation in Section 3.1 is r∞; please align the notation to avoid confusion between the reward function R and the constant reward r∞.
  2. [§5.3, Fig. 2] Using e both for the exponential base and as the symbol for the base parameter is confusing, especially since the base can be 8, 32, or 64; consider renaming the base to, for example, c or β.
  3. [§6.1.3] The paper states that base values larger than e = 32 “did not show any different results,” but no sensitivity plots or numerical evidence are provided; adding a brief sensitivity analysis would strengthen the choice of this hyperparameter.
  4. [Abstract and §1] The phrase “without having to adjust the initial Q-values” is potentially misleading, because the proposed bias still depends on knowing Q_init; a more precise wording would be “without modifying the Q-value initialization scheme, provided Q_init is known.”
  5. [General] The manuscript does not state whether code and hyperparameter configurations are publicly available; adding a reproducibility statement would be useful for a machine-learning audience.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the b-shift theorem is an algebraic consequence of a stated constructor choice, not a fit; the one self-citation (exponential potential) is re-derived and not load-bearing.

full rationale

The derivation chain is self-contained. The shifted potential Φ_b(s) = Φ(s) + b/(γ−1) adds the constant b to every non-terminal shaping reward, so substituting b = (1−γ)Q_init − r∞ into the first-update comparison reduces R'(s,a,s′) + γQ_init ≶ Q_init to γΦ(s′) − Φ(s) ≶ 0. This is an algebraic identity: b is defined from stated quantities (Q_init and r∞), and no fitted parameter is relabeled as a prediction. The first-update simplification Q(s_t,a_t) = max_a Q(s_{t+1},a) = Q_init is a transparent assumption, and its violation under function approximation is explicitly acknowledged in §6.2.2 ('we do not focus on matching the bias values to possible initial Q-values as the Q-values for not visited states can change at any time during the training'), so this is an applicability limitation, not a circular step. The only author self-citation is Müller et al. [11] for the exponentially growing potential; the present paper re-derives the relevant cancellation in eqs. (19)–(21), making the citation motivational rather than load-bearing. The deep-RL experiments test a range of biases without measuring Q_init, which weakens the empirical generalization but does not make the central claim circular. No step in the paper reduces to its inputs by construction, and no imported uniqueness theorem or fitted constant forces the conclusion.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central result rests on standard PBRS theory and the simplifying assumption of a known uniform Q_init. The bias formula itself introduces no fitted parameters, but the exponential potential base and the environment-specific potential functions are hand-chosen. The first-update assumption is the most fragile because it is violated in deep RL.

free parameters (3)
  • Exponential potential base e = 32
    Set to 32 for all experiments. The paper states 'Larger values for e did not show any different results in our experiments' and no systematic sensitivity analysis is provided. The choice affects the smallest potential difference δ that yields correct shaping rewards (Section 5.3).
  • Bias b in deep RL experiments = scanned over values (e.g., -1, 0, 1)
    In Cart Pole and Mountain Car, b is scanned rather than computed from the formula because Q_init is not measurable. The working values are interpreted post hoc. In tabular Gridworld, b is varied and the theoretically predicted value is confirmed.
  • Potential function designs = Gridworld: 1 - d(s,s_g)/D; Cart Pole: normalized |angle|; Mountain Car: normalized |velocity|
    Hand-selected heuristic potentials for each environment. The method requires a meaningful potential function; the choice is not derived from the theory.
assumptions (4)
  • domain assumption Potential-based reward shaping guarantees policy invariance (Ng et al. 1999).
    Used throughout to ensure the bias shift does not change the optimal policy. Cited from [12].
  • domain assumption Potential values in terminal states must be zero for policy invariance in episodic MDPs (Grześ 2017).
    Invoked in Section 5.1 and 5.2 to justify excluding terminal states from the bias shift.
  • domain assumption PBRS is equivalent to Q-value initialization (Wiewiora 2003).
    Basis for relating shaping rewards to initial Q-values. Cited from [17].
  • ad hoc to paper For the first TD update all Q-values equal the constant Q_init.
    The derivation of inequalities (6)-(10) assumes Q(s_t,a_t) = max_a Q(s_{t+1},a) = Q_init for first updates. This holds for tabular RL with uniform initialization but not for deep RL with function approximation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving the Effectiveness of Potential-Based Reward Shaping in Reinforcement Learning." pith.science (2026). https://pith.science/paper/ZNMFHG6I

@misc{pith2026250201307,
  author       = {Pith},
  title        = {Pith review of: Improving the Effectiveness of Potential-Based Reward Shaping in Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZNMFHG6I}},
  note         = {Machine review of arXiv:2502.01307}
}
read the original abstract

Potential-based reward shaping is commonly used to incorporate prior knowledge of how to solve the task into reinforcement learning because it can formally guarantee policy invariance. As such, the optimal policy and the ordering of policies by their returns are not altered by potential-based reward shaping. In this work, we highlight the dependence of effective potential-based reward shaping on the initial Q-values and external rewards, which determine the agent's ability to exploit the shaping rewards to guide its exploration and achieve increased sample efficiency. We formally derive how a simple linear shift of the potential function can be used to improve the effectiveness of reward shaping without changing the encoded preferences in the potential function, and without having to adjust the initial Q-values, which can be challenging and undesirable in deep reinforcement learning. We show the theoretical limitations of continuous potential functions for correctly assigning positive and negative reward shaping values. We verify our theoretical findings empirically on Gridworld domains with sparse and uninformative reward functions, as well as on the Cart Pole and Mountain Car environments, where we demonstrate the application of our results in deep reinforcement learning.

Figures

Figures reproduced from arXiv: 2502.01307 by the authors.

Figure 1
Figure 1. Average length of evaluation runs (with 𝜖 = 0.05) on a 25x25 Gridworld with potential-based reward shaping where Φ(𝑠) = 𝑉 ∗ (𝑠). be able to exploit this knowledge to bias the action selection to repeat these actions. But whether the RL agent is able to exploit to shaped rewards to easily recognize optimal actions relies entirely on the Q-Value initialization, which (although generally well established) has not been … view at source ↗
Figure 2
Figure 2. Plots of the reward shaping F that will be added to the reward given the difference [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Gridworld results for the two different reward functions [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Results of Cart Pole and Mountain Car experiments for different values of the bias parameter [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Unified Framework for Dynamic Reward Shaping in Reinforcement Learning

    cs.AI 2026-08 accept novelty 4.0 of 10

    A unified framework and review of dynamic reward shaping, with a new taxonomy over twelve method families and a gap analysis of optimality guarantees.

Reference graph

Works this paper leans on

18 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    Barto, Richard S

    Andrew G. Barto, Richard S. Sutton, and Charles W. Anderson. 1983. Neu- ronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics SMC-13, 5 (1983), 834–846. https://doi.org/10.1109/TSMC.1983.6313077

    work page arXiv 1983

  2. [2]

    Taylor, and Ann Nowé

    Tim Brys, Anna Harutyunyan, Halit Bener Suay, Sonia Chernova, Matthew E. Taylor, and Ann Nowé. 2015. Reinforcement Learning from Demonstration through Shaping. In Proceedings of the 24th International Conference on Artificial Intelligence (Buenos Aires, Argentina) (IJCAI’15). AAAI Press, 3352–3358

    work page 2015

  3. [3]

    Sam Devlin and Daniel Kudenko. 2012. Dynamic potential-based reward shap- ing. In International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2012, Valencia, Spain, June 4-8, 2012 (3 Volumes) . IFAAMAS, 433–440. http://dl.acm.org/citation.cfm?id=2343638

    work page 2012

  4. [4]

    Mahmoud Elbarbari, Florent Delgrange, Ivo Vervlimmeren, Kyriakos Efthymiadis, Bram Vanderborght, and Ann Nowe. 2022. A framework for flexibly guiding learning agents. Neural Computing and Applications (06 2022). https://doi.org/ 10.1007/s00521-022-07396-x

    work page doi:10.1007/s00521-022-07396-x 2022

  5. [5]

    Marek Grześ. 2017. Reward Shaping in Episodic Reinforcement Learning. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems (São Paulo, Brazil) (AAMAS ’17) . International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 565–573

    work page 2017

  6. [6]

    Marek Grzes and Daniel Kudenko. 2009. Theoretical and Empirical Analysis of Reward Shaping in Reinforcement Learning. In 2009 International Conference on Machine Learning and Applications . 337–344. https://doi.org/10.1109/ICMLA. 2009.33

    doi:10.1109/icmla 2009

  7. [7]

    Melham, and Daniel Kroening

    Mohammadhosein Hasanbeig, Natasha Yogananda Jeppu, Alessandro Abate, Thomas F. Melham, and Daniel Kroening. 2021. DeepSynth: Automata Synthesis for Automatic Task Segmentation in Deep Reinforcement Learning. In AAAI Conference on Artificial Intelligence. https://api.semanticscholar.org/CorpusID: 211542062

    work page 2021

  8. [8]

    Laurent, and Nadine Le Fort-Piat

    Laëtitia Matignon, Guillaume J. Laurent, and Nadine Le Fort-Piat. 2006. Reward Function and Initial Values: Better Choices for Accelerated Goal-Directed Rein- forcement Learning. In International Conference on Artificial Neural Networks . https://api.semanticscholar.org/CorpusID:3448745

    work page 2006

Show all 18 references
  1. [9]

    Riedmiller

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. 2013. Playing Atari with Deep Reinforcement Learning. CoRR abs/1312.5602 (2013). arXiv:1312.5602 http://arxiv.org/abs/1312.5602

    2013 arXiv

  2. [10]

    Andrew William Moore. 1990. Efficient Memory-based Learning for Robot Control . Technical Report. University of Cambridge

    1990

  3. [11]

    Henrik Müller, Lukas Berg, and Daniel Kudenko. 2025. Using incomplete and incorrect plans to shape reinforcement learning in long-sequence sparse-reward tasks. Neural Computing and Applications (10 Jan 2025). https://doi.org/10.1007/ s00521-024-10615-2

    2025

  4. [12]

    Ng, Daishi Harada, and Stuart J

    Andrew Y. Ng, Daishi Harada, and Stuart J. Russell. 1999. Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping. In Proceedings of the Sixteenth International Conference on Machine Learning (ICML ’99). Morgan Kaufmann Publishers Inc., San Fra...

    1999

  5. [13]

    Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernes- tus, and Noah Dormann. 2021. Stable-Baselines3: Reliable Reinforcement Learn- ing Implementations. Journal of Machine Learning Research 22, 268 (2021), 1–8. http://jmlr.org/papers/v22/20-1364.html

    2021

  6. [14]

    Taylor, and Sonia Chernova

    Halit Bener Suay, Tim Brys, Matthew E. Taylor, and Sonia Chernova. 2016. Learn- ing from Demonstration for Shaping through Inverse Reinforcement Learning. In Proceedings of the 2016 International Conference on Autonomous Agents & Mul- tiagent Systems (Singapore, Singapore) (AA...

    2016

  7. [15]

    Hao Sun, Lei Han, Rui Yang, Xiaoteng Ma, Jian Guo, and Bolei Zhou. 2022. Optimistic Curiosity Exploration and Conservative Exploitation with Linear Reward Shaping. In Advances in Neural Information Processing Systems , Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghy...

    2022

  8. [16]

    Caroline Wang, Garrett Warnell, and Peter Stone. 2023. D-Shape: Demonstration- Shaped Reinforcement Learning via Goal-Conditioning. InProceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems (London, United Kingdom) (AAMAS ’23). Internationa...

    2023

  9. [17]

    Eric Wiewiora. 2003. Potential-Based Shaping and Q-Value Initialization are Equivalent. J. Artif. Intell. Res. 19 (2003), 205–208. https://doi.org/10.1613/JAIR. 1190

    2003 doi

  10. [18]

    Yuchen Wu, Melissa Mozifian, and Florian Shkurti. 2021. Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations Using Generative Models. In 2021 IEEE International Conference on Robotics and Automation (ICRA) (Xi’an, China). IEEE Press, 6628–6634. https://doi....

    2021

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.