{"id":"78afe792-9f43-4dac-909f-b5c1e43ef1c1","arxiv_id":"2502.01307","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A constant bias added to a potential-based reward shaping function, set to compensate the external reward and the initial Q-values, improves sample efficiency while preserving the optimal policy.","lead":"This paper derives a formula for shifting the potential function in reward shaping so that the shaped rewards align with the agent's starting Q-values, improving sample efficiency in sparse-reward reinforcement learning. The result gives practitioners a concrete rule for choosing a bias term without changing the encoded task preferences.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The uniform-Q_init/first-update assumption underlies the b=(1-γ)Q_init-r∞ formula; deep-RL experiments never measure or satisfy it, so the paper's central practical claim is only demonstrated in tabular settings.","rationale":"The paper has a crisp formal contribution: under uniform Q_init and a constant external reward r∞, the linear shift b=(1−γ)Q_init−r∞ makes the first non-terminal TD update direction depend only on the potential difference F, not on Q_init or r∞ (Eqs. 4–8). I checked the algebra: after shifting Φ_b=Φ+b/(γ−1), F_b=F+b, so R′=r∞+F_b=F+(1−γ)Q_init, and R′+γQ_init−Q_init=F. The Gridworld experiments with Q_init∈{−1,0,1} and both reward functions are consistent with this first-update analysis, and the paper honestly flags the terminal-state side effects in §5.2 and §6.2.1. The load-bearing weakness is the scope of the generalization: the equality Q(s_t,a_t)=max_a Q(s_{t+1},a)=Q_init is not just a convenient simplification, it is the exact condition under which the bias formula is derived. With function approximation and experience replay, there is no constant Q_init and no first-update event; the authors' own §6.2.2 note confirms the Q-values for unvisited states change during training, and they choose bias values based on a rough probe rather than the formula. Therefore the central claim's practical motivation—applicability in deep RL without adjusting Q-values—is not established, although the tabular result is. This is a genuine scope gap, not a mathematical error, so a conditional verdict with a request for a direct Q-value measurement is the right disposition.","tokens_in":66,"tokens_out":8700,"duration_ms":143783,"concrete_test":"Run the DQN Mountain Car setup with logging of the network's Q-values just before each update for non-terminal transitions sampled from the replay buffer. For each early update, record (Q(s_t,a_t), max_a Q(s_{t+1},a)); evaluate whether the fixed bias b reproduces the sign of Eq. (5), i.e. whether R'(s,a,s')+γQ(s_t,a_t)−Q(s_t,a_t) has the same sign as the potential difference. Then vary b across the empirical distribution of (1−γ)Q(s,a)−r∞ (e.g., mean, median, 10th/90th percentile). If the learning curves do not peak near the value derived from the logged Q-values, or if no single b matches most early updates, the deep-RL generalization is not supported by the current experiments.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in §5.2 depends on the Section 5 identity Q(s_t,a_t)=max_a Q(s_{t+1},a)=Q_init, which requires all Q-values to start at one known constant and the transition to be the first update of both the current pair and the next state's maximizing action. The bias b=(1−γ)Q_init−r∞ removes the dependence on r∞ and Q_init only under that identity; if Q-values are non-uniform or already updated, the cancellation in Eq. (4) does not occur and the chosen bias is no longer the one that zeroes the mismatch. This matters precisely for the deep-RL part of the paper: DQN uses a neural-network Q-function and a replay buffer, so there is no single Q_init and no meaningful 'first update' for a transition sampled repeatedly. The authors concede in §6.2.2 that they do not focus on matching bias values to initial Q-values and that Q-values for unvisited states change during training, but they nevertheless conclude that the approach 'also holds for function approximation in deep RL.' The Gridworld experiments are a clean validation under the uniform-Q_init assumption; the deep-RL experiments use arbitrary bias values (e.g., b=1 for Mountain Car) and therefore do not test the formula or its purported independence from Q-values. This leaves the central claim's scope—improving sample efficiency without adjusting initial Q-values in deep reinforcement learning—unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper analyzes how potential-based reward shaping interacts with initial Q-values and the constant external reward in sparse-reward MDPs. It derives a set of inequalities (Eqs. (6)--(10)) that characterize when the first TD update increases or decreases an action's Q-value, and proposes shifting the potential by a constant b, with the recommended choice b = (1−γ)Q_init − r∞, so that the first-update condition no longer depends on r∞ or Q_init. The paper also argues that continuous potential functions cannot simultaneously assign correct positive/negative shaping rewards for all small potential differences, and it advocates exponentially growing potentials. Experiments on Gridworld with tabular Q-learning confirm the predicted bias values for the goal-directed and on-step reward functions. Experiments with DQN on Cart Pole and Mountain Car use a constant bias but do not compute b from measured initial Q-values; the authors nevertheless conclude that the approach also holds for function approximation in deep RL.","tokens_in":11681,"tokens_out":5487,"duration_ms":51568,"significance":"If the central claim is accepted, the paper gives a simple, parameter-free prescription for removing two nuisance dependencies in potential-based reward shaping, which is a widely used technique. The derivations in Section 5 are algebraically correct under the stated first-update assumption, and the Gridworld experiments cleanly validate the predicted bias values across three different initializations and two reward functions. The paper also correctly highlights the previously underappreciated coupling between shaping rewards and initial Q-values, and it usefully extends the requirements of Grzes and Kudenko. However, the deep-RL portion does not actually test the derived formula, and the first-update assumption is not satisfied under replay-based function approximation; the paper's own Section 6.2.2 admits this. The practical scope of the contribution is therefore narrower than the title and conclusion suggest, though the tabular results are solid and the theoretical framework is a worthwhile contribution.","major_comments":[{"comment":"The first-update simplification Q(s_t,a_t) = max_a Q(s_{t+1},a) = Q_init is valid only when the transition is the first update for both the current state–action pair and the maximizing action in the successor state; after any replay or revisit, the equality fails. The bias b = (1−γ)Q_init − r∞ is derived from this equality, so in deep RL with a replay buffer and a neural-network Q-function, the cancellation in Eq. (5) does not obtain. Section 6.2.2 concedes that Q-values for unvisited states change during training, yet Section 7 concludes that the approach “also holds for function approximation in deep RL.” Since the deep-RL experiments never measure Q_init or enforce the first-update condition, this generalization is not established by the paper.","section":"§5, Eqs. (4)–(5), §6.2.2"},{"comment":"The deep-RL experiments do not test the proposed bias formula. For Cart Pole (r∞ = +1) and Mountain Car (r∞ = −1), the authors probe a small set of arbitrary bias values (e.g., b = −1, 0, 1) and explicitly state that they do not match the bias to the initial Q-values. The observed improvement shows only that some constant shift can help in these environments; it does not validate b = (1−γ)Q_init − r∞ or the claimed independence from Q_init. To support the central claim in the deep-RL setting, the experiments should either compute b from an empirically determined Q_init (for example, the distribution of network outputs for randomly sampled states) and show that the predicted value works, or the conclusions should be restricted to the tabular setting.","section":"§6.2.2, §6.2.3, Fig. 4"},{"comment":"The requirement that the potential be zero at terminal states appears to be violated by the experimental potential functions as described. For Gridworld, Φ(s) = −d(s,s_g) normalized to [0,1] gives Φ(goal) = 1 unless the terminal value is explicitly reset to zero; the same issue affects the normalized angle and velocity potentials in Cart Pole and Mountain Car, where terminal states generally have nonzero potentials. Since §5.2 motivates the bias shift by excluding terminal states from the shift, the experiments must state explicitly how terminal potentials are handled. Without that, the policy-invariance guarantee is not applicable, and the observed effects may reflect a changed objective rather than the proposed first-update mechanism.","section":"§5.2, §6.2.1"}],"minor_comments":[{"comment":"The bias is written as “b = −R + (1−γ)Q_init,” but the reward notation in Section 3.1 is r∞; please align the notation to avoid confusion between the reward function R and the constant reward r∞.","section":"§6.2.1"},{"comment":"Using e both for the exponential base and as the symbol for the base parameter is confusing, especially since the base can be 8, 32, or 64; consider renaming the base to, for example, c or β.","section":"§5.3, Fig. 2"},{"comment":"The paper states that base values larger than e = 32 “did not show any different results,” but no sensitivity plots or numerical evidence are provided; adding a brief sensitivity analysis would strengthen the choice of this hyperparameter.","section":"§6.1.3"},{"comment":"The phrase “without having to adjust the initial Q-values” is potentially misleading, because the proposed bias still depends on knowing Q_init; a more precise wording would be “without modifying the Q-value initialization scheme, provided Q_init is known.”","section":"Abstract and §1"},{"comment":"The manuscript does not state whether code and hyperparameter configurations are publicly available; adding a reproducibility statement would be useful for a machine-learning audience.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper's tabular contribution is sound and the Gridworld validation is convincing. The main weakness is the overreach into deep RL: the experiments there use arbitrary bias values and do not satisfy the first-update assumption, so the concluding claim that the approach 'holds for function approximation in deep RL' is not supported. The remark in Section 6.2.2 honestly acknowledges the gap, but the conclusion does not reflect it. A revision that restricts the deep-RL claims, or that adds an experiment computing the bias from measured network outputs and testing the predicted value, would resolve my concern. There is also a possible discrepancy between the terminal-potential-zero requirement and the potentials used in the experiments; this should be clarified regardless."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper contributes a small but correct bias formula for potential-based reward shaping, b = (1−γ)Q_init − r∞, that removes the dependence of the shaped reward on the constant external reward and the initial Q-values for non-terminal transitions. That is a genuine new recommendation, derived directly from the known PBRS/Q-init equivalence and prior requirements. The algebra is clean, and the Gridworld experiments confirm the predicted bias values.\n\nWhat I like: the paper is clearly written, does not misstate policy invariance, and explicitly flags the terminal-state issue and the practical difficulty of applying the bias in deep RL. The inequality framework is a useful synthesis, and the Section 5.3 limitation of continuous potentials is a real (if incremental) observation.\n\nThe soft spots are in the deep RL part. The theory only characterizes the sign of the first TD update, not convergence speed, so the sample-efficiency claim rests on the experiments. Those experiments do not actually test the formula: the authors state they did not match bias to Q_init because Q-values drift, and they use hand-picked bias values (e.g., b=1 for Mountain Car). The Cart Pole and Mountain Car results show that some biases are better than others, but they do not verify that b=(1−γ)Q_init−r∞ is the right value. The conclusion that 'this approach holds for function approximation' is therefore an overclaim. The formula also depends on a uniform Q_init, which is not satisfied with a neural network; the paper acknowledges this but still generalizes.\n\nThese are addressable. The proper test would measure or control Q_init in DQN and compare the formula to alternatives. Given the paper's transparency and the correctness of the tabular results, I'd send it to review. It is a modest, useful contribution to the reward-shaping literature, not a breakthrough. A serious referee should insist on fixing the deep RL claims or softening them.\n\nI might bring it to a reading group only if people work on reward shaping. I'd cite the bias formula in my own work on PBRS.","headline":"Small, correct bias formula for PBRS with honest tabular validation; deep RL part is exploratory and overclaims, but the core idea is worth refereeing.","tokens_in":12279,"tokens_out":2684,"would_cite":true,"duration_ms":23620,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Potential-based reward shaping can be made reliably effective by adding one constant bias to the potential, chosen from the initial Q-values and the step reward, without changing the optimal policy.","keywords":["potential-based reward shaping","Q-value initialization","sample efficiency","reward shaping","reinforcement learning","policy invariance","sparse rewards","deep Q-network"],"falsifier":"Run tabular Q-learning in a Gridworld with known $Q_{\\mathrm{init}}$ and $r_\\infty$, and compare the bias $b=(1-\\gamma)Q_{\\mathrm{init}} - r_\\infty$ against the same bias perturbed by a small amount; if the perturbed bias converges to the optimal policy within the same training budget as often as the predicted one, the claim that this specific bias drives the sample-efficiency gain is refuted. In the deep RL setting, estimate the distribution of initial Q-values for unvisited states from freshly initialized networks, set the bias from that distribution, and check across network seeds whether the improvement over no shaping is stable; a seed where the bias makes learning slower than no shaping would show that the single-constant-$Q_{\\mathrm{init}}$ assumption is load-bearing.","tokens_in":11174,"feed_emoji":"🎯","tokens_out":7273,"duration_ms":59004,"temperature":0.7,"pith_summary":"This paper argues that the usefulness of potential-based reward shaping is silently controlled by two quantities usually treated as irrelevant: the initial Q-values and the constant external reward per step. When those do not match the shaped rewards, the shaping signal can incentivize the wrong transitions and slow learning. The paper's central proposal is to add a constant bias to the potential function, choosing the bias so that the shaped rewards are measured relative to $(1-\\gamma)Q_{\\mathrm{init}}$ and the step reward. With the right bias, the same potential function encodes the same state preferences but produces correct positive and negative incentives from the first updates, improving sample efficiency in sparse-reward tasks. The claim is verified in tabular Gridworlds and in DQN experiments on Cart Pole and Mountain Car.","feed_headline":"One bias term makes reward shaping guide exploration correctly","feed_subtitle":"Choosing the shift from initial Q-values and external reward makes shaped rewards guide exploration, not the goal","key_machinery":"The load-bearing device is the first-update simplification of Q-learning: before any update, $Q(s_t,a_t)$ and $\\max_a Q(s_{t+1},a)$ both equal a single constant $Q_{\\mathrm{init}}$, so the sign of the update is determined by whether the shaped reward $R'(s,a,s')$ is above or below $(1-\\gamma)Q_{\\mathrm{init}}$. The paper converts the requirement 'reward greater than this threshold for good transitions, less for bad ones' into inequalities, and then shows that shifting the potential by $b/(\\gamma-1)$ shifts every non-terminal shaped reward by $b$. Choosing $b=(1-\\gamma)Q_{\\mathrm{init}} - r_\\infty$ cancels the threshold and the constant step reward simultaneously, so the inequalities reduce to pure comparisons of potential values. An exponential variant $e^{\\Phi(s)}$ is introduced to handle small potential differences, and the same bias logic removes the dependence on the previous potential value.","core_discovery":"The central discovery is that a simple linear shift of the potential function, $\\Phi_b(s) = \\Phi(s) + b/(\\gamma-1)$, separates the shaping signal from the accidental baseline set by $Q_{\\mathrm{init}}$ and $r_\\infty$. Setting $b = (1-\\gamma)Q_{\\mathrm{init}} - r_\\infty$ makes the condition for a first update to raise or lower a Q-value depend only on the potential difference, so transitions toward higher potential are incentivized and transitions away are disincentivized from the very first visit. This removes the need to tune initial Q-values to make shaping effective, which matters in deep reinforcement learning where initialization cannot be freely adjusted. The paper also proves a limitation: no continuous potential function can assign correct positive and negative shaping rewards for all small potential changes, and scaling alone cannot fix the baseline mismatch in goal-directed MDPs.","pith_inferences":["A natural extension the paper does not pursue is to make the bias time-dependent, since in deep RL the Q baseline for unvisited states drifts during training; updating $b$ online could preserve the first-update benefit throughout learning.","Read alongside the equivalence between reward shifts and Q-value initialization, the bias formula gives a principled way to choose optimistic versus pessimistic starts without changing the network initialization.","Because the shift does not correct terminal transitions, short-episode and truncation-heavy tasks may need a supplementary terminal-aware shaping term; the paper notes the failure mode but does not propose that fix."],"forward_implications":["In tabular Q-learning with known $Q_{\\mathrm{init}}$ and step reward, setting the bias once makes every first visit to a state–action pair carry the correct incentive without changing the potential's encoded preferences.","Because the shifted shaping is still potential-based, policy invariance is preserved: only the speed of learning changes, not the optimal policy.","For goal-directed MDPs, scaling the potential cannot compensate for a mismatch between external rewards and initial Q-values; the only available correction is the shift.","In deep RL, the same bias logic transfers to DQN despite unvisited states having non-constant Q-values, as shown for Cart Pole and Mountain Car.","Continuous potential functions inevitably mislabel small potential improvements as negative or positive shaping rewards; exponential potentials with the correct bias reduce the range of these mislabels."],"supporting_citations":[{"why":"Defines potential-based reward shaping and proves policy invariance, the guarantee that the shifted potential preserves the optimal policy.","marker":"[12]"},{"why":"Proves that potential-based shaping is equivalent to shifting initial Q-values, which motivates the analysis of how initial values interact with shaped rewards.","marker":"[17]"},{"why":"Establishes that potential must be zero in terminal states for policy invariance in episodic MDPs, which limits the bias correction for terminal transitions.","marker":"[5]"},{"why":"Provides the earlier requirements for effective reward shaping that the paper extends with initial Q-values and external rewards.","marker":"[6]"},{"why":"Supplies the goal-directed reward formulation used in the theory and experiments.","marker":"[8]"},{"why":"Introduces the exponentially growing potential functions that the paper adapts to handle small potential differences with the bias correction.","marker":"[11]"}],"fun_headline_variants":["Shift potential to make shaping effective","A bias term fixes reward shaping in RL","How to align shaping rewards with exploration","Reward shaping: shift to guide correctly"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The bias prescription rests on the assumption that, at the time of the first update, every relevant Q-value equals one known constant $Q_{\\mathrm{init}}$ and that the update is indeed the first visit to that state–action pair; under function approximation or once states are revisited, that equality no longer holds and the promised improvement is not guaranteed.","fun_headline_variants_meta":{"raw":{"variants":["Shift potential to make shaping effective","A bias term fixes reward shaping in RL","How to align shaping rewards with exploration","Reward shaping: shift to guide correctly"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000246,"raw_usage":{"total_tokens":1517,"prompt_tokens":903,"completion_tokens":614,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":562}},"tokens_in":519,"tokens_out":614,"duration_ms":6409,"temperature":1.0,"reasoning_tokens":562,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T15:47:25.787948+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run tabular Q-learning in a Gridworld with known $Q_{\\mathrm{init}}$ and $r_\\infty$, and compare the bias $b=(1-\\gamma)Q_{\\mathrm{init}} - r_\\infty$ against the same bias perturbed by a small amount; if the perturbed bias converges to the optimal policy within the same training budget as often as the predicted one, the claim that this specific bias drives the sample-efficiency gain is refuted. In the deep RL setting, estimate the distribution of initial Q-values for unvisited states from freshly initialized networks, set the bias from that distribution, and check across network seeds whether the improvement over no shaping is stable; a seed where the bias makes learning slower than no shaping would show that the single-constant-$Q_{\\mathrm{init}}$ assumption is load-bearing.","supporting_citations":[{"cited_title":"Ng, Daishi Harada, and Stuart J","cited_arxiv_id":null,"evidence_quote":"Defines potential-based reward shaping and proves policy invariance, the guarantee that the shifted potential preserves the optimal policy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes that potential must be zero in terminal states for policy invariance in episodic MDPs, which limits the bias correction for terminal transitions."},{"cited_title":"Laurent, and Nadine Le Fort-Piat","cited_arxiv_id":null,"evidence_quote":"Supplies the goal-directed reward formulation used in the theory and experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the exponentially growing potential functions that the paper adapts to handle small potential differences with the bias correction."}],"review_version":1}