Pith. sign in

REVIEW 5 major objections 6 minor 26 references

Reward Prediction Error Prioritisation in Experience Replay: The RPE-PER Method

T0 review · 5 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Prioritising replay samples by the gap between predicted and observed reward speeds and improves TD3 and SAC on continuous-control tasks.

desk verdict Simple, plausible replay-prioritisation idea, but the current experiments confound the priority signal with the new critic architecture, so 'consistently outperforms' is not yet established. read the letter →

arxiv 2501.18093 v1 pith:H7J5H6LL submitted 2025-01-30 cs.LG cs.RO

Hoda Yamani , Yuning Xing , Lee Violet C. Ong , Bruce A. MacDonald , Henry Williams This is my paper · ORCID
classification cs.LGcs.RO
keywords experiencereplayrewardpredictionerrorprioritisedcontinuouscontroloff-policyreinforcementlearningactor-criticmethodsMuJoCo
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RPE-PER is a replay-buffer sampling rule for off-policy actor-critic reinforcement learning. The paper claims that, instead of prioritising stored transitions by temporal-difference error as PER does, an agent should prioritise by reward prediction error: the absolute difference between the reward a critic network predicts for a state-action pair and the reward actually observed. The paper introduces the Enhanced Model Critic Network (EMCN), which outputs Q-values, predicted rewards, and predicted next states, and uses the reward prediction error from this network as the sampling priority. In experiments across six MuJoCo continuous-control tasks, the paper reports that RPE-PER reaches higher final performance than random sampling, PER, LAP, LA3P, and MaPER for TD3 in most tasks and for SAC in four of the six tasks. The practical interest is that the priority signal uses only the rewards already stored in the buffer, so the method is simple to add to existing actor-critic implementations.

What carries the argument

The central object is the EMCN critic network together with the RPE priority it feeds. EMCN is written as $C_\theta(s,a) = (Q_\theta(s,a), R_\theta(s,a), T_\theta(s,a))$: for a state-action pair it outputs the action value, the reward it expects, and the next state it expects, trained with the combined loss $\mathcal{L}_{C_\theta} = \xi_1 L_{Q_\theta} + \xi_2 L_{R_\theta} + \xi_3 L_{T_\theta}$. The priority for transition $i$ replaces PER's TD-error term with $\mathrm{RPE}_i = |R_\theta(s_i,a_i) - r_i|$, giving sampling probability proportional to $p_i^\alpha$ with $p_i = |\mathrm{RPE}_i| + \epsilon$. What carries the argument is that the reward head is trained on actual rewards, not bootstrapped value estimates, so the prioritisation signal reflects how well the agent currently predicts the environment's reward structure.

What would settle it

Run the same six MuJoCo tasks with every baseline given an equal per-task hyperparameter search budget, including a search over the EMCN loss weights $\xi_2$ and $\xi_3$, and compare final returns; if RPE-PER no longer beats random sampling and PER on a majority of tasks, the claimed advantage of RPE prioritisation is unsupported.

Watch

Extended reading notes

Core claim

The central claim is that reward prediction error, computed as $|R_\theta(s,a)-r|$ rather than TD error, is a more informative prioritisation signal for experience replay in continuous control. The reasoning is that transitions whose rewards the agent mispredicts carry learning value, and resampling them more often lets the actor-critic update correct its model of the environment faster. The EMCN is the load-bearing component: a critic that predicts Q-values, rewards, and next states under the weighted loss $\xi_1 L_Q + \xi_2 L_R + \xi_3 L_T$, whose reward head supplies the RPE used in the priority $p_i = |\mathrm{RPE}_i| + \epsilon$. The reported final-ten-evaluation means favour RPE-PER over random sampling, PER, LAP, LA3P, and MaPER in the majority of the six MuJoCo environments under both TD3 and SAC, with the clearest gains under TD3 and in the more complex tasks.

Load-bearing premise

The load-bearing premise is that the baseline replay methods were implemented and tuned to their best possible performance; Section 4.2 concedes that some baselines underperformed their published results, and the three EMCN loss weights ($\xi_1,\xi_2,\xi_3$) are never reported, so uneven tuning could make RPE-PER's gains an artefact of the comparison rather than of the prioritisation principle.

Editorial extensions

If this is right

  • Under TD3, RPE-PER reports higher final-ten-evaluation returns than random, PER, LAP, LA3P, and MaPER on Humanoid, HalfCheetah, Ant, Walker2d, and Hopper.
  • Under SAC, RPE-PER reports higher final returns than the baselines on Humanoid, HalfCheetah, Ant, and Swimmer, though the gains are more modest than under TD3.
  • The Swimmer result, where simpler PER outperforms RPE-PER, suggests that the RPE prioritisation advantage is concentrated in higher-dimensional or more complex tasks.
  • In Walker2d and Hopper under SAC, LA3P performs comparably or better, indicating that separating actor and critic priorities by TD error remains competitive where precise balance matters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the paper's proposed signed-RPE extension would be to prioritise the critic with absolute RPE and the actor with signed RPE, which should reveal whether RPE-PER's weakness in balance tasks comes from over-prioritising noisy transitions.
  • Because EMCN also predicts next states, a natural but untested extension is to use its predicted transitions as synthetic rollouts, turning RPE-PER into a hybrid model-based replay method.
  • The Swimmer result implies a testable complexity threshold: the advantage of RPE-PER should grow with state-action dimensionality and reward stochasticity, which could be checked on a graded family of continuous-control environments.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The manuscript introduces RPE-PER, an experience-replay prioritisation scheme for off-policy actor-critic reinforcement learning. RPE-PER replaces the standard critic with an 'Enhanced Model Critic Network' (EMCN) that outputs Q-values, a predicted reward, and a predicted next state; the absolute difference between the predicted reward and the stored reward defines the priority used in a PER-style sampling distribution (Equations 11-13). The method is evaluated on six MuJoCo continuous-control tasks under TD3 and SAC, comparing against random sampling, PER, LAP, LA3P, and MaPER, with ten seeds and reporting final-10-performance means with 95% confidence intervals (Tables 1-2, Figures 3-4). The authors claim consistent improvements in learning speed and final performance, while also acknowledging exceptions for Swimmer under TD3 and Walker2d/Hopper under SAC.

Significance. The contribution is a simple, biologically motivated prioritisation signal that could be useful for continuous-control replay, and the paper includes several good practices: ten seeds, reported confidence intervals, learning curves, and a public GitHub repository. If the reported gains were robust and attributable specifically to the RPE priority, this would be a useful addition to the experience-replay literature. However, the current evidence is not yet convincing: the experimental design conflates the prioritisation rule with the auxiliary-loss critic architecture, Tables 1-2 contain apparent data anomalies, and the headline claim of consistency is stronger than the overlapping confidence intervals support. The paper's value will depend on fixing these issues and adding the missing control experiments.

major comments (5)
  1. [Section 4, Tables 1 and 2] RPE-PER differs from every baseline in two ways: it uses the EMCN critic whose total loss includes the auxiliary reward and next-state prediction terms (Equation 11), and it replaces the sampling score with RPE. The current comparison cannot separate these factors; the reported gains could come entirely from the auxiliary losses under uniform sampling, with RPE prioritisation contributing nothing. The authors should add a 2x2 ablation crossing {standard critic, EMCN} with {random sampling, RPE priority} on at least a subset of tasks, or otherwise demonstrate that EMCN plus random sampling does not already match RPE-PER.
  2. [Table 2] In Table 2, the HalfCheetah rows for PER and LAP are identical (7558.46 +/- 1010.69), which is effectively impossible for two different algorithms across ten seeds unless a copy-paste error occurred. Additionally, the Random-Sampling Humanoid entry in Table 2 (5177.45 +/- 585.14) matches the TD3 random-sampling value in Table 1 to the second decimal, and MaPER's Hopper CI (+/- 7.42) is implausibly small relative to the other entries. These anomalies must be resolved or the corresponding conclusions cannot be evaluated.
  3. [Section 5.1 and Table 1] The statement that RPE-PER 'consistently outperforms most baseline methods' (TD3, plots a-e) is not supported by Table 1: in every TD3 environment the RPE-PER 95% CI overlaps with that of random sampling, and in Swimmer PER's mean is higher (89.64 vs 86.61). The authors should either report paired statistical tests (e.g., bootstrap or Wilcoxon across seeds) that account for the run pairing, or soften the 'consistent outperformance' claim to what the data actually show.
  4. [Sections 3.2 and 3.3] The three loss weights xi_1, xi_2, xi_3 in Equation (11) are never reported, and the statement in Section 3.2 that 'increasing xi_2 improves performance' is made without supporting evidence or sensitivity analysis. Since these weights control how much of the EMCN gradient comes from reward prediction, they are essential for reproducibility and for interpreting whether the RPE signal is actually the driver of prioritisation. Please report the values used in all experiments and include a sensitivity study or at least justify a single choice.
  5. [Section 4.2] The text acknowledges that 'some baselines underperformed compared to their published results' but provides no details on how PER, LAP, LA3P, and MaPER were tuned for these tasks. Because the central claim is comparative, the authors should supply the baseline hyperparameters, the source of each baseline implementation, and evidence that they were not disadvantaged relative to RPE-PER.
minor comments (6)
  1. [Equations (12)-(13) and Algorithm 1] Equation (12) defines RPE_i as an MSE quantity, but Equation (13) then writes p_i = |RPE_i| + epsilon; since RPE is already non-negative, the absolute value is redundant. Algorithm 1 line 12 sets p_i = (RPE_i)^alpha, which is inconsistent with Equation (13)'s additive epsilon. Please align the notation.
  2. [Abstract] The abstract uses 'Reward Predictive Error' while the rest of the paper and the method name use 'Reward Prediction Error'; please make the terminology consistent.
  3. [Figures 3 and 4] The figures use the label 'MAPER' whereas the text and tables use 'MaPER'; please standardise the label.
  4. [Algorithm 1] The pseudo-code computes RPE at line 5 using R_theta(s_t, a_t) before the buffer is updated; it would clarify whether the priority is intended to be based on the predicted reward from the current critic or from the target network.
  5. [Section 4.2] The text says the shaded regions indicate 'standard deviations calculated from ten evaluations across ten runs' while the tables report 95% confidence intervals; please clarify which quantity is shown in the figures.
  6. [References] The reference [Saglam et al., 2022] is cited as LA3P but the linked arXiv title appears to be 'Actor Prioritized Experience Replay'; please verify the citation and the naming.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's RPE priority is an endogenous network output by design, but the central claim is an empirical performance comparison, not a derived prediction; there is no load-bearing self-citation or algebraic self-reduction.

full rationale

The paper's methodological chain is definitional, not derivational. Equation (8) defines delta_R_theta = R_theta(s,a) - r, Eq. (12) sets RPE_i = |R_theta(s_i,a_i) - r_i|_MSE, and Eq. (13) sets priority p_i = |RPE_i| + epsilon. That is a method specification: RPE-PER is defined to prioritise by reward prediction error, and this does not by itself force any empirical outcome. The paper's central claim is that RPE-PER improves learning speed and final performance over random sampling, PER, LAP, LA3P, and MaPER on six MuJoCo tasks, supported by learning curves and Tables 1-2. This claim is not statistically forced by construction: the RPE is computed from the EMCN network being trained, but that is exactly the same status as TD error in PER (Eqs. 2-3); an endogenous training signal is the intended mechanism, not a fitted input mislabeled as an out-of-sample prediction. There are no self-citations used as load-bearing evidence; the only related-method citation is to external prior work MaCN/MaPER (Oh et al., 2021), and no uniqueness theorem is invoked. The Sec. 4.2 admission that 'some baselines underperformed compared to their published results due to stochasticity, different seeds, and environment updates' is a benchmarking limitation and a correctness risk, not a circularity. No equation or claim reduces to its own input by construction, so the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim relies on three tuned loss weights whose values are undisclosed, two replay hyperparameters adopted from prior work, and several domain assumptions about the usefulness and stability of the RPE signal. The fairness of the benchmarking depends on the baselines being properly tuned, which the paper suggests is not the case.

free parameters (4)
  • xi_1, xi_2, xi_3 (loss weights) = not reported
    Coefficients in the EMCN loss (Eq. 11) are never given numeric values; Section 3.2 states increasing xi_2 improves performance, indicating the weights were selected by hand or tuning, which is a free parameter affecting the RPE signal.
  • alpha (prioritisation exponent) = 0.7
    Adopted from Oh et al. (2021); chosen by hand, not swept for RPE-PER.
  • beta (importance sampling exponent) = 0.4
    Adopted from Oh et al. (2021); annealing schedule not described.
  • network hidden size and learning rate = 256 units, 3e-4
    Standard values from Fujimoto et al. and Haarnoja et al.; not tuned for RPE-PER, but they are free choices that affect the results.
assumptions (4)
  • standard math The RL problem is a finite MDP with rewards and transition probabilities as defined in Section 3.1.
    Foundation of the formalisation; unproved background assumption.
  • domain assumption Absolute reward prediction error is a useful signal for selecting valuable experiences.
    Core premise of RPE-PER; no theoretical or biological derivation is given beyond analogy.
  • domain assumption The EMCN reward predictor trained with MSE loss yields accurate and stable reward estimates throughout training.
    If the reward head is inaccurate, RPE priorities are noisy; the paper offers no analysis of reward predictor accuracy.
  • domain assumption The baseline implementations are fair and correctly tuned.
    Section 4.2 admits that some baselines underperformed compared to published results, so the comparison may be biased.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reward Prediction Error Prioritisation in Experience Replay: The RPE-PER Method." pith.science (2026). https://pith.science/paper/H7J5H6LL

@misc{pith2026250118093,
  author       = {Pith},
  title        = {Pith review of: Reward Prediction Error Prioritisation in Experience Replay: The RPE-PER Method},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H7J5H6LL}},
  note         = {Machine review of arXiv:2501.18093}
}
read the original abstract

Reinforcement Learning algorithms aim to learn optimal control strategies through iterative interactions with an environment. A critical element in this process is the experience replay buffer, which stores past experiences, allowing the algorithm to learn from a diverse range of interactions rather than just the most recent ones. This buffer is especially essential in dynamic environments with limited experiences. However, efficiently selecting high-value experiences to accelerate training remains a challenge. Drawing inspiration from the role of reward prediction errors (RPEs) in biological systems, where they are essential for adaptive behaviour and learning, we introduce Reward Predictive Error Prioritised Experience Replay (RPE-PER). This novel approach prioritises experiences in the buffer based on RPEs. Our method employs a critic network, EMCN, that predicts rewards in addition to the Q-values produced by standard critic networks. The discrepancy between these predicted and actual rewards is computed as RPE and utilised as a signal for experience prioritisation. Experimental evaluations across various continuous control tasks demonstrate RPE-PER's effectiveness in enhancing the learning speed and performance of off-policy actor-critic algorithms compared to baseline approaches.

Figures

Figures reproduced from arXiv: 2501.18093 by the authors.

Figure 1
Figure 1. A high-level representation of the RPE-PER framework and EMCN critic network. RPE is determined [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Six MuJoCo Tasks with Varying Complexity [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Learning curves are generated in selected continuous control tasks in MuJoCo under [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Learning curves are generated in selected continuous control tasks in MuJoCo under the [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 15 canonical work pages

  1. [1]

    Retroactive and graded prioritization of memory by reward

    Erin Kendall Braun, G Elliott Wimmer, and Daphna Shohamy. Retroactive and graded prioritization of memory by reward. Nature communications , 9(1):4886, 2018

    work page 2018

  2. [2]

    Prioritized Sequence Experience Replay

    Marc Brittain, Josh Bertram, Xuxi Yang, and Peng Wei. Prioritized sequence experience replay. arXiv preprint arXiv:1905.12726 , 2019

    work page Pith review arXiv 1905

  3. [3]

    Openai gym

    Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 , 2016

    arXiv 2016

  4. [4]

    Addressing function approximation error in actor-critic methods

    Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning , pages 1587--1596. PMLR, 2018

    2018

  5. [5]

    An equivalence between loss functions and non-uniform sampling in experience replay

    Scott Fujimoto, David Meger, and Doina Precup. An equivalence between loss functions and non-uniform sampling in experience replay. Advances in Neural Information Processing Systems , 33, 2020

    work page 2020

  6. [6]

    World models

    David Ha and J \"u rgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122 , 2018

    arXiv 2018

  7. [7]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861--1870. PMLR, 2018

    work page 2018

  8. [8]

    Deep reinforcement learning that matters

    Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018

    2018

Show all 26 references
  1. [9]

    Rainbow: Combining improvements in deep reinforcement learning

    Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence ...

    2018

  2. [10]

    How to train your robot with deep reinforcement learning: lessons we have learned

    Julian Ibarz, Jie Tan, Chelsea Finn, Mrinal Kalakrishnan, Peter Pastor, and Sergey Levine. How to train your robot with deep reinforcement learning: lessons we have learned. The International Journal of Robotics Research , 40(4-5):698--721, 2021

    2021

  3. [11]

    Model-free and model-based reinforcement learning, the intersection of learning and planning

    Piotr Januszewski. Model-free and model-based reinforcement learning, the intersection of learning and planning. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems , pages 1849--1851, 2022

    2022

  4. [12]

    Dopamine, updated: reward prediction error and beyond

    Talia N Lerner, Ashley L Holloway, and Jillian L Seiler. Dopamine, updated: reward prediction error and beyond. Current opinion in neurobiology , 67:123--130, 2021

    2021

  5. [13]

    Self-improving reactive agents based on reinforcement learning, planning and teaching

    Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning , 8:293--321, 1992

    1992

  6. [14]

    Human-level control through deep reinforcement learning

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature , 518(7540):529--533, 2015

    2015

  7. [15]

    Model-augmented prioritized experience replay

    Youngmin Oh, Jinwoo Shin, Eunho Yang, and Sung Ju Hwang. Model-augmented prioritized experience replay. In International Conference on Learning Representations , 2021

    2021

  8. [16]

    Curiosity-driven exploration by self-supervised prediction

    Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning , pages 2778--2787. PMLR, 2017

    2017

  9. [17]

    Building and breaking the chain: A model of reward prediction error integration and segmentation of memory

    Nina Rouhani, David Clewett, and James W Antony. Building and breaking the chain: A model of reward prediction error integration and segmentation of memory. Journal of Cognitive Neuroscience , pages 1--13, 2024

    2024

  10. [18]

    Actor prioritized experience replay

    Baturay Saglam, Furkan B Mutlu, Dogan C Cicek, and Suleyman S Kozat. Actor prioritized experience replay. arXiv preprint arXiv:2209.00532 , 2022

    2022 arXiv

  11. [19]

    Prioritized experience replay

    Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05952 , 2015

    2015 arXiv

  12. [20]

    Reward prediction error

    Wolfram Schultz. Reward prediction error. Current Biology , 27(10):R369--R371, 2017

    2017

  13. [21]

    Loss is its own reward: Self-supervision for reinforcement learning

    Evan Shelhamer, Parsa Mahmoudieh, Max Argus, and Trevor Darrell. Loss is its own reward: Self-supervision for reinforcement learning. arXiv preprint arXiv:1612.07307 , 2016

    2016 arXiv

  14. [22]

    Reward prediction error as an exploration objective in deep rl

    Riley Simmons-Edler, Ben Eisner, Daniel Yang, Anthony Bisulco, Eric Mitchell, Sebastian Seung, and Daniel Lee. Reward prediction error as an exploration objective in deep rl. arXiv preprint arXiv:1906.08189 , 2019

    1906 arXiv

  15. [23]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems , pages 5026--5033. IEEE, 2012

    2012

  16. [24]

    Experience replay optimization

    Daochen Zha, Kwei-Herng Lai, Kaixiong Zhou, and Xia Hu. Experience replay optimization. arXiv preprint arXiv:1906.08387 , 2019

    1906 arXiv

  17. [25]

    Continuously discovering novel strategies via reward-switching policy optimization

    Zihan Zhou, Wei Fu, Bingliang Zhang, and Yi Wu. Continuously discovering novel strategies via reward-switching policy optimization. arXiv preprint arXiv:2204.02246 , 2022

    2022 arXiv

  18. [26]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.