{"id":"0d1da0d6-43c9-4db8-8cc8-f8c808f116f5","arxiv_id":"2501.18093","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"RPE-PER prioritises replay buffer samples by the absolute error between a learned reward model and the actual reward, and reports improved continuous-control RL performance in MuJoCo benchmarks.","lead":"This paper proposes RPE-PER, which prioritises past experiences in reinforcement learning replay buffers by how wrong the agent's reward predictions were. The authors test it on six MuJoCo control tasks and report faster learning and better scores than several existing replay-prioritisation baselines.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RPE-PER's reported gains are confounded with the EMCN critic architecture; no ablation isolates the RPE prioritisation rule from the auxiliary reward and next-state losses, so the central attribution to prioritisation is unsupported.","rationale":"The reader's weakest assumption is that baselines may be under-tuned and the loss weights undisclosed; that is a valid comparison-fairness concern. My concern is more direct: RPE-PER changes both the sampling rule and the critic architecture, so the experiments cannot attribute the observed improvement to the RPE priority signal even if every baseline were perfectly tuned. The manuscript's admission that some baselines underperformed relative to their published results (Sec. 4.2) and the duplicate PER/LAP table entries are supporting signs, but the missing architecture control is load-bearing because the central claim is causal. I do not recommend rejecting the paper: the confusion is addressable, the algorithm is concrete, and the authors promise a public implementation. With the missing ablation and hyperparameters supplied, the claim could become tenable; without them, the current evidence is compatible with a null effect of RPE prioritisation. This is consistent with the reader's conditional verdict, so the final label remains CONDITIONAL.","tokens_in":9683,"tokens_out":9912,"duration_ms":101979,"concrete_test":"Run a 2x2 ablation on HalfCheetah and Humanoid with TD3 and 10 seeds per cell: (1) standard critic + random sampling; (2) EMCN with the same xi_1, xi_2, xi_3 as RPE-PER + random sampling; (3) standard critic + RPE priority from a separately trained reward-prediction head; (4) EMCN + RPE priority (RPE-PER). Compare mean returns over the last 10 evaluations and 95% CIs. If cell (2) matches cell (4), or cell (3) does not beat cell (1), the claimed benefit of RPE prioritisation is not supported and the paper should be re-scoped as an EMCN auxiliary-loss contribution.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The abstract claims that prioritising by reward prediction error improves learning speed and final performance. The only empirical comparison, however, varies two factors simultaneously. RPE-PER replaces the standard critic with EMCN, whose total loss (Eq. 11) includes auxiliary terms xi_2 L_R and xi_3 L_T for reward and next-state prediction, and it replaces the sampling score with RPE. Against random sampling, PER, LAP, and LA3P the critic architecture also changes; against MaPER the model-augmented critic is MaCN, not EMCN. A faster learning curve could therefore come entirely from the auxiliary losses providing richer gradients under uniform sampling, with RPE prioritisation contributing nothing. The paper itself flags (Sec. 4.2) that 'some baselines underperformed compared to their published results,' and Table 2's identical HalfCheetah entries for PER and LAP (7558.46 +/- 1010.69) weaken confidence in the table; but the missing control is the more central defect. Without a 2x2 ablation crossing {standard critic, EMCN} with {random sampling, RPE priority}, the reported results cannot establish the causal statement in the abstract.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces RPE-PER, an experience-replay prioritisation scheme for off-policy actor-critic reinforcement learning. RPE-PER replaces the standard critic with an 'Enhanced Model Critic Network' (EMCN) that outputs Q-values, a predicted reward, and a predicted next state; the absolute difference between the predicted reward and the stored reward defines the priority used in a PER-style sampling distribution (Equations 11-13). The method is evaluated on six MuJoCo continuous-control tasks under TD3 and SAC, comparing against random sampling, PER, LAP, LA3P, and MaPER, with ten seeds and reporting final-10-performance means with 95% confidence intervals (Tables 1-2, Figures 3-4). The authors claim consistent improvements in learning speed and final performance, while also acknowledging exceptions for Swimmer under TD3 and Walker2d/Hopper under SAC.","tokens_in":9920,"tokens_out":4960,"duration_ms":46937,"significance":"The contribution is a simple, biologically motivated prioritisation signal that could be useful for continuous-control replay, and the paper includes several good practices: ten seeds, reported confidence intervals, learning curves, and a public GitHub repository. If the reported gains were robust and attributable specifically to the RPE priority, this would be a useful addition to the experience-replay literature. However, the current evidence is not yet convincing: the experimental design conflates the prioritisation rule with the auxiliary-loss critic architecture, Tables 1-2 contain apparent data anomalies, and the headline claim of consistency is stronger than the overlapping confidence intervals support. The paper's value will depend on fixing these issues and adding the missing control experiments.","major_comments":[{"comment":"RPE-PER differs from every baseline in two ways: it uses the EMCN critic whose total loss includes the auxiliary reward and next-state prediction terms (Equation 11), and it replaces the sampling score with RPE. The current comparison cannot separate these factors; the reported gains could come entirely from the auxiliary losses under uniform sampling, with RPE prioritisation contributing nothing. The authors should add a 2x2 ablation crossing {standard critic, EMCN} with {random sampling, RPE priority} on at least a subset of tasks, or otherwise demonstrate that EMCN plus random sampling does not already match RPE-PER.","section":"Section 4, Tables 1 and 2"},{"comment":"In Table 2, the HalfCheetah rows for PER and LAP are identical (7558.46 +/- 1010.69), which is effectively impossible for two different algorithms across ten seeds unless a copy-paste error occurred. Additionally, the Random-Sampling Humanoid entry in Table 2 (5177.45 +/- 585.14) matches the TD3 random-sampling value in Table 1 to the second decimal, and MaPER's Hopper CI (+/- 7.42) is implausibly small relative to the other entries. These anomalies must be resolved or the corresponding conclusions cannot be evaluated.","section":"Table 2"},{"comment":"The statement that RPE-PER 'consistently outperforms most baseline methods' (TD3, plots a-e) is not supported by Table 1: in every TD3 environment the RPE-PER 95% CI overlaps with that of random sampling, and in Swimmer PER's mean is higher (89.64 vs 86.61). The authors should either report paired statistical tests (e.g., bootstrap or Wilcoxon across seeds) that account for the run pairing, or soften the 'consistent outperformance' claim to what the data actually show.","section":"Section 5.1 and Table 1"},{"comment":"The three loss weights xi_1, xi_2, xi_3 in Equation (11) are never reported, and the statement in Section 3.2 that 'increasing xi_2 improves performance' is made without supporting evidence or sensitivity analysis. Since these weights control how much of the EMCN gradient comes from reward prediction, they are essential for reproducibility and for interpreting whether the RPE signal is actually the driver of prioritisation. Please report the values used in all experiments and include a sensitivity study or at least justify a single choice.","section":"Sections 3.2 and 3.3"},{"comment":"The text acknowledges that 'some baselines underperformed compared to their published results' but provides no details on how PER, LAP, LA3P, and MaPER were tuned for these tasks. Because the central claim is comparative, the authors should supply the baseline hyperparameters, the source of each baseline implementation, and evidence that they were not disadvantaged relative to RPE-PER.","section":"Section 4.2"}],"minor_comments":[{"comment":"Equation (12) defines RPE_i as an MSE quantity, but Equation (13) then writes p_i = |RPE_i| + epsilon; since RPE is already non-negative, the absolute value is redundant. Algorithm 1 line 12 sets p_i = (RPE_i)^alpha, which is inconsistent with Equation (13)'s additive epsilon. Please align the notation.","section":"Equations (12)-(13) and Algorithm 1"},{"comment":"The abstract uses 'Reward Predictive Error' while the rest of the paper and the method name use 'Reward Prediction Error'; please make the terminology consistent.","section":"Abstract"},{"comment":"The figures use the label 'MAPER' whereas the text and tables use 'MaPER'; please standardise the label.","section":"Figures 3 and 4"},{"comment":"The pseudo-code computes RPE at line 5 using R_theta(s_t, a_t) before the buffer is updated; it would clarify whether the priority is intended to be based on the predicted reward from the current critic or from the target network.","section":"Algorithm 1"},{"comment":"The text says the shaded regions indicate 'standard deviations calculated from ten evaluations across ten runs' while the tables report 95% confidence intervals; please clarify which quantity is shown in the figures.","section":"Section 4.2"},{"comment":"The reference [Saglam et al., 2022] is cited as LA3P but the linked arXiv title appears to be 'Actor Prioritized Experience Replay'; please verify the citation and the naming.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The duplicate entries in Table 2 and the near-identical random-sampling values across Tables 1 and 2 suggest a data-handling problem that should be checked before external review advances. The missing ablation is the main scientific gap; if the authors cannot provide the 2x2 control, the paper's central claim should be substantially revised. Given the journal's standards, I recommend major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: RPE-PER is a reasonable incremental idea. It replaces TD-error priority with the absolute reward-prediction error from a critic that also predicts rewards and next states. Compared to MaPER, the change is small and clearly stated: EMCN uses actual rewards in the TD target rather than estimated rewards, and the priority score uses only the reward error. The pseudo-code is clean, and the paper positions itself honestly against MaPER rather than claiming a new paradigm. The empirical setup is standard: six MuJoCo tasks, ten seeds, 95% CIs, and a GitHub link.\n\nThe problem is that the experiments cannot isolate the claimed mechanism. RPE-PER differs from every baseline in two ways at once: the critic architecture (EMCN with auxiliary reward and next-state losses) and the sampling priority (RPE instead of TD error or model error). Against random sampling, PER, LAP, and LA3P, the critic changes; against MaPER, both the score definition and the critic change. Without a 2x2 ablation crossing {standard critic, EMCN} with {random, RPE priority}, the faster learning could come entirely from the auxiliary losses, with prioritisation doing nothing. The paper does not include that ablation.\n\nOther soft spots are in proportion. The abstract's 'consistently outperforms' is too strong: in TD3 every 95% CI overlaps random sampling, and PER beats RPE-PER on Swimmer; in SAC it loses on Walker2d and Hopper. Table 2 has identical PER and LAP scores for HalfCheetah (7558.46 ± 1010.69), which looks like a data handling error. The loss weights xi_1, xi_2, xi_3 are never reported, and Section 4.2 admits some baselines underperformed their published results. Without the weights and with under-tuned baselines, the comparison could be unfair.\n\nNone of this kills the idea. The priority rule is plausible, cheap, and connected to a real biological analogy. But the current evidence supports only 'EMCN with auxiliary losses and RPE sampling works on some tasks,' not 'the RPE signal causes the improvement.'\n\nWho should read it: people working on replay prioritisation in continuous control, especially those building on MaPER. It is not a foundational contribution.\n\nFor review: send it out, but expect major revision. Ask for the 2x2 ablation, the missing hyperparameters, a corrected table, and either significance tests or a claim scaled down to what the CIs support.","headline":"Simple, plausible replay-prioritisation idea, but the current experiments confound the priority signal with the new critic architecture, so 'consistently outperforms' is not yet established.","tokens_in":10478,"tokens_out":2177,"would_cite":false,"duration_ms":23578,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Prioritising replay samples by the gap between predicted and observed reward speeds and improves TD3 and SAC on continuous-control tasks.","keywords":["experience replay","reward prediction error","prioritised experience replay","continuous control","off-policy reinforcement learning","actor-critic methods","MuJoCo"],"falsifier":"Run the same six MuJoCo tasks with every baseline given an equal per-task hyperparameter search budget, including a search over the EMCN loss weights $\\xi_2$ and $\\xi_3$, and compare final returns; if RPE-PER no longer beats random sampling and PER on a majority of tasks, the claimed advantage of RPE prioritisation is unsupported.","tokens_in":9463,"feed_emoji":"🧠","tokens_out":10531,"duration_ms":97224,"temperature":0.7,"pith_summary":"RPE-PER is a replay-buffer sampling rule for off-policy actor-critic reinforcement learning. The paper claims that, instead of prioritising stored transitions by temporal-difference error as PER does, an agent should prioritise by reward prediction error: the absolute difference between the reward a critic network predicts for a state-action pair and the reward actually observed. The paper introduces the Enhanced Model Critic Network (EMCN), which outputs Q-values, predicted rewards, and predicted next states, and uses the reward prediction error from this network as the sampling priority. In experiments across six MuJoCo continuous-control tasks, the paper reports that RPE-PER reaches higher final performance than random sampling, PER, LAP, LA3P, and MaPER for TD3 in most tasks and for SAC in four of the six tasks. The practical interest is that the priority signal uses only the rewards already stored in the buffer, so the method is simple to add to existing actor-critic implementations.","feed_headline":"Reward prediction gap beats TD error for replay priority","feed_subtitle":"Sampling by predicted-vs-actual reward gap speeds TD3/SAC on six MuJoCo tasks.","key_machinery":"The central object is the EMCN critic network together with the RPE priority it feeds. EMCN is written as $C_\\theta(s,a) = (Q_\\theta(s,a), R_\\theta(s,a), T_\\theta(s,a))$: for a state-action pair it outputs the action value, the reward it expects, and the next state it expects, trained with the combined loss $\\mathcal{L}_{C_\\theta} = \\xi_1 L_{Q_\\theta} + \\xi_2 L_{R_\\theta} + \\xi_3 L_{T_\\theta}$. The priority for transition $i$ replaces PER's TD-error term with $\\mathrm{RPE}_i = |R_\\theta(s_i,a_i) - r_i|$, giving sampling probability proportional to $p_i^\\alpha$ with $p_i = |\\mathrm{RPE}_i| + \\epsilon$. What carries the argument is that the reward head is trained on actual rewards, not bootstrapped value estimates, so the prioritisation signal reflects how well the agent currently predicts the environment's reward structure.","core_discovery":"The central claim is that reward prediction error, computed as $|R_\\theta(s,a)-r|$ rather than TD error, is a more informative prioritisation signal for experience replay in continuous control. The reasoning is that transitions whose rewards the agent mispredicts carry learning value, and resampling them more often lets the actor-critic update correct its model of the environment faster. The EMCN is the load-bearing component: a critic that predicts Q-values, rewards, and next states under the weighted loss $\\xi_1 L_Q + \\xi_2 L_R + \\xi_3 L_T$, whose reward head supplies the RPE used in the priority $p_i = |\\mathrm{RPE}_i| + \\epsilon$. The reported final-ten-evaluation means favour RPE-PER over random sampling, PER, LAP, LA3P, and MaPER in the majority of the six MuJoCo environments under both TD3 and SAC, with the clearest gains under TD3 and in the more complex tasks.","pith_inferences":["A direct test of the paper's proposed signed-RPE extension would be to prioritise the critic with absolute RPE and the actor with signed RPE, which should reveal whether RPE-PER's weakness in balance tasks comes from over-prioritising noisy transitions.","Because EMCN also predicts next states, a natural but untested extension is to use its predicted transitions as synthetic rollouts, turning RPE-PER into a hybrid model-based replay method.","The Swimmer result implies a testable complexity threshold: the advantage of RPE-PER should grow with state-action dimensionality and reward stochasticity, which could be checked on a graded family of continuous-control environments."],"forward_implications":["Under TD3, RPE-PER reports higher final-ten-evaluation returns than random, PER, LAP, LA3P, and MaPER on Humanoid, HalfCheetah, Ant, Walker2d, and Hopper.","Under SAC, RPE-PER reports higher final returns than the baselines on Humanoid, HalfCheetah, Ant, and Swimmer, though the gains are more modest than under TD3.","The Swimmer result, where simpler PER outperforms RPE-PER, suggests that the RPE prioritisation advantage is concentrated in higher-dimensional or more complex tasks.","In Walker2d and Hopper under SAC, LA3P performs comparably or better, indicating that separating actor and critic priorities by TD error remains competitive where precise balance matters."],"supporting_citations":[{"why":"Introduces prioritised experience replay and the TD-error priority that RPE-PER replaces.","marker":"[Schaul et al., 2015]"},{"why":"Introduces MaCN/MaPER, the model-augmented critic and replay method that EMCN extends and the main comparison.","marker":"[Oh et al., 2021]"},{"why":"Defines the LAP baseline that samples by loss rather than TD error.","marker":"[Fujimoto et al., 2020]"},{"why":"Defines the LA3P baseline that splits actor and critic priorities.","marker":"[Saglam et al., 2022]"},{"why":"Provides the TD3 algorithm used as one of the two base actor-critic learners.","marker":"[Fujimoto et al., 2018]"},{"why":"Provides the SAC algorithm used as the other base learner.","marker":"[Haarnoja et al., 2018]"},{"why":"Supplies the MuJoCo continuous-control benchmark tasks used in all evaluations.","marker":"[Todorov et al., 2012]"},{"why":"Supplies the random-sampling replay baseline and the general experience-replay setup.","marker":"[Mnih et al., 2015]"}],"fun_headline_variants":["RPE-PER: Replay priority from reward prediction gaps","Reward prediction error boosts replay for actor-critic","Prioritize replay by reward surprise, not TD error","RPE-PER speeds TD3/SAC with reward-prediction sampling","Better replay: sample by reward prediction error"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the baseline replay methods were implemented and tuned to their best possible performance; Section 4.2 concedes that some baselines underperformed their published results, and the three EMCN loss weights ($\\xi_1,\\xi_2,\\xi_3$) are never reported, so uneven tuning could make RPE-PER's gains an artefact of the comparison rather than of the prioritisation principle.","fun_headline_variants_meta":{"raw":{"variants":["RPE-PER: Replay priority from reward prediction gaps","Reward prediction error boosts replay for actor-critic","Prioritize replay by reward surprise, not TD error","RPE-PER speeds TD3/SAC with reward-prediction sampling","Better replay: sample by reward prediction error"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000597,"raw_usage":{"total_tokens":2791,"prompt_tokens":944,"completion_tokens":1847,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":1779}},"tokens_in":560,"tokens_out":1847,"duration_ms":12824,"temperature":1.0,"reasoning_tokens":1779,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T00:40:54.719692+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same six MuJoCo tasks with every baseline given an equal per-task hyperparameter search budget, including a search over the EMCN loss weights $\\xi_2$ and $\\xi_3$, and compare final returns; if RPE-PER no longer beats random sampling and PER on a majority of tasks, the claimed advantage of RPE prioritisation is unsupported.","supporting_citations":[{"cited_title":"An equivalence between loss functions and non-uniform sampling in experience replay","cited_arxiv_id":null,"evidence_quote":"Defines the LAP baseline that samples by loss rather than TD error."},{"cited_title":"Actor Prioritized Experience Replay","cited_arxiv_id":"2209.00532","evidence_quote":"Defines the LA3P baseline that splits actor and critic priorities."},{"cited_title":"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor","cited_arxiv_id":null,"evidence_quote":"Provides the SAC algorithm used as the other base learner."},{"cited_title":"Mujoco: A physics engine for model-based control","cited_arxiv_id":null,"evidence_quote":"Supplies the MuJoCo continuous-control benchmark tasks used in all evaluations."}],"review_version":1}