Pith. sign in

REVIEW 3 major objections 9 minor 169 references

A deep RL agent can match a million-transition replay buffer with 10–50× less storage by keeping only chained n-step endpoints and updating them with expectile Sarsa.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Chained n-step endpoint coresets plus expectile Sarsa match million-transition DQN buffers at 10–50× less storage by keeping bootstrap targets anchored.

T0 review reviewed 2026-07-31 challenge →

load-bearing objection Clean failure-mode diagnosis plus a simple two-buffer fix that actually matches large-buffer Atari/Pinball at 10–50× less storage; theory is honest about its deterministic limit. the 3 major comments →

arxiv 2607.25123 v1 pith:N4A3H3FD submitted 2026-07-27 cs.LG

Endpoint Replay: Compressing the Recency Buffer in Deep Reinforcement Learning

Parham Mohammad Panahi , Armin Ashrafi , Haoyu Du , Andrew Patterson , Martha White , Adam White This is my paper
classification cs.LG
keywords Deep Reinforcement LearningExperience ReplayCoresetsMultistep methodsExpectile SarsaBuffer compressionUnanchored bootstrap targets
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Standard deep RL keeps a huge first-in-first-out buffer of recent transitions—often a million of them—and samples it uniformly. This paper asks whether that buffer can be shrunk by an order of magnitude without losing the performance those large buffers buy. The answer it offers is Endpoint Replay: keep a small recency buffer for fresh one-step updates, and compress older experience into a coreset of connected n-step jumps so every bootstrap target stays inside the coreset. Isolated “representative” transitions fail because their next states are never themselves updated, producing unanchored and eventually corrupted targets; chaining the endpoints fixes that. An expectile loss then offsets the pessimistic bias that comes from multi-step returns collected under older, weaker policies. Empirically the method matches large-buffer scores in Pinball and twelve Atari games while beating equal-sized buffers and other compression baselines.

Core claim

Naive coresets of isolated transitions create unanchored bootstrap targets—states and actions that are never updated—so value estimates drift and control performance collapses. Storing instead the endpoints of chained n-step sequences keeps every bootstrap target inside the buffer, and pairing those updates with an n-step expectile Sarsa loss removes the pessimistic bias of older multi-step returns. With that construction, a buffer one-tenth to one-fiftieth the usual size matches a standard million-transition recency buffer.

What carries the argument

Endpoint Replay: a small recency buffer plus a coreset of chained n-step transitions updated by n-step Expectile Sarsa. The chain anchors every bootstrap state and action; the expectile loss (τ > 0.5) counteracts multi-step pessimism from older policies.

Load-bearing premise

The proof that policy iteration reaches the optimal policy holds only when the environment is deterministic; with high reward noise the expectile can chase lucky outcomes instead of better actions.

What would settle it

In a controlled control setting, replace a large buffer with an equal-sized coreset of isolated one-step transitions versus chained n-step endpoints; if bootstrap-target error keeps rising and return collapses only for the isolated coreset, the unanchored-target diagnosis is confirmed.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Production DQN-style agents can cut replay memory by 10–50× while retaining large-buffer learning curves.
  • Any coreset or prototype selection that stores isolated transitions is at risk of the same unanchored-bootstrap failure.
  • n-step expectile operators are sound Bellman operators and induce an ordered family of value functions indexed by n.
  • Under deterministic dynamics, greedifying on n-step expectile values converges to the ordinary optimal policy.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same anchoring idea could be applied inside prioritized or generative replay, not only uniform FIFO coresets.
  • Decaying τ toward 0.5 over training would give a practical way to retain early multi-step optimism while recovering mean returns in noisier domains.
  • If unanchored targets are the dominant failure mode, distance-based prototype selection without chaining should systematically underperform Endpoint Replay even when coverage looks good.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 9 minor

Summary. The paper asks whether the standard 1M-transition FIFO replay buffer in DQN-style algorithms can be compressed 10–50× without performance loss. It identifies a failure mode of naive coreset compression: isolated transitions yield "unanchored" bootstrap targets whose state-action values are never directly updated, and demonstrates this in controlled prediction (Fig. 1) and control (Fig. 2) experiments. The proposed fix, Endpoint Replay, keeps a small recency buffer plus a coreset of chained n-step transitions (so every bootstrap endpoint is itself updated), trained with an n-step Expectile Sarsa update (τ=0.7) to counteract the pessimistic bias of n-step targets from stale policies. Theory: the n-step expectile Bellman operator is a γⁿ-contraction (Thm 1), n-step expectile values are ordered in n (Thm 2), policy iteration converges under a monotonic-greedification assumption (Thm 3), and to the optimal policy under deterministic dynamics (Cor. 1); Prop. 1 shows expectile greedification can strictly reduce expected return. Empirically, Endpoint Replay matches a 10–50× larger recency buffer in Pinball (100 seeds) and 12 Atari games (10 seeds, sign tests), beating equal-sized buffers, reservoir coresets, and MeDQN; ablations attribute gains to state anchoring and the expectile loss.

Significance. If the results hold, this is a practically useful contribution: an order-of-magnitude reduction in replay memory with maintained performance matters for deployment and for scaling studies, and the "unanchored bootstrap target" diagnosis is a clean, previously unrecognized conceptual point backed by direct error measurements rather than only return curves. Strengths worth naming: publicly released code; unusually careful experimental hygiene for the area (100-seed Pinball runs with bootstrap CIs, two-stage tuning for the MeDQN baseline, sign tests, per-game breakdowns); controlled experiments that measure bootstrap-target error directly; and honest theory, including a counterexample (Prop. 1) against the authors' own update. The n-step expectile operator analysis (contraction for n≥1, ordering in n) is, to my knowledge, new. The main risks are a prose/pseudocode inconsistency about what the coreset actually contains, an untested robustness boundary for the expectile under stochastic dynamics, and a gap in the policy-iteration convergence proof.

major comments (3)
  1. [§3.3 vs. Algorithm 1 (App. 8), lines 11–17] The prose describes a stride-n chain: the lag buffer fills with n transitions, one n-step summary is emitted, and 'the lag buffer would simply reset' (and resets after k steps on termination), giving D_c = {(s_0,...s_n),(s_n,...s_2n),...} as in §3.1. The pseudocode instead pops only the FIRST transition from D_lag after emitting a summary and never resets: this implements a stride-1 sliding window in which every evicted transition generates an overlapping n-step summary. The two versions differ materially. Under stride-1, a 90k coreset covers ~90k environment steps of history, not ~900k, so the abstract's claim of 'an effective memory horizon comparable to a standard large buffer' no longer follows; anchoring also becomes trivial (every state is a start state) rather than a consequence of chaining. Additionally, line 14 computes g with constant γ^i and ignores the stored per-transition (
  2. [§3.2 and §5 (Table 3, Figs. 4c/d, 7, 8)] The mechanism attribution for the expectile is underdetermined by the evidence. The paper itself states the failure mode: with stochastic dynamics the expectile 'can chase environment stochasticity rather than ... actions that result in higher expected return' (§3.2), Corollary 1 gives optimality only under deterministic dynamics, and Proposition 1 constructs a case where expectile greedification strictly lowers expected return. Yet the headline benchmark deliberately injects stochasticity: sticky actions with repeat probability 0.25 in all 12 games (Table 3), and Ms. Pac-Man selected for 'high variance induced by unpredictable ghost movements'. The No-Expectile ablation is run only at this fixed noise level, so it cannot distinguish 'expectile corrects stale-policy pessimism' from 'expectile is generic optimism that happens to help here'. τ=0.7 is used everywhere with no reported sweep,
  3. [App. 7.3, Theorem 3 (and Corollary 1)] The proof of Theorem 3 establishes monotone, bounded improvement of V^{(n)}_{τ,π_t}, hence convergence to a fixed point of the greedification operator — but the conclusion that the limit is the OPTIMAL expectile policy ('will stop at some π*_τ that has the largest V') does not follow. In classical policy iteration the fixed-point argument gives V = max_{π'} T_{π'}V because improvement maximizes over policies; here greedification produces a single π′ satisfying Definition 2, so a fixed point only satisfies T_{π'}V_π = V_π for that particular π′, which need not be globally optimal without an additional argument that every suboptimal policy admits a strictly improving monotonic expectile greedy successor. Relatedly, convergence of π_t itself (rather than of the values) is asserted, and Assumption 1 is acknowledged to be unverified for the ε-greedy policy the algorithm actually uses — so the
minor comments (9)
  1. [App. 14, Figure 10] Caption says '100k buffer setting' but the legend and content are the 20k (50×) setting; also the caption says 'ablations' while showing baselines.
  2. [§3.1, Figure 1] Caption states 'Average over 100 seeds' while the text describes 10 datasets; the light lines are described as 10 random seeds. Please reconcile.
  3. [§5 / App. 12] Min–max normalization mapping the worst seed to 0 and best seed to 1 per game is sensitive to single-seed outliers and discards effect magnitude; consider reporting IQM with stratified bootstrap CIs (rliable-style) alongside, at least in the supplement.
  4. [App. 13] The sign test treats 120 game–seed pairs as independent; pairs within a game share game-level effects, which can inflate significance. Aggregating per game first (12 pairs) or a hierarchical test would be more conservative.
  5. [§5] MeDQN is excluded from Atari because it failed in Pinball, but its Pinball failure is attributed to invalid random states in a constrained domain — a domain-specific reason. The contribution claim 'outperforms MeDQN' therefore rests on a single, unfavorable setting; either soften the claim or include one Atari data point.
  6. [Abstract / Fig. 3d] The abstract says the method 'matches the performance of traditional large buffers'; in the 50× setting it does not quite reach the large baseline (5/12 games at parity). Qualify the claim by compression level.
  7. [§3.3 / §5] The 7:1 minibatch mix and n=10 are described as 'found effective' with no sensitivity results; even a coarse sweep in Pinball would help practitioners.
  8. [App. 7.2, Theorem 2] The ordering V^{(n)} ≥ V^{(a·n)} is stated for τ ≥ 1/2; a one-line remark on whether the ordering reverses for τ < 1/2 would complete the picture.
  9. [Typos] §5 'We preformed'; §4 'boostrap'; App. 7.1 'analagously'; §3.1 'a nearby states t'; Broader Impact 'bare little resemblance' (→ 'bear'). §2 'from early learn' (→ 'early learning').

Circularity Check

0 steps flagged

No significant circularity: theory is self-contained contraction/policy-iteration analysis; empirical claims are scored on external environment returns against independent baselines.

full rationale

The paper's load-bearing claims do not reduce to their inputs by construction. The n-step expectile action-values are introduced as a fixed-point definition Q^{(n)}_{τ,π}=e_τ(G^{(n)}(Q^{(n)}_{τ,π})), which is the standard Bellman-style recursion; existence and uniqueness follow from an explicit γ^n-contraction proof (Theorem 1) that does not bake in target performance. Ordering (Theorem 2) and policy-iteration convergence (Theorem 3, Corollary 1) are derived from monotonicity/subadditivity of expectiles under stated assumptions (including deterministic dynamics for optimality). None of these steps fit a parameter to data and rename the fit a prediction, import a uniqueness theorem from overlapping authors, or smuggle an ansatz via self-citation. Empirically, Endpoint Replay is evaluated on held-out Pinball and Atari returns against large-buffer DDQN, equal-sized recency buffers, reservoir coresets, and MeDQN; ablations isolate anchoring and the expectile loss on the same external metrics. Self-citations (e.g., Panahi et al. 2024, Patterson et al. 2024, Lo et al. 2024) supply environment or methodology context and are not load-bearing for the compression or soundness claims. The known limitation that optimality holds only under deterministic dynamics is a correctness/scope issue, not circularity. Score 0 is therefore appropriate.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 3 invented entities

The central empirical claim rests on the standard MDP/DQN stack plus several design knobs (n, τ, buffer split, mix ratio) chosen by the authors, and on a theory side-condition (deterministic dynamics for optimality; monotonic expectile greedification for PI) that is stronger than what ε-greedy Atari agents satisfy. No new physical entities; the main invented conceptual objects are the unanchored-target diagnosis and the n-step expectile action-value fixed point.

free parameters (5)
  • expectile level τ = 0.7
    Fixed to 0.7 for all Endpoint experiments; controls how aggressively n-step targets chase upper returns and is not derived.
  • chain length n = 10
    Subsampling stride and n-step horizon; set to 10 throughout main experiments without a selection principle beyond common multi-step practice.
  • recency/coreset split and minibatch mix = 90-10 capacities; 28:4 batch mix
    Default 90–10 capacity split and 28:4 (7:1) recency:coreset samples per batch of 32; described as found effective, not theoretically fixed.
  • Pinball learning rate = 0.002
    Tuned on the large-buffer baseline over a five-point grid and then frozen for all methods.
  • MeDDQN regularization weight λ = 2
    Two-stage sweep {1,2,4} then best λ=2 reported; baseline-only but affects comparative claims.
axioms (5)
  • domain assumption Environment is an MDP with bounded rewards; episodic γ set to 0 at termination (White 2017).
    Standard RL problem formulation in §2; required for Bellman operators and returns.
  • standard math n-step expectile Bellman operator is a γ^n-contraction in sup norm, hence has a unique fixed point (Theorem 1).
    Proved from monotonicity and translation invariance of expectiles; load-bearing for soundness of the update.
  • ad hoc to paper Policy class is compact and closed under a monotonic expectile greedification operator (Assumption 1 / Definition 2).
    Needed for PI convergence (Theorem 3); paper notes standard softmax/ε-greedy are not shown to satisfy it.
  • domain assumption Deterministic transitions and rewards for Corollary 1 (expectile PI recovers the optimal expected-return policy).
    Stated explicitly; without it expectiles may chase environment noise. Atari sticky actions violate strict determinism.
  • domain assumption Double DQN / Adam / Dopamine-style Atari protocol is a valid performance yardstick for buffer compression.
    All comparisons are relative to this stack; claim is not architecture-agnostic.
invented entities (3)
  • Unanchored bootstrap targets independent evidence
    purpose: Name the failure mode of isolated coreset transitions whose next state-action pairs are never updated.
    Diagnostic concept introduced in §3.1 and probed in Figs. 1–2; parallels offline OOD actions but applied to online coresets.
  • n-step expectile action-values Q^{(n)}_{τ,π} no independent evidence
    purpose: Fixed points of the n-step expectile Bellman operator used by Expectile Sarsa.
    Defined in Eqs. 1–4; existence via Theorem 1. Independent mathematical object, but optimality link needs determinism.
  • Endpoint Replay (recency + lag + chained n-step coreset) independent evidence
    purpose: Concrete buffer-compression algorithm pairing anchored n-step coresets with expectile Sarsa and one-step DDQN on fresh data.
    Algorithm 1–2; evaluated empirically. No claim of a new physical quantity—purely algorithmic.

reviewed 2026-07-31 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Endpoint Replay: Compressing the Recency Buffer in Deep Reinforcement Learning." pith.science (2026). https://pith.science/paper/N4A3H3FD

@misc{pith2026260725123,
  author       = {Pith},
  title        = {Pith review of: Endpoint Replay: Compressing the Recency Buffer in Deep Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N4A3H3FD}},
  note         = {Machine review of arXiv:2607.25123}
}
Share X Bluesky LinkedIn Reddit HN
abstract

Experience replay remains one of the most practical and useful algorithmic tools in the deep reinforcement learning (DRL) toolbox. Aside from the limited success of prioritized replay and specialized approaches for large asynchronous systems, most DRL algorithms make use of a large, uniformly sampled recency buffer---even the size, one million, remains unchanged. Could we store less data, reduce redundancy, or more effectively chain experience together to speed up value propagation and still retain the performance of large buffers? In this paper, we investigate a simple compression approach that stores representative transitions derived from the end-points of a chain of connected $n$-step sequences. By curating these end-points in a smaller recency buffer, our method maintains an effective memory horizon comparable to a standard large buffer while requiring an order of magnitude less storage. Through empirical evaluation, we demonstrate that this approach prevents the systematic bias inherent in naive compression strategies and matches the performance of traditional large buffers in the Pinball environment and the Atari 2600 benchmark.

Figures

Figures reproduced from arXiv: 2607.25123 by Adam White, Andrew Patterson, Armin Ashrafi, Haoyu Du, Martha White, Parham Mohammad Panahi.

Figure 1
Figure 1. Figure 1: Mean square error between the boot￾strapped target Q(st+10, at+10) and sample return gt+10 of anchored, unanchored, recency and su￾pervised learners. Average over 100 seeds. Indi￾vidual runs of 10 random seeds are shown in light thin lines to improve visibility. For this experiment, we also store the Monte Carlo returns Gt+10 from the bootstrap state and action for each transition in the coreset, to comput… view at source ↗
Figure 2
Figure 2. Figure 2: Endpoint replay can maintain better performance (left) and has lower bootstrap target value [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Endpoint replay performs comparably to a much larger recency buffer while outperforms [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Ablation study of Endpoint replay in Pinball and Atari. In Pinball we report mean perfor [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Performance of Endpoint replay and other baselines in individual Atari games in the 100k [PITH_FULL_IMAGE:figures/full_fig_p027_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Performance of Endpoint replay and other baselines in individual Atari games in the 20k [PITH_FULL_IMAGE:figures/full_fig_p028_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Performance of Endpoint replay and its ablations in individual Atari games in the 100k [PITH_FULL_IMAGE:figures/full_fig_p029_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Performance of Endpoint replay and its ablations in individual Atari games in the 20k [PITH_FULL_IMAGE:figures/full_fig_p030_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Performance of Endpoint replay and other baselines in individual Atari games in the 100k [PITH_FULL_IMAGE:figures/full_fig_p031_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Performance of Endpoint replay and its ablations in individual Atari games in the 100k [PITH_FULL_IMAGE:figures/full_fig_p032_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Performance of Endpoint replay and its ablations in individual Atari games in the 100k [PITH_FULL_IMAGE:figures/full_fig_p033_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Performance of Endpoint replay and its ablations in individual Atari games in the 20k [PITH_FULL_IMAGE:figures/full_fig_p034_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

169 extracted references · 10 linked inside Pith

  1. [2]

    Neural computation , volume=

    Risk-sensitive reinforcement learning , author=. Neural computation , volume=. 2014 , publisher=

    2014

  2. [3]

    Advances in Neural Information Processing Systems , volume=

    The nature of temporal difference errors in multi-step distributional reinforcement learning , author=. Advances in Neural Information Processing Systems , volume=

  3. [5]

    K-percent Evaluation for Lifelong RL , author=

  4. [6]

    International Conference on Learning Representations , year=

    Distributed Distributional Deterministic Policy Gradients , author=. International Conference on Learning Representations , year=

  5. [7]

    Advances in Neural Information Processing Systems , volume=

    State chrono representation for enhancing generalization in reinforcement learning , author=. Advances in Neural Information Processing Systems , volume=

  6. [8]

    arXiv preprint arXiv:2507.09087 , year=

    Deep reinforcement learning with gradient eligibility traces , author=. arXiv preprint arXiv:2507.09087 , year=

    arXiv

  7. [9]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Using hindsight to anchor past knowledge in continual learning , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  8. [10]

    ArXiv , year=

    The Laplacian in RL: Learning Representations with Efficient Approximations , author=. ArXiv , year=

  9. [11]

    International Conference on Machine Learning , year=

    Towards Sustainable Learning: Coresets for Data-efficient Deep Learning , author=. International Conference on Machine Learning , year=

  10. [12]

    IEEE Transactions on Signal Processing , year=

    Analyzing Sparse Dictionaries for Online Learning With Kernels , author=. IEEE Transactions on Signal Processing , year=

  11. [13]

    Journal of Machine Learning Research , volume=

    Empirical design in reinforcement learning , author=. Journal of Machine Learning Research , volume=

  12. [14]

    The Thirteenth International Conference on Learning Representations , year=

    Prioritized Generative Replay , author=. The Thirteenth International Conference on Learning Representations , year=

  13. [15]

    International Conference on Machine Learning , pages=

    Large Batch Experience Replay , author=. International Conference on Machine Learning , pages=. 2022 , organization=

    2022

  14. [16]

    Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence , pages=

    Efficient diversity-based experience replay for deep reinforcement learning , author=. Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence , pages=

  15. [17]

    IEEE Transactions on Signal Processing , year=

    The kernel recursive least-squares algorithm , author=. IEEE Transactions on Signal Processing , year=

  16. [18]

    AAAI Conference on Artificial Intelligence , year=

    Deep Reinforcement Learning with Double Q-Learning , author=. AAAI Conference on Artificial Intelligence , year=

  17. [19]

    2022 , journaltitle =

    A Generalized Projected Bellman Error for Off-Policy Value Estimation in Reinforcement Learning , author =. 2022 , journaltitle =

    2022

  18. [20]

    nature , volume=

    Human-level control through deep reinforcement learning , author=. nature , volume=. 2015 , publisher=

    2015

  19. [21]

    Proceedings of the Ninth National Conference on Artificial Intelligence - Volume 2 , pages =

    Lin, Long-Ji , title =. Proceedings of the Ninth National Conference on Artificial Intelligence - Volume 2 , pages =. 1991 , publisher =

    1991

  20. [23]

    and Barto, Andrew G

    Sutton, Richard S. and Barto, Andrew G. , publisher=. Reinforcement Learning:. 1998 , address=

    1998

  21. [24]

    R. S. Sutton and D. McAllester and S. Singh and Y. Mansour. Policy Gradient Methods for Reinforcement Learning with Function Approximation. Advances in Neural Information Processing Systems 12. 2000

    2000

  22. [25]

    R. J. Williams. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Machine Learning. 1992

    1992

  23. [26]

    International Conference on Learning Representations , year=

    Offline Reinforcement Learning with Implicit Q-Learning , author=. International Conference on Learning Representations , year=

  24. [27]

    , title =

    Indyk, Piotr and Mahabadi, Sepideh and Mahdian, Mohammad and Mirrokni, Vahab S. , title =. Proceedings of the 33rd ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems , pages =. 2014 , publisher =

    2014

  25. [28]

    Journal of Machine Learning Research (JMLR) , volume =

    Temporal Abstraction in Reinforcement Learning with the Successor Representation , author =. Journal of Machine Learning Research (JMLR) , volume =

  26. [29]

    International Conference on Machine Learning (ICML) , year =

    Deep Laplacian-based Options for Temporally-Extended Exploration , author =. International Conference on Machine Learning (ICML) , year =

  27. [30]

    International Conference on Learning Representations (ICLR) , year =

    Proper Laplacian Representation Learning , author =. International Conference on Learning Representations (ICLR) , year =

  28. [31]

    3rd International Conference on Learning Representations (ICLR) , year=

    Adam: A Method for Stochastic Optimization , author=. 3rd International Conference on Learning Representations (ICLR) , year=

  29. [32]

    Sutton , title =

    Shangtong Zhang and Richard S. Sutton , title =. Deep Reinforcement Learning Symposium, NIPS 2017 , volume =

    2017

  30. [33]

    The Reinforcement Learning Journal , volume=

    Investigating the Interplay of Prioritized Replay and Generalization , author=. The Reinforcement Learning Journal , volume=

  31. [34]

    Advances in Neural Information Processing Systems , volume=

    Synthetic experience replay , author=. Advances in Neural Information Processing Systems , volume=

  32. [35]

    International Conference on Machine Learning , pages=

    Adapting kernel representations online using submodular maximization , author=. International Conference on Machine Learning , pages=. 2017 , organization=

    2017

  33. [36]

    Advances in neural information processing systems , volume=

    Double Q-learning , author=. Advances in neural information processing systems , volume=

  34. [37]

    International Conference on Machine Learning , pages=

    Unifying task specification in reinforcement learning , author=. International Conference on Machine Learning , pages=. 2017 , organization=

    2017

  35. [38]

    arXiv preprint arXiv:1912.01603 , year=

    Dream to control: Learning behaviors by latent imagination , author=. arXiv preprint arXiv:1912.01603 , year=

    Pith/arXiv arXiv 1912

  36. [39]

    arXiv preprint arXiv:1903.00374 , year=

    Model-based reinforcement learning for atari , author=. arXiv preprint arXiv:1903.00374 , year=

    Pith/arXiv arXiv 1903

  37. [40]

    Medical Imaging with Deep Learning , pages =

    Selective experience replay compression using coresets for lifelong deep reinforcement learning in medical imaging , author =. Medical Imaging with Deep Learning , pages =. 2024 , volume =

    2024

  38. [41]

    Proceedings of the 40th International Conference on Machine Learning , pages =

    Prototype-Sample Relation Distillation: Towards Replay-Free Continual Learning , author =. Proceedings of the 40th International Conference on Machine Learning , pages =. 2023 , volume =

    2023

  39. [42]

    The big world hypothesis and its ramifications for artificial intelligence , author=

  40. [43]

    Artificial intelligence , volume=

    Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning , author=. Artificial intelligence , volume=. 1999 , publisher=

    1999

  41. [44]

    Artificial Intelligence , volume=

    Reward-respecting subtasks for model-based reinforcement learning , author=. Artificial Intelligence , volume=. 2023 , publisher=

    2023

  42. [45]

    Journal of Machine Learning Research , volume=

    Goal-space planning with subgoal models , author=. Journal of Machine Learning Research , volume=

  43. [46]

    Journal of Artificial Intelligence Research , volume=

    Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents , author=. Journal of Artificial Intelligence Research , volume=

  44. [47]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Deep reinforcement learning with double q-learning , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  45. [48]

    Machine learning , volume=

    Convergence results for single-step on-policy reinforcement-learning algorithms , author=. Machine learning , volume=. 2000 , publisher=

    2000

  46. [49]

    and Naddaf, Yavar and Veness, Joel and Bowling, Michael , journal =

    Bellemare, Marc G. and Naddaf, Yavar and Veness, Joel and Bowling, Michael , journal =. The

  47. [50]

    International Conference on Machine Learning , pages=

    Atari-5: Distilling the arcade learning environment down to five games , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  48. [51]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Rainbow: Combining improvements in deep reinforcement learning , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  49. [52]

    International Conference on Machine Learning , pages=

    Revisiting rainbow: Promoting more insightful and inclusive deep reinforcement learning research , author=. International Conference on Machine Learning , pages=. 2021 , organization=

    2021

  50. [53]

    Proceedings of the 27th International Joint Conference on Artificial Intelligence , pages=

    Organizing experience: a deeper look at replay mechanisms for sample-based planning in continuous state domains , author=. Proceedings of the 27th International Joint Conference on Artificial Intelligence , pages=

  51. [54]

    Proceedings of the 37th International Conference on Machine Learning , pages =

    Coresets for Data-efficient Training of Machine Learning Models , author =. Proceedings of the 37th International Conference on Machine Learning , pages =. 2020 , volume =

    2020

  52. [55]

    Experience Selection in Deep Reinforcement Learning for Control , journal =

    Tim de Bruin and Jens Kober and Karl Tuyls and Robert Babu. Experience Selection in Deep Reinforcement Learning for Control , journal =. 2018 , volume =

    2018

  53. [56]

    Fedorov, Valerii , year =

  54. [57]

    International conference on machine learning , pages=

    Revisiting fundamentals of experience replay , author=. International conference on machine learning , pages=. 2020 , organization=

    2020

  55. [58]

    International Conference on Learning Representations , year=

    Distributed Prioritized Experience Replay , author=. International Conference on Learning Representations , year=

  56. [59]

    Proceedings of the 40th International Conference on Machine Learning , pages =

    Towards Sustainable Learning: Coresets for Data-efficient Deep Learning , author =. Proceedings of the 40th International Conference on Machine Learning , pages =. 2023 , editor =

    2023

  57. [60]

    Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence,

    Organizing Experience: a Deeper Look at Replay Mechanisms for Sample-Based Planning in Continuous State Domains , author =. Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence,. 2018 , month =

    2018

  58. [61]

    No More Pesky Hyperparameters: Offline Hyperparameter Tuning for

    Han Wang and Archit Sakhadeo and Adam M White and James M Bell and Vincent Liu and Xutong Zhao and Puer Liu and Tadashi Kozuno and Alona Fyshe and Martha White , journal=. No More Pesky Hyperparameters: Offline Hyperparameter Tuning for

  59. [62]

    2013 , journal=

    Playing Atari with Deep Reinforcement Learning , author=. 2013 , journal=

    2013

  60. [63]

    International Conference on Learning Representations (ICLR) , year=

    Prioritized Experience Replay , author=. International Conference on Learning Representations (ICLR) , year=

  61. [64]

    Reinforcement Learning Conference , year =

    Investigating the Interplay of Prioritized Replay and Generalization , author=. Reinforcement Learning Conference , year =

  62. [65]

    Reachability-Aware

    Wang, Kaixin and Zhou, Kuangqi and Feng, Jiashi and Hooi, Bryan and Wang, Xinchao , booktitle =. Reachability-Aware. 2023 , volume =

    2023

  63. [66]

    Advances in neural information processing systems , volume=

    Safe and efficient off-policy reinforcement learning , author=. Advances in neural information processing systems , volume=

  64. [67]

    A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning , volume =

    Adkins, Jacob and Bowling, Michael and White, Adam , booktitle =. A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning , volume =

  65. [69]

    Medical Imaging with Deep Learning , pages=

    Selective experience replay compression using coresets for lifelong deep reinforcement learning in medical imaging , author=. Medical Imaging with Deep Learning , pages=. 2024 , organization=

    2024

  66. [70]

    SIAM Journal on Computing , volume=

    Bisimulation Metrics for Continuous Markov Decision Processes , author=. SIAM Journal on Computing , volume=. 2011 , publisher=

    2011

  67. [71]

    A Kernel Perspective on Behavioural Metrics for Markov Decision Processes , author=

  68. [72]

    Advances in Neural Information Processing Systems , volume=

    MICo: Improved representations via sampling-based state similarity for Markov decision processes , author=. Advances in Neural Information Processing Systems , volume=

  69. [73]

    International conference on machine learning , pages=

    Asynchronous methods for deep reinforcement learning , author=. International conference on machine learning , pages=. 2016 , organization=

    2016

  70. [74]

    International Conference on Machine Learning , pages=

    Trajectory-aware eligibility traces for off-policy reinforcement learning , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  71. [75]

    Machine learning , volume=

    Self-improving reactive agents based on reinforcement learning, planning and teaching , author=. Machine learning , volume=. 1992 , publisher=

    1992

  72. [76]

    arXiv preprint arXiv:2509.15032 , year=

    Sample Efficient Experience Replay in Non-stationary Environments , author=. arXiv preprint arXiv:2509.15032 , year=

    arXiv

  73. [77]

    Transactions on Machine Learning Research , volume=

    Uncertainty-Based Experience Replay for Task-Agnostic Continual Reinforcement Learning , author=. Transactions on Machine Learning Research , volume=

  74. [78]

    Advances in Neural Information Processing Systems , volume=

    Continual Learning with Deep Generative Replay , author=. Advances in Neural Information Processing Systems , volume=

  75. [81]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Selective experience replay for lifelong learning , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  76. [82]

    2023 IEEE/CVF International Conference on Computer Vision (ICCV) , pages=

    Prototype Reminiscence and Augmented Asymmetric Knowledge Aggregation for Non-Exemplar Class-Incremental Learning , author=. 2023 IEEE/CVF International Conference on Computer Vision (ICCV) , pages=. 2023 , organization=

    2023

  77. [83]

    Advances in Neural Information Processing Systems , volume=

    Experience Replay for Continual Learning , author=. Advances in Neural Information Processing Systems , volume=

  78. [84]

    Proceedings of the 28th International Joint Conference on Artificial Intelligence , pages=

    Experience replay optimization , author=. Proceedings of the 28th International Joint Conference on Artificial Intelligence , pages=

  79. [85]

    Frontiers in neurorobotics , volume=

    Map-based experience replay: a memory-efficient solution to catastrophic forgetting in reinforcement learning , author=. Frontiers in neurorobotics , volume=

  80. [86]

    Advances in Neural Information Processing Systems , volume=

    Sample-Efficient Deep Reinforcement Learning via Episodic Backward Update , author=. Advances in Neural Information Processing Systems , volume=

Showing first 80 references.

This paper was first reviewed by grok-4.5 on July 31, 2026.