Pith. sign in

REVIEW 3 major objections 9 minor 169 references

Endpoint Replay: Compressing the Recency Buffer in Deep Reinforcement Learning

T0 review · 3 major / 9 minor · reviewed 2026-07-31 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A deep RL agent can match a million-transition replay buffer with 10–50× less storage by keeping only chained n-step endpoints and updating them with expectile Sarsa.

desk verdict Clean failure-mode diagnosis plus a simple two-buffer fix that actually matches large-buffer Atari/Pinball at 10–50× less storage; theory is honest about its deterministic limit. read the letter →

arxiv 2607.25123 v1 pith:N4A3H3FD submitted 2026-07-27 cs.LG

Parham Mohammad Panahi , Armin Ashrafi , Haoyu Du , Andrew Patterson , Martha White , Adam White This is my paper
classification cs.LG
keywords DeepReinforcementLearningExperienceReplayCoresetsMultistepmethodsExpectileSarsaBuffercompressionUnanchoredbootstraptargets
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Standard deep RL keeps a huge first-in-first-out buffer of recent transitions—often a million of them—and samples it uniformly. This paper asks whether that buffer can be shrunk by an order of magnitude without losing the performance those large buffers buy. The answer it offers is Endpoint Replay: keep a small recency buffer for fresh one-step updates, and compress older experience into a coreset of connected n-step jumps so every bootstrap target stays inside the coreset. Isolated “representative” transitions fail because their next states are never themselves updated, producing unanchored and eventually corrupted targets; chaining the endpoints fixes that. An expectile loss then offsets the pessimistic bias that comes from multi-step returns collected under older, weaker policies. Empirically the method matches large-buffer scores in Pinball and twelve Atari games while beating equal-sized buffers and other compression baselines.

What carries the argument

Endpoint Replay: a small recency buffer plus a coreset of chained n-step transitions updated by n-step Expectile Sarsa. The chain anchors every bootstrap state and action; the expectile loss (τ > 0.5) counteracts multi-step pessimism from older policies.

What would settle it

In a controlled control setting, replace a large buffer with an equal-sized coreset of isolated one-step transitions versus chained n-step endpoints; if bootstrap-target error keeps rising and return collapses only for the isolated coreset, the unanchored-target diagnosis is confirmed.

Watch

Extended reading notes

Core claim

Naive coresets of isolated transitions create unanchored bootstrap targets—states and actions that are never updated—so value estimates drift and control performance collapses. Storing instead the endpoints of chained n-step sequences keeps every bootstrap target inside the buffer, and pairing those updates with an n-step expectile Sarsa loss removes the pessimistic bias of older multi-step returns. With that construction, a buffer one-tenth to one-fiftieth the usual size matches a standard million-transition recency buffer.

Load-bearing premise

The proof that policy iteration reaches the optimal policy holds only when the environment is deterministic; with high reward noise the expectile can chase lucky outcomes instead of better actions.

Editorial extensions

If this is right

  • Production DQN-style agents can cut replay memory by 10–50× while retaining large-buffer learning curves.
  • Any coreset or prototype selection that stores isolated transitions is at risk of the same unanchored-bootstrap failure.
  • n-step expectile operators are sound Bellman operators and induce an ordered family of value functions indexed by n.
  • Under deterministic dynamics, greedifying on n-step expectile values converges to the ordinary optimal policy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same anchoring idea could be applied inside prioritized or generative replay, not only uniform FIFO coresets.
  • Decaying τ toward 0.5 over training would give a practical way to retain early multi-step optimism while recovering mean returns in noisier domains.
  • If unanchored targets are the dominant failure mode, distance-based prototype selection without chaining should systematically underperform Endpoint Replay even when coverage looks good.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 9 minor

Summary. The paper asks whether the standard 1M-transition FIFO replay buffer in DQN-style algorithms can be compressed 10–50× without performance loss. It identifies a failure mode of naive coreset compression: isolated transitions yield "unanchored" bootstrap targets whose state-action values are never directly updated, and demonstrates this in controlled prediction (Fig. 1) and control (Fig. 2) experiments. The proposed fix, Endpoint Replay, keeps a small recency buffer plus a coreset of chained n-step transitions (so every bootstrap endpoint is itself updated), trained with an n-step Expectile Sarsa update (τ=0.7) to counteract the pessimistic bias of n-step targets from stale policies. Theory: the n-step expectile Bellman operator is a γⁿ-contraction (Thm 1), n-step expectile values are ordered in n (Thm 2), policy iteration converges under a monotonic-greedification assumption (Thm 3), and to the optimal policy under deterministic dynamics (Cor. 1); Prop. 1 shows expectile greedification can strictly reduce expected return. Empirically, Endpoint Replay matches a 10–50× larger recency buffer in Pinball (100 seeds) and 12 Atari games (10 seeds, sign tests), beating equal-sized buffers, reservoir coresets, and MeDQN; ablations attribute gains to state anchoring and the expectile loss.

Significance. If the results hold, this is a practically useful contribution: an order-of-magnitude reduction in replay memory with maintained performance matters for deployment and for scaling studies, and the "unanchored bootstrap target" diagnosis is a clean, previously unrecognized conceptual point backed by direct error measurements rather than only return curves. Strengths worth naming: publicly released code; unusually careful experimental hygiene for the area (100-seed Pinball runs with bootstrap CIs, two-stage tuning for the MeDQN baseline, sign tests, per-game breakdowns); controlled experiments that measure bootstrap-target error directly; and honest theory, including a counterexample (Prop. 1) against the authors' own update. The n-step expectile operator analysis (contraction for n≥1, ordering in n) is, to my knowledge, new. The main risks are a prose/pseudocode inconsistency about what the coreset actually contains, an untested robustness boundary for the expectile under stochastic dynamics, and a gap in the policy-iteration convergence proof.

major comments (3)
  1. [§3.3 vs. Algorithm 1 (App. 8), lines 11–17] The prose describes a stride-n chain: the lag buffer fills with n transitions, one n-step summary is emitted, and 'the lag buffer would simply reset' (and resets after k steps on termination), giving D_c = {(s_0,...s_n),(s_n,...s_2n),...} as in §3.1. The pseudocode instead pops only the FIRST transition from D_lag after emitting a summary and never resets: this implements a stride-1 sliding window in which every evicted transition generates an overlapping n-step summary. The two versions differ materially. Under stride-1, a 90k coreset covers ~90k environment steps of history, not ~900k, so the abstract's claim of 'an effective memory horizon comparable to a standard large buffer' no longer follows; anchoring also becomes trivial (every state is a start state) rather than a consequence of chaining. Additionally, line 14 computes g with constant γ^i and ignores the stored per-transition (
  2. [§3.2 and §5 (Table 3, Figs. 4c/d, 7, 8)] The mechanism attribution for the expectile is underdetermined by the evidence. The paper itself states the failure mode: with stochastic dynamics the expectile 'can chase environment stochasticity rather than ... actions that result in higher expected return' (§3.2), Corollary 1 gives optimality only under deterministic dynamics, and Proposition 1 constructs a case where expectile greedification strictly lowers expected return. Yet the headline benchmark deliberately injects stochasticity: sticky actions with repeat probability 0.25 in all 12 games (Table 3), and Ms. Pac-Man selected for 'high variance induced by unpredictable ghost movements'. The No-Expectile ablation is run only at this fixed noise level, so it cannot distinguish 'expectile corrects stale-policy pessimism' from 'expectile is generic optimism that happens to help here'. τ=0.7 is used everywhere with no reported sweep,
  3. [App. 7.3, Theorem 3 (and Corollary 1)] The proof of Theorem 3 establishes monotone, bounded improvement of V^{(n)}_{τ,π_t}, hence convergence to a fixed point of the greedification operator — but the conclusion that the limit is the OPTIMAL expectile policy ('will stop at some π*_τ that has the largest V') does not follow. In classical policy iteration the fixed-point argument gives V = max_{π'} T_{π'}V because improvement maximizes over policies; here greedification produces a single π′ satisfying Definition 2, so a fixed point only satisfies T_{π'}V_π = V_π for that particular π′, which need not be globally optimal without an additional argument that every suboptimal policy admits a strictly improving monotonic expectile greedy successor. Relatedly, convergence of π_t itself (rather than of the values) is asserted, and Assumption 1 is acknowledged to be unverified for the ε-greedy policy the algorithm actually uses — so the
minor comments (9)
  1. [App. 14, Figure 10] Caption says '100k buffer setting' but the legend and content are the 20k (50×) setting; also the caption says 'ablations' while showing baselines.
  2. [§3.1, Figure 1] Caption states 'Average over 100 seeds' while the text describes 10 datasets; the light lines are described as 10 random seeds. Please reconcile.
  3. [§5 / App. 12] Min–max normalization mapping the worst seed to 0 and best seed to 1 per game is sensitive to single-seed outliers and discards effect magnitude; consider reporting IQM with stratified bootstrap CIs (rliable-style) alongside, at least in the supplement.
  4. [App. 13] The sign test treats 120 game–seed pairs as independent; pairs within a game share game-level effects, which can inflate significance. Aggregating per game first (12 pairs) or a hierarchical test would be more conservative.
  5. [§5] MeDQN is excluded from Atari because it failed in Pinball, but its Pinball failure is attributed to invalid random states in a constrained domain — a domain-specific reason. The contribution claim 'outperforms MeDQN' therefore rests on a single, unfavorable setting; either soften the claim or include one Atari data point.
  6. [Abstract / Fig. 3d] The abstract says the method 'matches the performance of traditional large buffers'; in the 50× setting it does not quite reach the large baseline (5/12 games at parity). Qualify the claim by compression level.
  7. [§3.3 / §5] The 7:1 minibatch mix and n=10 are described as 'found effective' with no sensitivity results; even a coarse sweep in Pinball would help practitioners.
  8. [App. 7.2, Theorem 2] The ordering V^{(n)} ≥ V^{(a·n)} is stated for τ ≥ 1/2; a one-line remark on whether the ordering reverses for τ < 1/2 would complete the picture.
  9. [Typos] §5 'We preformed'; §4 'boostrap'; App. 7.1 'analagously'; §3.1 'a nearby states t'; Broader Impact 'bare little resemblance' (→ 'bear'). §2 'from early learn' (→ 'early learning').

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: theory is self-contained contraction/policy-iteration analysis; empirical claims are scored on external environment returns against independent baselines.

full rationale

The paper's load-bearing claims do not reduce to their inputs by construction. The n-step expectile action-values are introduced as a fixed-point definition Q^{(n)}_{τ,π}=e_τ(G^{(n)}(Q^{(n)}_{τ,π})), which is the standard Bellman-style recursion; existence and uniqueness follow from an explicit γ^n-contraction proof (Theorem 1) that does not bake in target performance. Ordering (Theorem 2) and policy-iteration convergence (Theorem 3, Corollary 1) are derived from monotonicity/subadditivity of expectiles under stated assumptions (including deterministic dynamics for optimality). None of these steps fit a parameter to data and rename the fit a prediction, import a uniqueness theorem from overlapping authors, or smuggle an ansatz via self-citation. Empirically, Endpoint Replay is evaluated on held-out Pinball and Atari returns against large-buffer DDQN, equal-sized recency buffers, reservoir coresets, and MeDQN; ablations isolate anchoring and the expectile loss on the same external metrics. Self-citations (e.g., Panahi et al. 2024, Patterson et al. 2024, Lo et al. 2024) supply environment or methodology context and are not load-bearing for the compression or soundness claims. The known limitation that optimality holds only under deterministic dynamics is a correctness/scope issue, not circularity. Score 0 is therefore appropriate.

Assumptions & free parameters 5 free parameters · 5 assumptions · 3 invented entities

The central empirical claim rests on the standard MDP/DQN stack plus several design knobs (n, τ, buffer split, mix ratio) chosen by the authors, and on a theory side-condition (deterministic dynamics for optimality; monotonic expectile greedification for PI) that is stronger than what ε-greedy Atari agents satisfy. No new physical entities; the main invented conceptual objects are the unanchored-target diagnosis and the n-step expectile action-value fixed point.

free parameters (5)
  • expectile level τ = 0.7
    Fixed to 0.7 for all Endpoint experiments; controls how aggressively n-step targets chase upper returns and is not derived.
  • chain length n = 10
    Subsampling stride and n-step horizon; set to 10 throughout main experiments without a selection principle beyond common multi-step practice.
  • recency/coreset split and minibatch mix = 90-10 capacities; 28:4 batch mix
    Default 90–10 capacity split and 28:4 (7:1) recency:coreset samples per batch of 32; described as found effective, not theoretically fixed.
  • Pinball learning rate = 0.002
    Tuned on the large-buffer baseline over a five-point grid and then frozen for all methods.
  • MeDDQN regularization weight λ = 2
    Two-stage sweep {1,2,4} then best λ=2 reported; baseline-only but affects comparative claims.
assumptions (5)
  • domain assumption Environment is an MDP with bounded rewards; episodic γ set to 0 at termination (White 2017).
    Standard RL problem formulation in §2; required for Bellman operators and returns.
  • standard math n-step expectile Bellman operator is a γ^n-contraction in sup norm, hence has a unique fixed point (Theorem 1).
    Proved from monotonicity and translation invariance of expectiles; load-bearing for soundness of the update.
  • ad hoc to paper Policy class is compact and closed under a monotonic expectile greedification operator (Assumption 1 / Definition 2).
    Needed for PI convergence (Theorem 3); paper notes standard softmax/ε-greedy are not shown to satisfy it.
  • domain assumption Deterministic transitions and rewards for Corollary 1 (expectile PI recovers the optimal expected-return policy).
    Stated explicitly; without it expectiles may chase environment noise. Atari sticky actions violate strict determinism.
  • domain assumption Double DQN / Adam / Dopamine-style Atari protocol is a valid performance yardstick for buffer compression.
    All comparisons are relative to this stack; claim is not architecture-agnostic.
invented entities (3)
  • Unanchored bootstrap targets independent evidence
    purpose: Name the failure mode of isolated coreset transitions whose next state-action pairs are never updated.
    Diagnostic concept introduced in §3.1 and probed in Figs. 1–2; parallels offline OOD actions but applied to online coresets.
  • n-step expectile action-values Q^{(n)}_{τ,π}
    purpose: Fixed points of the n-step expectile Bellman operator used by Expectile Sarsa.
    Defined in Eqs. 1–4; existence via Theorem 1. Independent mathematical object, but optimality link needs determinism.
  • Endpoint Replay (recency + lag + chained n-step coreset) independent evidence
    purpose: Concrete buffer-compression algorithm pairing anchored n-step coresets with expectile Sarsa and one-step DDQN on fresh data.
    Algorithm 1–2; evaluated empirically. No claim of a new physical quantity—purely algorithmic.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Endpoint Replay: Compressing the Recency Buffer in Deep Reinforcement Learning." pith.science (2026). https://pith.science/paper/N4A3H3FD

@misc{pith2026260725123,
  author       = {Pith},
  title        = {Pith review of: Endpoint Replay: Compressing the Recency Buffer in Deep Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N4A3H3FD}},
  note         = {Machine review of arXiv:2607.25123}
}
abstract

Experience replay remains one of the most practical and useful algorithmic tools in the deep reinforcement learning (DRL) toolbox. Aside from the limited success of prioritized replay and specialized approaches for large asynchronous systems, most DRL algorithms make use of a large, uniformly sampled recency buffer---even the size, one million, remains unchanged. Could we store less data, reduce redundancy, or more effectively chain experience together to speed up value propagation and still retain the performance of large buffers? In this paper, we investigate a simple compression approach that stores representative transitions derived from the end-points of a chain of connected $n$-step sequences. By curating these end-points in a smaller recency buffer, our method maintains an effective memory horizon comparable to a standard large buffer while requiring an order of magnitude less storage. Through empirical evaluation, we demonstrate that this approach prevents the systematic bias inherent in naive compression strategies and matches the performance of traditional large buffers in the Pinball environment and the Atari 2600 benchmark.

Figures

Figures reproduced from arXiv: 2607.25123 by the authors.

Figure 1
Figure 1. Mean square error between the boot￾strapped target Q(st+10, at+10) and sample return gt+10 of anchored, unanchored, recency and su￾pervised learners. Average over 100 seeds. Indi￾vidual runs of 10 random seeds are shown in light thin lines to improve visibility. For this experiment, we also store the Monte Carlo returns Gt+10 from the bootstrap state and action for each transition in the coreset, to compute the retu… view at source ↗
Figure 2
Figure 2. Endpoint replay can maintain better performance (left) and has lower bootstrap target value [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Endpoint replay performs comparably to a much larger recency buffer while outperforms [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Ablation study of Endpoint replay in Pinball and Atari. In Pinball we report mean perfor [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Performance of Endpoint replay and other baselines in individual Atari games in the 100k [PITH_FULL_IMAGE:figures/full_fig_p027_5.png]
Figure 6
Figure 6. Figure 6: Performance of Endpoint replay and other baselines in individual Atari games in the 20k [PITH_FULL_IMAGE:figures/full_fig_p028_6.png]
Figure 7
Figure 7. Figure 7: Performance of Endpoint replay and its ablations in individual Atari games in the 100k [PITH_FULL_IMAGE:figures/full_fig_p029_7.png]
Figure 8
Figure 8. Figure 8: Performance of Endpoint replay and its ablations in individual Atari games in the 20k [PITH_FULL_IMAGE:figures/full_fig_p030_8.png]
Figure 9
Figure 9. Figure 9: Performance of Endpoint replay and other baselines in individual Atari games in the 100k [PITH_FULL_IMAGE:figures/full_fig_p031_9.png]
Figure 10
Figure 10. Figure 10: Performance of Endpoint replay and its ablations in individual Atari games in the 100k [PITH_FULL_IMAGE:figures/full_fig_p032_10.png]
Figure 11
Figure 11. Figure 11: Performance of Endpoint replay and its ablations in individual Atari games in the 100k [PITH_FULL_IMAGE:figures/full_fig_p033_11.png]
Figure 12
Figure 12. Figure 12: Performance of Endpoint replay and its ablations in individual Atari games in the 20k [PITH_FULL_IMAGE:figures/full_fig_p034_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

169 extracted references · 10 linked inside Pith

  1. [2]

    Neural computation , volume=

    Risk-sensitive reinforcement learning , author=. Neural computation , volume=. 2014 , publisher=

    2014

  2. [3]

    Advances in Neural Information Processing Systems , volume=

    The nature of temporal difference errors in multi-step distributional reinforcement learning , author=. Advances in Neural Information Processing Systems , volume=

  3. [5]

    K-percent Evaluation for Lifelong RL , author=

  4. [6]

    International Conference on Learning Representations , year=

    Distributed Distributional Deterministic Policy Gradients , author=. International Conference on Learning Representations , year=

  5. [7]

    Advances in Neural Information Processing Systems , volume=

    State chrono representation for enhancing generalization in reinforcement learning , author=. Advances in Neural Information Processing Systems , volume=

  6. [8]

    arXiv preprint arXiv:2507.09087 , year=

    Deep reinforcement learning with gradient eligibility traces , author=. arXiv preprint arXiv:2507.09087 , year=

    arXiv

  7. [9]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Using hindsight to anchor past knowledge in continual learning , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  8. [10]

    ArXiv , year=

    The Laplacian in RL: Learning Representations with Efficient Approximations , author=. ArXiv , year=

Show all 169 references
  1. [11]

    International Conference on Machine Learning , year=

    Towards Sustainable Learning: Coresets for Data-efficient Deep Learning , author=. International Conference on Machine Learning , year=

  2. [12]

    IEEE Transactions on Signal Processing , year=

    Analyzing Sparse Dictionaries for Online Learning With Kernels , author=. IEEE Transactions on Signal Processing , year=

  3. [13]

    Journal of Machine Learning Research , volume=

    Empirical design in reinforcement learning , author=. Journal of Machine Learning Research , volume=

  4. [14]

    The Thirteenth International Conference on Learning Representations , year=

    Prioritized Generative Replay , author=. The Thirteenth International Conference on Learning Representations , year=

  5. [15]

    International Conference on Machine Learning , pages=

    Large Batch Experience Replay , author=. International Conference on Machine Learning , pages=. 2022 , organization=

    2022

  6. [16]

    Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence , pages=

    Efficient diversity-based experience replay for deep reinforcement learning , author=. Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence , pages=

  7. [17]

    IEEE Transactions on Signal Processing , year=

    The kernel recursive least-squares algorithm , author=. IEEE Transactions on Signal Processing , year=

  8. [18]

    AAAI Conference on Artificial Intelligence , year=

    Deep Reinforcement Learning with Double Q-Learning , author=. AAAI Conference on Artificial Intelligence , year=

  9. [19]

    2022 , journaltitle =

    A Generalized Projected Bellman Error for Off-Policy Value Estimation in Reinforcement Learning , author =. 2022 , journaltitle =

    2022

  10. [20]

    nature , volume=

    Human-level control through deep reinforcement learning , author=. nature , volume=. 2015 , publisher=

    2015

  11. [21]

    Proceedings of the Ninth National Conference on Artificial Intelligence - Volume 2 , pages =

    Lin, Long-Ji , title =. Proceedings of the Ninth National Conference on Artificial Intelligence - Volume 2 , pages =. 1991 , publisher =

    1991

  12. [23]

    and Barto, Andrew G

    Sutton, Richard S. and Barto, Andrew G. , publisher=. Reinforcement Learning:. 1998 , address=

    1998

  13. [24]

    R. S. Sutton and D. McAllester and S. Singh and Y. Mansour. Policy Gradient Methods for Reinforcement Learning with Function Approximation. Advances in Neural Information Processing Systems 12. 2000

    2000

  14. [25]

    R. J. Williams. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Machine Learning. 1992

    1992

  15. [26]

    International Conference on Learning Representations , year=

    Offline Reinforcement Learning with Implicit Q-Learning , author=. International Conference on Learning Representations , year=

  16. [27]

    , title =

    Indyk, Piotr and Mahabadi, Sepideh and Mahdian, Mohammad and Mirrokni, Vahab S. , title =. Proceedings of the 33rd ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems , pages =. 2014 , publisher =

    2014

  17. [28]

    Journal of Machine Learning Research (JMLR) , volume =

    Temporal Abstraction in Reinforcement Learning with the Successor Representation , author =. Journal of Machine Learning Research (JMLR) , volume =

  18. [29]

    International Conference on Machine Learning (ICML) , year =

    Deep Laplacian-based Options for Temporally-Extended Exploration , author =. International Conference on Machine Learning (ICML) , year =

  19. [30]

    International Conference on Learning Representations (ICLR) , year =

    Proper Laplacian Representation Learning , author =. International Conference on Learning Representations (ICLR) , year =

  20. [31]

    3rd International Conference on Learning Representations (ICLR) , year=

    Adam: A Method for Stochastic Optimization , author=. 3rd International Conference on Learning Representations (ICLR) , year=

  21. [32]

    Sutton , title =

    Shangtong Zhang and Richard S. Sutton , title =. Deep Reinforcement Learning Symposium, NIPS 2017 , volume =

    2017

  22. [33]

    The Reinforcement Learning Journal , volume=

    Investigating the Interplay of Prioritized Replay and Generalization , author=. The Reinforcement Learning Journal , volume=

  23. [34]

    Advances in Neural Information Processing Systems , volume=

    Synthetic experience replay , author=. Advances in Neural Information Processing Systems , volume=

  24. [35]

    International Conference on Machine Learning , pages=

    Adapting kernel representations online using submodular maximization , author=. International Conference on Machine Learning , pages=. 2017 , organization=

    2017

  25. [36]

    Advances in neural information processing systems , volume=

    Double Q-learning , author=. Advances in neural information processing systems , volume=

  26. [37]

    International Conference on Machine Learning , pages=

    Unifying task specification in reinforcement learning , author=. International Conference on Machine Learning , pages=. 2017 , organization=

    2017

  27. [38]

    arXiv preprint arXiv:1912.01603 , year=

    Dream to control: Learning behaviors by latent imagination , author=. arXiv preprint arXiv:1912.01603 , year=

    1912 arXiv

  28. [39]

    arXiv preprint arXiv:1903.00374 , year=

    Model-based reinforcement learning for atari , author=. arXiv preprint arXiv:1903.00374 , year=

    1903 arXiv

  29. [40]

    Medical Imaging with Deep Learning , pages =

    Selective experience replay compression using coresets for lifelong deep reinforcement learning in medical imaging , author =. Medical Imaging with Deep Learning , pages =. 2024 , volume =

    2024

  30. [41]

    Proceedings of the 40th International Conference on Machine Learning , pages =

    Prototype-Sample Relation Distillation: Towards Replay-Free Continual Learning , author =. Proceedings of the 40th International Conference on Machine Learning , pages =. 2023 , volume =

    2023

  31. [42]

    The big world hypothesis and its ramifications for artificial intelligence , author=

  32. [43]

    Artificial intelligence , volume=

    Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning , author=. Artificial intelligence , volume=. 1999 , publisher=

    1999

  33. [44]

    Artificial Intelligence , volume=

    Reward-respecting subtasks for model-based reinforcement learning , author=. Artificial Intelligence , volume=. 2023 , publisher=

    2023

  34. [45]

    Journal of Machine Learning Research , volume=

    Goal-space planning with subgoal models , author=. Journal of Machine Learning Research , volume=

  35. [46]

    Journal of Artificial Intelligence Research , volume=

    Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents , author=. Journal of Artificial Intelligence Research , volume=

  36. [47]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Deep reinforcement learning with double q-learning , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  37. [48]

    Machine learning , volume=

    Convergence results for single-step on-policy reinforcement-learning algorithms , author=. Machine learning , volume=. 2000 , publisher=

    2000

  38. [49]

    and Naddaf, Yavar and Veness, Joel and Bowling, Michael , journal =

    Bellemare, Marc G. and Naddaf, Yavar and Veness, Joel and Bowling, Michael , journal =. The

  39. [50]

    International Conference on Machine Learning , pages=

    Atari-5: Distilling the arcade learning environment down to five games , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  40. [51]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Rainbow: Combining improvements in deep reinforcement learning , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  41. [52]

    International Conference on Machine Learning , pages=

    Revisiting rainbow: Promoting more insightful and inclusive deep reinforcement learning research , author=. International Conference on Machine Learning , pages=. 2021 , organization=

    2021

  42. [53]

    Proceedings of the 27th International Joint Conference on Artificial Intelligence , pages=

    Organizing experience: a deeper look at replay mechanisms for sample-based planning in continuous state domains , author=. Proceedings of the 27th International Joint Conference on Artificial Intelligence , pages=

  43. [54]

    Proceedings of the 37th International Conference on Machine Learning , pages =

    Coresets for Data-efficient Training of Machine Learning Models , author =. Proceedings of the 37th International Conference on Machine Learning , pages =. 2020 , volume =

    2020

  44. [55]

    Experience Selection in Deep Reinforcement Learning for Control , journal =

    Tim de Bruin and Jens Kober and Karl Tuyls and Robert Babu. Experience Selection in Deep Reinforcement Learning for Control , journal =. 2018 , volume =

    2018

  45. [56]

    Fedorov, Valerii , year =

  46. [57]

    International conference on machine learning , pages=

    Revisiting fundamentals of experience replay , author=. International conference on machine learning , pages=. 2020 , organization=

    2020

  47. [58]

    International Conference on Learning Representations , year=

    Distributed Prioritized Experience Replay , author=. International Conference on Learning Representations , year=

  48. [59]

    Proceedings of the 40th International Conference on Machine Learning , pages =

    Towards Sustainable Learning: Coresets for Data-efficient Deep Learning , author =. Proceedings of the 40th International Conference on Machine Learning , pages =. 2023 , editor =

    2023

  49. [60]

    Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence,

    Organizing Experience: a Deeper Look at Replay Mechanisms for Sample-Based Planning in Continuous State Domains , author =. Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence,. 2018 , month =

    2018

  50. [61]

    No More Pesky Hyperparameters: Offline Hyperparameter Tuning for

    Han Wang and Archit Sakhadeo and Adam M White and James M Bell and Vincent Liu and Xutong Zhao and Puer Liu and Tadashi Kozuno and Alona Fyshe and Martha White , journal=. No More Pesky Hyperparameters: Offline Hyperparameter Tuning for

  51. [62]

    2013 , journal=

    Playing Atari with Deep Reinforcement Learning , author=. 2013 , journal=

    2013

  52. [63]

    International Conference on Learning Representations (ICLR) , year=

    Prioritized Experience Replay , author=. International Conference on Learning Representations (ICLR) , year=

  53. [64]

    Reinforcement Learning Conference , year =

    Investigating the Interplay of Prioritized Replay and Generalization , author=. Reinforcement Learning Conference , year =

  54. [65]

    Reachability-Aware

    Wang, Kaixin and Zhou, Kuangqi and Feng, Jiashi and Hooi, Bryan and Wang, Xinchao , booktitle =. Reachability-Aware. 2023 , volume =

    2023

  55. [66]

    Advances in neural information processing systems , volume=

    Safe and efficient off-policy reinforcement learning , author=. Advances in neural information processing systems , volume=

  56. [67]

    A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning , volume =

    Adkins, Jacob and Bowling, Michael and White, Adam , booktitle =. A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning , volume =

  57. [69]

    Medical Imaging with Deep Learning , pages=

    Selective experience replay compression using coresets for lifelong deep reinforcement learning in medical imaging , author=. Medical Imaging with Deep Learning , pages=. 2024 , organization=

    2024

  58. [70]

    SIAM Journal on Computing , volume=

    Bisimulation Metrics for Continuous Markov Decision Processes , author=. SIAM Journal on Computing , volume=. 2011 , publisher=

    2011

  59. [71]

    A Kernel Perspective on Behavioural Metrics for Markov Decision Processes , author=

  60. [72]

    Advances in Neural Information Processing Systems , volume=

    MICo: Improved representations via sampling-based state similarity for Markov decision processes , author=. Advances in Neural Information Processing Systems , volume=

  61. [73]

    International conference on machine learning , pages=

    Asynchronous methods for deep reinforcement learning , author=. International conference on machine learning , pages=. 2016 , organization=

    2016

  62. [74]

    International Conference on Machine Learning , pages=

    Trajectory-aware eligibility traces for off-policy reinforcement learning , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  63. [75]

    Machine learning , volume=

    Self-improving reactive agents based on reinforcement learning, planning and teaching , author=. Machine learning , volume=. 1992 , publisher=

    1992

  64. [76]

    arXiv preprint arXiv:2509.15032 , year=

    Sample Efficient Experience Replay in Non-stationary Environments , author=. arXiv preprint arXiv:2509.15032 , year=

  65. [77]

    Transactions on Machine Learning Research , volume=

    Uncertainty-Based Experience Replay for Task-Agnostic Continual Reinforcement Learning , author=. Transactions on Machine Learning Research , volume=

  66. [78]

    Advances in Neural Information Processing Systems , volume=

    Continual Learning with Deep Generative Replay , author=. Advances in Neural Information Processing Systems , volume=

  67. [81]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Selective experience replay for lifelong learning , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  68. [82]

    2023 IEEE/CVF International Conference on Computer Vision (ICCV) , pages=

    Prototype Reminiscence and Augmented Asymmetric Knowledge Aggregation for Non-Exemplar Class-Incremental Learning , author=. 2023 IEEE/CVF International Conference on Computer Vision (ICCV) , pages=. 2023 , organization=

    2023

  69. [83]

    Advances in Neural Information Processing Systems , volume=

    Experience Replay for Continual Learning , author=. Advances in Neural Information Processing Systems , volume=

  70. [84]

    Proceedings of the 28th International Joint Conference on Artificial Intelligence , pages=

    Experience replay optimization , author=. Proceedings of the 28th International Joint Conference on Artificial Intelligence , pages=

  71. [85]

    Frontiers in neurorobotics , volume=

    Map-based experience replay: a memory-efficient solution to catastrophic forgetting in reinforcement learning , author=. Frontiers in neurorobotics , volume=

  72. [86]

    Advances in Neural Information Processing Systems , volume=

    Sample-Efficient Deep Reinforcement Learning via Episodic Backward Update , author=. Advances in Neural Information Processing Systems , volume=

  73. [87]

    Proceedings of the National Academy of Sciences of the United States of America , volume=

    Prioritized experience replays on a hippocampal predictive map for learning , author=. Proceedings of the National Academy of Sciences of the United States of America , volume=

  74. [88]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Attentive Experience Replay , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  75. [89]

    Transactions on Machine Learning Research , year=

    Introspective Experience Replay: Look Back When Surprised , author=. Transactions on Machine Learning Research , year=

  76. [90]

    International Conference on Learning Representations , year=

    Topological Experience Replay , author=. International Conference on Learning Representations , year=

  77. [91]

    Applied Intelligence , volume=

    Revisiting experience replayable conditions , author=. Applied Intelligence , volume=. 2024 , publisher=

    2024

  78. [92]

    arXiv preprint arXiv:2102.03261 , year=

    Revisiting prioritized experience replay: A value perspective , author=. arXiv preprint arXiv:2102.03261 , year=

    arXiv

  79. [93]

    The Eleventh International Conference on Learning Representations , year =

    Sample-Efficient Reinforcement Learning by Breaking the Replay Ratio Barrier , author=. The Eleventh International Conference on Learning Representations , year =

  80. [94]

    The Twelfth International Conference on Learning Representations , year=

    Revisiting plasticity in visual reinforcement learning: Data, modules and training stages , author=. The Twelfth International Conference on Learning Representations , year=

  81. [95]

    International Conference on Machine Learning , pages=

    A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control , author=. International Conference on Machine Learning , pages=. 2025 , organization=

    2025

  82. [96]

    arXiv preprint arXiv:2512.01034 , year=

    Addressing the Plasticity-Stability Dilemma in Reinforcement Learning , author=. arXiv preprint arXiv:2512.01034 , year=

  83. [97]

    Transactions on Machine Learning Research , issn=

    Memory-efficient Reinforcement Learning with Value-based Knowledge Consolidation , author=. Transactions on Machine Learning Research , issn=

  84. [98]

    Advances in Neural Information Processing Systems , volume=

    Deep policy gradient methods without batch updates, target networks, or replay buffers , author=. Advances in Neural Information Processing Systems , volume=

  85. [99]

    Advances in Neural Information Processing Systems , volume=

    Hindsight Experience Replay , author=. Advances in Neural Information Processing Systems , volume=

  86. [100]

    International Conference on Machine Learning , pages=

    Universal Value Function Approximators , author=. International Conference on Machine Learning , pages=. 2015 , organization=

    2015

  87. [101]

    , booktitle=

    Peng, Jing and Williams, Ronald J. , booktitle=. Efficient learning and planning within the. 1993 , organization=

    1993

  88. [102]

    Advances in Neural Information Processing Systems , volume=

    When to use parametric models in reinforcement learning? , author=. Advances in Neural Information Processing Systems , volume=

  89. [103]

    International Conference on Learning Representations , year=

    Reinforcement Learning with Unsupervised Auxiliary Tasks , author=. International Conference on Learning Representations , year=

  90. [104]

    Advances in neural information processing systems , volume=

    Memory-based reinforcement learning: Efficient computation with prioritized sweeping , author=. Advances in neural information processing systems , volume=

  91. [105]

    Learning for Dynamics and Control Conference , pages=

    Experience replay with likelihood-free importance weights , author=. Learning for Dynamics and Control Conference , pages=. 2022 , organization=

    2022

  92. [106]

    Nature , volume=

    Outracing champion Gran Turismo drivers with deep reinforcement learning , author=. Nature , volume=. 2022 , publisher=

    2022

  93. [108]

    PLoS One , volume=

    Dual experience replay enhanced deep deterministic policy gradient for efficient continuous data sampling , author=. PLoS One , volume=. 2025 , publisher=

    2025

  94. [109]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Deep q-learning from demonstrations , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  95. [110]

    the Multidisciplinary Symposium on Reinforcement Learning, Montreal, Canada , year=

    Skill chaining: Skill discovery in continuous domains , author=. the Multidisciplinary Symposium on Reinforcement Learning, Montreal, Canada , year=

  96. [111]

    and da Silva, Bruno Castro , booktitle=

    Maheshwari, Mansi and Raisbeck, John C. and da Silva, Bruno Castro , booktitle=

  97. [112]

    Permanent and Transient Representations for Continual Reinforcement Learning , author=

  98. [113]

    International conference on machine learning , pages=

    The primacy bias in deep reinforcement learning , author=. International conference on machine learning , pages=. 2022 , organization=

    2022

  99. [114]

    Proceedings of the 1st Reinforcement Learning Conference (RLC) , year=

    An Analysis of Action-Value Temporal-Difference Methods That Learn State Values , author=. Proceedings of the 1st Reinforcement Learning Conference (RLC) , year=

  100. [115]

    International Conference on Machine Learning , pages=

    VA-learning as a more efficient alternative to Q-learning , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  101. [116]

    Atari-5: Distilling the arcade learning environment down to five games

    Matthew Aitchison, Penny Sweetser, and Marcus Hutter. Atari-5: Distilling the arcade learning environment down to five games. In International Conference on Machine Learning, pp.\ 421--438. PMLR, 2023

    2023

  102. [117]

    Permanent and transient representations for continual reinforcement learning, 2026

    Nishanth Anand and Doina Precup. Permanent and transient representations for continual reinforcement learning, 2026

    2026

  103. [118]

    Hindsight experience replay

    Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, volume 30, 2017

    2017

  104. [119]

    Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling

    Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The Arcade Learning Environment : An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 0 253--279, 2013

    2013

  105. [120]

    On tiny episodic memories in continual learning

    Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc'Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10486, 2019

    1902 arXiv

  106. [121]

    Using hindsight to anchor past knowledge in continual learning

    Arslan Chaudhry, Albert Gordo, Puneet Dokania, Philip Torr, and David Lopez-Paz. Using hindsight to anchor past knowledge in continual learning. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp.\ 6993--7001, 2021

    2021

  107. [122]

    Bootstrapping expectiles in reinforcement learning

    Pierre Clavier, Emmanuel Rachelson, Erwan Le Pennec, and Matthieu Geist. Bootstrapping expectiles in reinforcement learning. arXiv preprint arXiv:2406.04081, 2024

    2024 arXiv

  108. [123]

    Sample-efficient reinforcement learning by breaking the replay ratio barrier

    Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Marc G Bellemare, and Aaron Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. The Eleventh International Conference on Learning Representations, 2022

    2022

  109. [124]

    Rupam Mahmood

    Mohamed Elsayed, Gautham Vasan, and A. Rupam Mahmood. Streaming deep reinforcement learning finally works. arXiv preprint arXiv:2410.14606, 2024

    2024 arXiv

  110. [125]

    Revisiting fundamentals of experience replay

    William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay. In International conference on machine learning, pp.\ 3061--3071. PMLR, 2020

    2020

  111. [126]

    Etgl-ddpg: a deep deterministic policy gradient algorithm for sparse reward continuous control

    Ehsan Futuhi, Shayan Karimi, Chao Gao, and Martin M \"u ller. Etgl-ddpg: a deep deterministic policy gradient algorithm for sparse reward continuous control. arXiv preprint arXiv:2410.05225, 2024

    2024

  112. [127]

    Double q-learning

    Hado Hasselt. Double q-learning. Advances in neural information processing systems, 23, 2010

    2010

  113. [128]

    Rainbow: Combining improvements in deep reinforcement learning

    Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence,...

    2018

  114. [129]

    Deep q-learning from demonstrations

    Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  115. [130]

    Topological experience replay

    Zhang-Wei Hong, Tao Chen, Yen-Chen Lin, Joni Pajarinen, and Pulkit Agrawal. Topological experience replay. In International Conference on Learning Representations, 2022

    2022

  116. [131]

    Prioritized experience replays on a hippocampal predictive map for learning

    Hideyoshi Igata, Yuji Ikegaya, and Takuya Sasaki. Prioritized experience replays on a hippocampal predictive map for learning. Proceedings of the National Academy of Sciences of the United States of America, 118 0 (1): 0 e2011266118, 2021

    2021

  117. [132]

    Selective experience replay for lifelong learning

    David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  118. [133]

    Reinforcement learning with unsupervised auxiliary tasks

    Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In International Conference on Learning Representations, 2017

    2017

  119. [134]

    A forget-and-grow strategy for deep reinforcement learning scaling in continuous control

    Zilin Kang, Chenyuan Hu, Yu Luo, Zhecheng Yuan, Ruijie Zheng, and Huazhe Xu. A forget-and-grow strategy for deep reinforcement learning scaling in continuous control. In International Conference on Machine Learning, pp.\ 28921--28942. PMLR, 2025

    2025

  120. [135]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations (ICLR), 2015

    2015

  121. [136]

    Skill chaining: Skill discovery in continuous domains

    George Konidaris and Andrew Barto. Skill chaining: Skill discovery in continuous domains. In the Multidisciplinary Symposium on Reinforcement Learning, Montreal, Canada, 2009

    2009

  122. [137]

    Offline reinforcement learning with implicit q-learning

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022

    2022

  123. [138]

    Introspective experience replay: Look back when surprised

    Ramnath Kumar and Dheeraj Mysore Nagaraj. Introspective experience replay: Look back when surprised. Transactions on Machine Learning Research, 2022

    2022

  124. [139]

    Memory-efficient reinforcement learning with value-based knowledge consolidation

    Qingfeng Lan, Yangchen Pan, Jun Luo, and A Rupam Mahmood. Memory-efficient reinforcement learning with value-based knowledge consolidation. Transactions on Machine Learning Research, 2022. ISSN 2835-8856

    2022

  125. [140]

    Sample-efficient deep reinforcement learning via episodic backward update

    Su Young Lee, Choi Sungik, and Sae-Young Chung. Sample-efficient deep reinforcement learning via episodic backward update. Advances in Neural Information Processing Systems, 32, 2019

    2019

  126. [141]

    Programming robots using reinforcement learning and teaching

    Long-Ji Lin. Programming robots using reinforcement learning and teaching. In Proceedings of the Ninth National Conference on Artificial Intelligence - Volume 2, AAAI'91, pp.\ 781–786. AAAI Press, 1991

    1991

  127. [142]

    Goal-space planning with subgoal models

    Chunlok Lo, Kevin Roice, Parham Mohammad Panahi, Scott M Jordan, Adam White, Gabor Mihucz, Farzane Aminmansour, and Martha White. Goal-space planning with subgoal models. Journal of Machine Learning Research, 25 0 (330): 0 1--57, 2024

    2024

  128. [143]

    Synthetic experience replay

    Cong Lu, Philip Ball, Yee Whye Teh, and Jack Parker-Holder. Synthetic experience replay. Advances in Neural Information Processing Systems, 36: 0 46323--46344, 2023

    2023

  129. [144]

    Actor-critic algorithm for dynamic expectile and cvar

    Yudong Luo and Erick Delage. Actor-critic algorithm for dynamic expectile and cvar. arXiv preprint arXiv:2605.07857, 2026

    2026 arXiv

  130. [145]

    Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents

    Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61: 0 523--562, 2018

    2018

  131. [146]

    Raisbeck, and Bruno Castro da Silva

    Mansi Maheshwari, John C. Raisbeck, and Bruno Castro da Silva. AltNet : Alternating network resets for plasticity. In Proceedings of the Fourth Conference on Lifelong Learning Agents (CoLLAs), Workshop Track, 2025

    2025

  132. [147]

    Coresets for data-efficient training of machine learning models

    Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pp.\ 6950--6960. Proceedings of Machine Learning Research, PMLR, 2020

    2020

  133. [148]

    Playing atari with deep reinforcement learning

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. NIPS Deep Learning Workshop 2013, 2013

    2013

  134. [149]

    Human-level control through deep reinforcement learning

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518 0 (7540): 0 529--533, 2015

    2015

  135. [150]

    Asynchronous methods for deep reinforcement learning

    Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp.\ 1928--1937. PMLR, 2016

    1928

  136. [151]

    Dual experience replay enhanced deep deterministic policy gradient for efficient continuous data sampling

    Teh Noranis Mohd Aris, Ningning Chen, Norwati Mustapha, and Maslina Zolkepli. Dual experience replay enhanced deep deterministic policy gradient for efficient continuous data sampling. PLoS One, 20 0 (11): 0 e0334411, 2025

    2025

  137. [152]

    Memory-based reinforcement learning: Efficient computation with prioritized sweeping

    Andrew Moore and Christopher Atkeson. Memory-based reinforcement learning: Efficient computation with prioritized sweeping. Advances in neural information processing systems, 5, 1992

    1992

  138. [153]

    Safe and efficient off-policy reinforcement learning

    R \'e mi Munos, Tom Stepleton, Anna Harutyunyan, and Marc Bellemare. Safe and efficient off-policy reinforcement learning. Advances in neural information processing systems, 29, 2016

    2016

  139. [154]

    Organizing experience: a deeper look at replay mechanisms for sample-based planning in continuous state domains

    Yangchen Pan, Muhammad Zaheer, Adam White, Andrew Patterson, and Martha White. Organizing experience: a deeper look at replay mechanisms for sample-based planning in continuous state domains. In Proceedings of the 27th International Joint Conference on Artificial Intelligence,...

    2018

  140. [155]

    Investigating the interplay of prioritized replay and generalization

    Parham Mohammad Panahi, Andrew Patterson, Martha White, and Adam White. Investigating the interplay of prioritized replay and generalization. The Reinforcement Learning Journal, 1, 2024

    2024

  141. [156]

    Empirical design in reinforcement learning

    Andrew Patterson, Samuel Neumann, Martha White, and Adam White. Empirical design in reinforcement learning. Journal of Machine Learning Research, 25 0 (318): 0 1--63, 2024

    2024

  142. [157]

    Williams

    Jing Peng and Ronald J. Williams. Efficient learning and planning within the Dyna framework. In Proceedings of 1993 International Conference on Neural Networks (ICNN-93), volume 1, pp.\ 168--174. IEEE, 1993

    1993

  143. [158]

    Experience replay for continual learning

    David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in Neural Information Processing Systems, 32, 2019

    2019

  144. [159]

    Universal value function approximators

    Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International Conference on Machine Learning, pp.\ 1312--1320. PMLR, 2015

    2015

  145. [160]

    Prioritized experience replay

    Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In International Conference on Learning Representations (ICLR), 2016. Poster

    2016

  146. [161]

    Risk-sensitive reinforcement learning

    Yun Shen, Michael J Tobia, Tobias Sommer, and Klaus Obermayer. Risk-sensitive reinforcement learning. Neural computation, 26 0 (7): 0 1298--1328, 2014

    2014

  147. [162]

    Experience replay with likelihood-free importance weights

    Samarth Sinha, Jiaming Song, Animesh Garg, and Stefano Ermon. Experience replay with likelihood-free importance weights. In Learning for Dynamics and Control Conference, pp.\ 110--123. PMLR, 2022

    2022

  148. [163]

    Attentive experience replay

    Peiquan Sun, Wengang Zhou, and Houqiang Li. Attentive experience replay. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp.\ 5900--5907, 2020

    2020

  149. [164]

    The nature of temporal difference errors in multi-step distributional reinforcement learning

    Yunhao Tang, R \'e mi Munos, Mark Rowland, Bernardo Avila Pires, Will Dabney, and Marc Bellemare. The nature of temporal difference errors in multi-step distributional reinforcement learning. Advances in Neural Information Processing Systems, 35: 0 30265--30276, 2022

    2022

  150. [165]

    Va-learning as a more efficient alternative to q-learning

    Yunhao Tang, R \'e mi Munos, Mark Rowland, and Michal Valko. Va-learning as a more efficient alternative to q-learning. In International Conference on Machine Learning, pp.\ 33739--33757. PMLR, 2023

    2023

  151. [166]

    Deep reinforcement learning with double q-learning

    Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016

    2016

  152. [167]

    When to use parametric models in reinforcement learning? Advances in Neural Information Processing Systems, 32, 2019

    Hado P Van Hasselt, Matteo Hessel, and John Aslanides. When to use parametric models in reinforcement learning? Advances in Neural Information Processing Systems, 32, 2019

    2019

  153. [168]

    Alireza Azimi, Jincheng He, Farzaneh Shahriar, Colin Bellinger, Martha White, and A

    Gautham Vasan, Mohamed Elsayed, S. Alireza Azimi, Jincheng He, Farzaneh Shahriar, Colin Bellinger, Martha White, and A. Rupam Mahmood. Deep policy gradient methods without batch updates, target networks, or replay buffers. In Advances in Neural Information Processing Systems, ...

    2024

  154. [169]

    Prioritized generative replay

    Renhao Wang, Kevin Frans, Pieter Abbeel, Sergey Levine, and Alexei A Efros. Prioritized generative replay. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  155. [170]

    Unifying task specification in reinforcement learning

    Martha White. Unifying task specification in reinforcement learning. In International Conference on Machine Learning, pp.\ 3742--3750. PMLR, 2017

    2017

  156. [171]

    Outracing champion gran turismo drivers with deep reinforcement learning

    Peter R Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas J Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, et al. Outracing champion gran turismo drivers with deep reinforcement learning. Nature, 602 0 (7896): 0 22...

    2022

  157. [172]

    The in-sample softmax for offline reinforcement learning

    Chenjun Xiao, Han Wang, Yangchen Pan, Adam White, and Martha White. The in-sample softmax for offline reinforcement learning. arXiv preprint arXiv:2302.14372, 2023

    2023 arXiv

  158. [173]

    Augmenting replay in world models for continual reinforcement learning

    Luke Yang, Levin Kuhlmann, and Gideon Kowadlo. Augmenting replay in world models for continual reinforcement learning. arXiv preprint arXiv:2401.16650, 2024

    2024 arXiv

  159. [174]

    Shangtong Zhang and Richard S. Sutton. A deeper look at experience replay. Deep Reinforcement Learning Symposium, NIPS 2017, abs/1712.01275, 2017

    2017 arXiv

  160. [175]

    Jacobs, and Vishwa Sanjay Parekh

    Guangyao Zheng, Samson Zhou, Vladimir Braverman, Michael A. Jacobs, and Vishwa Sanjay Parekh. Selective experience replay compression using coresets for lifelong deep reinforcement learning in medical imaging. In Medical Imaging with Deep Learning, volume 227, pp.\ 1751--1764....

    2024

  161. [176]

    Selective experience replay compression using coresets for lifelong deep reinforcement learning in medical imaging

    Guangyao Zheng, Samson Zhou, Vladimir Braverman, Michael A Jacobs, and Vishwa Sanjay Parekh. Selective experience replay compression using coresets for lifelong deep reinforcement learning in medical imaging. In Medical Imaging with Deep Learning, pp.\ 1751--1764. PMLR, 2024 b

    2024

Pith tools

Reviewed July 31, 2026 · model on record in the stance chip above.