Pith. sign in

REVIEW 3 major objections 5 minor 54 references

Reward Structure Shapes the Interaction Between Episodic Exploration and Neural Memory in Reinforcement Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that an identical exploration bonus helps, equalizes, or does nothing to neural memory agents depending on what the reward supervises, not on how often it pays.

desk verdict A large, carefully analyzed empirical matrix with a real formal overlay; the cross-environment typology is confounded, but the within-environment results and the sparsity definitions earn it a referee. read the letter →

arxiv 2608.05111 v1 pith:SDLQ6PBF submitted 2026-08-05 cs.LG

Jai Malegaonkar , Rohan Patil , Henrik I. Christensen This is my paper
classification cs.LG
keywords partialobservabilityexplorationbonusesmemoryarchitecturesrewardmachinessparsityreinforcementlearningepisodicsequential
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper studies how episodic exploration bonuses interact with neural memory architectures in partially observable reinforcement learning. It claims that the same bonus produces three distinct outcomes—amplifying differences between architectures, equalizing them to a shared ceiling, or doing nothing—depending on how the environment's reward relates to the memory the task requires. The paper argues that what matters is not how often a reward is paid but whether it supervises the latent memory that solving the task demands. This matters because exploration and memory are usually evaluated separately, and the standard notion of sparse reward is defined by reward frequency, which the results here show can mislead.

What carries the argument

The central object is the observation-anchored reward machine (POMDPRM), which attaches a finite automaton to the observation/action stream to reproduce the expected return of a POMDP reward. On this foundation the paper defines two properties: structural sparsity, where an automaton can reproduce the return without retaining the task-required history so the latent memory is never reward-supervised, and potential sparsity, where the one-step reward misprices local exploratory actions. It then proves Theorem 1: if a reward is structurally dense and potentially dense, the greedy policy over any perfect encoding machine is already optimal, leaving no exploratory deficit for a bonus to fix. These definitions organize the three empirical regimes by the retention burden each task exposes, while the experimental matrix—six memory architectures crossed with two episodic bonuses under controlled reward variants—provides the evidence that the patterns track reward structure rather than density.

What would settle it

Construct two environments that are identical in observation format, action space, horizon, training budget, and hyperparameters, and differ only in how memory content is acquired (active discovery versus a purely scheduled stream). The paper's taxonomy predicts the amplification/equalization/null ordering should follow that axis alone; if the pattern does not transfer, the proposed axis is not the governing variable.

Watch

Extended reading notes

Core claim

The central discovery is a structural taxonomy: when memory content must be actively discovered and retained without reward supervision (MysteryPath), an episodic bonus amplifies differences in sequence-modeling capacity, taking high-capacity architectures like RetNet and GatedDeltaNet from roughly 0.16 to 0.6 success while leaving GRU and LSTM near baseline. When the critical information is a single reward-supervised cue that must first be found (MiniGrid-MemoryS13 in the 3×3 view), the same bonus equalizes architectures by lifting stuck cells to the ceiling, while architectures that already succeed gain little. When the observation stream is purely scheduled and action-independent (TinyReproduce), the bonus is null. Controlled reward manipulations within MysteryPath verify the mechanism: a dense reward that directly supervises the required memory makes the bonus redundant or harmful, a density-matched distractor reward does not, and a small avoidable penalty on exploratory moves freezes all architectures at zero success, which either bonus breaks.

Load-bearing premise

The three interaction patterns are attributed to differences in how memory content is acquired, but the three environments also differ in observation format, action space, horizon, training budget, and learning-rate overrides; no environment pair varies only the memory-acquisition mode.

Editorial extensions

If this is right

  • Exploration bonuses cannot be evaluated at a fixed architecture; measured bonus gains are architecture-dependent and task-structure-dependent.
  • Reward sparsity should be redefined as a property of what the reward supervises, not of how often it pays; two rewards with identical frequency can have opposite effects on a bonus.
  • A dense reward does not automatically remove the need for exploration: it only does so when it supervises the memory the task requires.
  • Adding avoidable penalties on exploratory actions can drive policies into suboptimal stationary states even when the theoretical optimum is unchanged, and exploration bonuses can rescue this without outbidding the penalty.
  • Practitioners must ask what the reward inherently supervises and whether the architecture can retain what the bonus exposes, because state coverage without recurrent capacity actively degrades performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the reward-structure axis survives a matched-environment test, memory benchmarks should report both the reward's density and its supervision target, and architecture rankings should be conditioned on the task's retention burden.
  • The equalization result suggests that for tasks with a single reward-supervised cue, architecture choice matters less than exploration, while for tasks with unsupervised memory content, capacity differences become decisive.
  • The null result on a purely scheduled stream points to a broader candidate principle: an intrinsic reward cannot reorder policies when the observation stream is action-independent, which could be tested by making the dictation schedule partially action-dependent while holding everything else fixed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies how episodic exploration bonuses interact with memory architectures in partially observable RL. Six architectures (GRU, LSTM, RetNet, GatedDeltaNet, Mamba-2, Memoryless) are crossed with no bonus, E3B, and NovelD in three POMDP environments (MysteryPath-Grid, MiniGrid-MemoryS13, TinyReproduce). The authors report three interaction patterns: the bonus amplifies architecture differences where memory content must be discovered and retained unsupervised (MysteryPath), equalizes architectures where the required content is a single reward-supervised cue (MemoryS13), and has no effect where the observation stream is purely scheduled (TinyReproduce). Reward variants within MysteryPath and MemoryS13 (sparse, penalty, distractor, aligned) are used to argue that reward structure, not density, governs bonus efficacy, including a penalty-induced convergence freeze that either bonus resolves. The paper then formalizes reward sparsity with observation-anchored reward machines, separating structural from potential sparsity, with proofs including a theorem that a structurally and potentially dense reward makes the greedy reward-machine policy optimal. A pre-registered probe study finds that the bonus changes behavior but not the decodability of task-relevant memory content.

Significance. The empirical core is substantial and unusually careful: 1,800 seeded runs (n=10 per cell), tail-mean evaluation with 95% seed-bootstrap intervals, permutation-based ANOVA on interaction terms, IQM robustness checks, per-seed tables, and a pre-registered probe study whose misses are reported rather than buried. The within-environment dissociations — aligned vs. distractor on MysteryPath (Tables 1, 19, 20) and the penalty freeze rescued by either bonus regardless of magnitude (Table 14) — are clean tests of reward structure against reward density, and the memoryless control (Table 23) is a genuine falsification test. The formal part is non-vacuous: an encoder non-existence result (Proposition 1), density certificates (Lemmas 1–2), a scheduled-stream redundancy theorem (Proposition 4), and a correct proof of Theorem 1. If the within-environment claims hold, the paper justifies evaluating exploration and memory jointly and defining sparsity by what the reward supervises. The main risk is the inferential leap from these pieces to the confounded cross-environment taxonomy and the overstated "only if" claim; with those reframed, the contribution is a solid empirical and formal study.

major comments (3)
  1. [Abstract; Experimental Setup – Environments] The abstract's headline claim — that an identical bonus signal yields three distinct interaction patterns across three environments that "vary how the content of memory is acquired" — is not identifiable from the reported design. The three environments differ simultaneously on several axes beyond the acquisition mode: observation encoding (84×84×3 pixels, 3×3×20 one-hot tensors, 6-dim vectors), action-space size (4, 7, and 4), horizon (128, 845, and 19), training budget (20M, 20M, and 10M), and learning rate (1e-4, 3e-4, and 1e-3; Tables 4–8). Because each pattern occurs in exactly one environment, the between-environment attribution is confounded: the TinyReproduce null could be a horizon-19 or action-independence artifact, and the MysteryPath amplification could track input dimensionality (Table 4) rather than unsupervised discovery. The "identical" bonus is also delivered at unequal per-step magnitudes across environments (E3B: 0.0207 vs. 0.0149; NovelD: 0.0020 vs. 0.0000; Table 14). The within-environment manipulations (MysteryPath sparse/aligned/distractor in Tables 1 and 20; the memoryless falsification in Table 23) do support the reward-structure claims, but they vary the reward target, not the acquisition mode. Please either add a matched manipulation that varies only the acquisition axis or explicitly demote the three-pattern taxonomy to an interpretive scheme and state these confounds in the abstract.
  2. [Abstract; Conclusion; Results – Reward Structure vs. Reward Density] The abstract and conclusion assert that "a dense reward neutralizes a bonus only if it directly supervises the required latent memory." The supporting evidence is the aligned-vs-distractor contrast on MysteryPath (Table 1) and the distractor arm on MemoryS13 (Table 2), and note that MemoryS13 has no aligned arm, so the "supervises the memory" cell of the dissociation exists only on MysteryPath. Within the aligned arm the neutralization is in fact incomplete: E3B degrades all five memory cells, yet NovelD leaves LSTM, RetNet, and Mamba-2 within ±0.03 of baseline and improves GatedDeltaNet by +0.15 (0.67→0.82 in Table 1). The results section presents this nuance correctly, but the abstract and conclusion drop it. Please restate the claim as a tested two-way dissociation rather than a universal, e.g., "in the tested comparisons, a dense reward that directly supervises the required memory eliminates or reverses the bonus's positive effect, whereas a density-matched reward that does not supervise it preserves the effect."
  3. [Reward Sparsity & Reward Machines; Appendix Lemmas 1–2; Discussion] The formal taxonomy is applied to reward-preserving abstractions chosen by the authors after the empirical patterns were observed, and at several points the classified object is not the object that was trained. For MemoryS13, the raw environment is classified structurally sparse (corridor view-aliasing; appendix), yet the equalization pattern is explained through the cue abstraction, which is structurally dense and potentially sparse (Lemma 1); for TinyReproduce, the redundancy result (Proposition 4) covers the dictation abstraction. The manuscript acknowledges this in the Limitations, and the Discussion does say "the abstraction admits a one-state machine," but the main-text classifications (e.g., "MysteryPath is structurally sparse, therefore amplification" and "TinyReproduce is structurally dense and potentially dense, [so] Theorem 1 accounts for this null effect") do not each carry that object tag. As written, a reader cannot tell whether Definitions 3–5 are a predictive diagnostic or a post-hoc organizing vocabulary. Please make the status explicit at each classification and, ideally, demonstrate the framework's use out of sample on one held-out task so the vocabulary has testable content.
minor comments (5)
  1. [Evaluation & Budgets; Table 8; Proposition 4 remark] Gamma inconsistency: the main text's Evaluation & Budgets says "TinyReproduce (γ=0.99)", while Table 8 and the shared defaults (Table 5) state that TinyReproduce relies on the shared γ=0.995, and the remark after Proposition 4 cites "γ=0.99 (Table 8)". Reconcile these numbers; the formal condition γ>0.9 is satisfied either way, so the results are unaffected.
  2. [Table 15 caption] Table 15's "df = 4,1,4,90" is correct only if each column is read as a two-level contrast (none vs. E3B; none vs. NovelD); add a footnote to that effect so readers do not infer a three-level bonus factor with df=2.
  3. [Abstract] The abstract's "identical bonus signal" should be qualified on first use: the two algorithms share the coefficient λ=0.03, but the delivered per-step bonus differs across environments (Table 14), so "identical" describes the algorithm, not the received magnitude.
  4. [Compute and Reproducibility] The paper promises code and per-seed records "upon publication"; for a 1,800-run suite whose central pattern claims rest on per-seed distributions (Table 25), please provide a review-time artifact (e.g., an anonymous repository link) so the per-seed records and run counts in Table 30 can be verified during review.
  5. [Figures 4 and 7 captions] Figures 4 and 7 captions contain empty cross-references ("discussed in Section ."); fill them in.

Circularity Check

0 steps flagged · score 0.0 of 10

No definitional circularity: the empirical interaction patterns are measured independently of the reward-machine taxonomy, which is introduced after the results and flagged by the authors as abstraction-dependent.

full rationale

Walking the derivation chain, the load-bearing empirical claims (amplification, equalization, null; reward-structure-not-density) come from complete Architecture x Bonus matrices reported in Tables 1-3 with per-seed data, bootstrap intervals, permutation tests, and robust aggregates. The reward-machine definitions (Definitions 1-5) and Theorem 1 are introduced after the results as an organizing vocabulary, with proofs in the appendix; they do not assume the empirical patterns and are not used to generate them. No parameter is fitted to a subset and then renamed as a prediction; no self-citation is load-bearing; no uniqueness result is imported from the authors. The paper's own limitation statement -- 'every classification applies strictly to reward-preserving abstractions' and 'the amplify/equalize distinction rests on empirical retention-burden analysis' -- is an honest caveat about the post-hoc, abstraction-dependent nature of the taxonomy, and the cross-environment comparison is indeed confounded by observation space, horizon, budget, and learning-rate overrides. These are internal-validity and generalizability concerns, not circularity: the derivation is not equivalent to its inputs by construction. Score 0 reflects the absence of definitional or fitted-input circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central empirical claims rely on several chosen reward coefficients (bonus scale, distractor, penalty, aligned progress reward) and on the comparability of three unmatched environments. The formal sparsity taxonomy is independent of these numbers but is applied under author-chosen reward-preserving abstractions. No new physical entities are introduced.

free parameters (5)
  • intrinsic bonus coefficient lambda = 0.03
    Chosen via sweep on MysteryPath E3B (Table 9) and applied uniformly to all environments and both bonuses; bracketing at 0.1 collapses performance, so the reported pattern is tied to this coefficient.
  • distractor reward epsilon = 0.1/T_max (approx 7.8e-4 on MysteryPath, 1.2e-4 on MemoryS13)
    Hand-chosen density-matched reward for re-entering known tiles; constrained to be below the discount criterion epsilon/(1-gamma) < 1 so the optimal policy is unchanged.
  • penalty per exploratory action = 1/T_max (0.008 MysteryPath, 0.0012 MemoryS13)
    Chosen to be return-matched to the sparse reward, leaving the theoretical optimum unchanged while inducing the freeze.
  • aligned reward per progress step = +0.1
    Arbitrary dense reward for memory-advancing forward progress in MysteryPath; used as the negative control for the bonus.
  • per-environment learning-rate overrides = 1e-4 baseline; 3e-4 for MemoryS13; 1e-3 for TinyReproduce
    Selected per environment without architecture-specific tuning; the shared-HP assumption is load-bearing for the architecture comparison.
assumptions (4)
  • standard math Reward machines and potential-based shaping theory (Ng et al. 1999, Toro Icarte et al. 2018) are taken as background.
    Used to define structural and potential sparsity and to prove Theorem 1.
  • domain assumption The three environments are representative of distinct memory-acquisition modes.
    The amplification, equalization, and null classification is drawn from these three specific tasks, not from a controlled variation of one environment.
  • domain assumption A single shared hyperparameter configuration is a fair comparison across architectures.
    The authors explicitly avoid architecture-specific tuning; if the shared configuration systematically disadvantages some architectures, the amplification pattern would be confounded.
  • ad hoc to paper The chosen reward-preserving abstractions (cue machine, dictation machine, one-state MysteryPath machine) capture the task-required memory.
    The sparsity classification is only valid for these abstractions, and the authors acknowledge classifications apply strictly to reward-preserving abstractions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reward Structure Shapes the Interaction Between Episodic Exploration and Neural Memory in Reinforcement Learning." pith.science (2026). https://pith.science/paper/SDLQ6PBF

@misc{pith2026260805111,
  author       = {Pith},
  title        = {Pith review of: Reward Structure Shapes the Interaction Between Episodic Exploration and Neural Memory in Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SDLQ6PBF}},
  note         = {Machine review of arXiv:2608.05111}
}
read the original abstract

In partially observable reinforcement learning, agents face a dual bottleneck: they must explore to encounter rewarding states and retain that experience in memory to optimize their policies. Exploration bonuses and memory architectures are traditionally evaluated in isolation, leaving their interaction unmeasured, and standard notions of sparse reward conflate temporal signal density with what the reward actually supervises. We present a controlled study crossing episodic exploration bonuses with diverse neural memory architectures across three environments that vary how the content of memory is acquired. An identical bonus signal yields three distinct interaction patterns: it amplifies architectural capacity differences where memory content must be actively discovered and retained unsupervised; equalizes architectures to a shared ceiling where the content, once sought out, is a single reward-supervised cue; and is null where the observation stream is purely scheduled. Controlled reward manipulations verify that these patterns track reward structure rather than density: a dense reward neutralizes a bonus only if it directly supervises the required latent memory, and a small avoidable penalty on exploratory actions (leaving the optimum unchanged) induces policy convergence to suboptimal stationary states, which either bonus resolves. We then formalize reward sparsity with observation-anchored reward machines, separating structural sparsity (an automaton reproduces the return without the task-required history) from potential sparsity (the one-step reward misprices local exploratory actions); the resulting vocabulary organizes the three regimes by the retention burden each task exposes. Together, these results show exploration and memory are complements, not substitutes: a bonus induces exposure, and only memory converts exposure into return.

Figures

Figures reproduced from arXiv: 2608.05111 by the authors.

Figure 1
Figure 1. Architectural separation of bonus and memory in [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Reward machines for the three environments. Transitions are labeled ⟨φ, c⟩, where φ denotes the label condition and c denotes the reward. The symbol o/w matches any remaining label, and uf indicates the terminal state. (a) The MysteryPath machine, featuring a single non-terminal state and issuing +1 upon reaching the visible goal. (b) The MemoryS13 machine, which stores the initially observed cue. The transition mat… view at source ↗
Figure 3
Figure 3. MysteryPath-Grid across all four reward variants (rows) [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: MiniGrid-MemoryS13 across four reward variants (rows) [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: TinyReproduce across both reward variants (rows) [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: The penalty-variant freeze on MysteryPath (median [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Per-seed tail-mean success for every cell in the suite. The bimodal nature of MemoryS13 necessitates the scale [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Probe AUC by architecture, grouped by training signal. Dashed lines mark obs-only and random-init controls. [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 10
Figure 10. Figure 10: MemoryS13 probe AUC vs. steps since cue was [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

54 extracted references · 50 canonical work pages

  1. [1]

    2025 , note =

    Lidayan, Aly and Dennis, Michael and Russell, Stuart , booktitle =. 2025 , note =

    work page 2025

  2. [2]

    Proceedings of the 41st International Conference on Machine Learning (ICML) , year =

    Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learning , author =. Proceedings of the 41st International Conference on Machine Learning (ICML) , year =

    work page

  3. [3]

    Proceedings of the 40th International Conference on Machine Learning (ICML) , year =

    Lazy Agents: A New Perspective on Solving Sparse Reward Problem in Multi-Agent Reinforcement Learning , author =. Proceedings of the 40th International Conference on Machine Learning (ICML) , year =

    work page

  4. [4]

    Proceedings of the 35th International Conference on Machine Learning (ICML) , pages =

    Using Reward Machines for High-Level Task Specification and Decomposition in Reinforcement Learning , author =. Proceedings of the 35th International Conference on Machine Learning (ICML) , pages =

    work page

  5. [5]

    Advances in Neural Information Processing Systems (NeurIPS) , volume =

    Learning Reward Machines for Partially Observable Reinforcement Learning , author =. Advances in Neural Information Processing Systems (NeurIPS) , volume =

    work page

  6. [6]

    Journal of Artificial Intelligence Research (JAIR) , volume =

    Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning , author =. Journal of Artificial Intelligence Research (JAIR) , volume =

    work page

  7. [7]

    Artificial Intelligence , year =

    Learning Reward Machines: A Study in Partially Observable Reinforcement Learning , author =. Artificial Intelligence , year =

    work page

  8. [8]

    and Valenzano, Richard and McIlraith, Sheila A

    Camacho, Alberto and Toro Icarte, Rodrigo and Klassen, Toryn Q. and Valenzano, Richard and McIlraith, Sheila A. , booktitle =

    work page

Show all 54 references
  1. [9]

    Proceedings of the 40th International Conference on Machine Learning (ICML) , year =

    Hierarchies of Reward Machines , author =. Proceedings of the 40th International Conference on Machine Learning (ICML) , year =

  2. [10]

    Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS) , pages =

    Exploration in Reward Machines with Low Regret , author =. Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS) , pages =

  3. [11]

    arXiv preprint arXiv:2312.11364 , year =

    Counting Reward Automata: Sample Efficient Reinforcement Learning Through the Exploitation of Reward Function Structure , author =. arXiv preprint arXiv:2312.11364 , year =

    arXiv

  4. [12]

    and Chen, Zizhao and Klassen, Toryn Q

    Li, Andrew C. and Chen, Zizhao and Klassen, Toryn Q. and Vaezipoor, Pashootan and Toro Icarte, Rodrigo and McIlraith, Sheila A. , booktitle =. Reward Machines for Deep

  5. [13]

    Journal of Artificial Intelligence Research (JAIR) , volume =

    Induction and Exploitation of Subgoal Automata for Reinforcement Learning , author =. Journal of Artificial Intelligence Research (JAIR) , volume =

  6. [14]

    Hasanbeig, Mohammadhosein and Jeppu, Natasha Yogananda and Abate, Alessandro and Melham, Tom and Kroening, Daniel , booktitle =

  7. [15]

    Artificial Intelligence , volume =

    Planning and Acting in Partially Observable Stochastic Domains , author =. Artificial Intelligence , volume =

  8. [16]

    Deep Recurrent Q-Learning for Partially Observable

    Hausknecht, Matthew and Stone, Peter , booktitle =. Deep Recurrent Q-Learning for Partially Observable

  9. [17]

    International Conference on Learning Representations (ICLR) , year =

    Recurrent Experience Replay in Distributed Reinforcement Learning , author =. International Conference on Learning Representations (ICLR) , year =

  10. [18]

    Proceedings of the 37th International Conference on Machine Learning (ICML) , year =

    Stabilizing Transformers for Reinforcement Learning , author =. Proceedings of the 37th International Conference on Machine Learning (ICML) , year =

  11. [19]

    International Conference on Learning Representations (ICLR) , year =

    Efficiently Modeling Long Sequences with Structured State Spaces , author =. International Conference on Learning Representations (ICLR) , year =

  12. [20]

    International Conference on Learning Representations (ICLR) , year =

    Simplified State Space Layers for Sequence Modeling , author =. International Conference on Learning Representations (ICLR) , year =

  13. [21]

    Proceedings of the 40th International Conference on Machine Learning (ICML) , year =

    Resurrecting Recurrent Neural Networks for Long Sequences , author =. Proceedings of the 40th International Conference on Machine Learning (ICML) , year =

  14. [22]

    arXiv preprint arXiv:2312.00752 , year =

    Mamba: Linear-Time Sequence Modeling with Selective State Spaces , author =. arXiv preprint arXiv:2312.00752 , year =

    arXiv

  15. [23]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Structured State Space Models for In-Context Reinforcement Learning , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  16. [24]

    Morad, Steven and Kortvelesy, Ryan and Bettini, Matteo and Liwicki, Stephan and Prorok, Amanda , booktitle =

  17. [25]

    Journal of Machine Learning Research , volume =

    Memory Gym: Towards Endless Tasks to Benchmark Memory Capabilities of Agents , author =. Journal of Machine Learning Research , volume =

  18. [26]

    2025 , eprint =

    Memory, Benchmark & Robots: A Benchmark for Solving Complex Tasks with Reinforcement Learning , author =. 2025 , eprint =

    2025

  19. [27]

    Recurrent Model-Free

    Ni, Tianwei and Eysenbach, Benjamin and Salakhutdinov, Ruslan , booktitle =. Recurrent Model-Free

  20. [28]

    When Do Transformers Shine in

    Ni, Tianwei and Ma, Michel and Eysenbach, Benjamin and Bacon, Pierre-Luc , booktitle =. When Do Transformers Shine in

  21. [29]

    Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track , year =

    Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks , author =. Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track , year =

  22. [30]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Unifying Count-Based Exploration and Intrinsic Motivation , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  23. [31]

    Proceedings of the 34th International Conference on Machine Learning (ICML) , year =

    Count-Based Exploration with Neural Density Models , author =. Proceedings of the 34th International Conference on Machine Learning (ICML) , year =

  24. [32]

    Proceedings of the 34th International Conference on Machine Learning (ICML) , year =

    Curiosity-Driven Exploration by Self-Supervised Prediction , author =. Proceedings of the 34th International Conference on Machine Learning (ICML) , year =

  25. [33]

    International Conference on Learning Representations (ICLR) , year =

    Exploration by Random Network Distillation , author =. International Conference on Learning Representations (ICLR) , year =

  26. [34]

    International Conference on Learning Representations (ICLR) , year =

    Episodic Curiosity through Reachability , author =. International Conference on Learning Representations (ICLR) , year =

  27. [35]

    International Conference on Learning Representations (ICLR) , year =

    Never Give Up: Learning Directed Exploration Strategies , author =. International Conference on Learning Representations (ICLR) , year =

  28. [36]

    Proceedings of the 37th International Conference on Machine Learning (ICML) , year =

    Agent57: Outperforming the Atari Human Benchmark , author =. Proceedings of the 37th International Conference on Machine Learning (ICML) , year =

  29. [37]

    and Tian, Yuandong , booktitle =

    Zhang, Tianjun and Xu, Huazhe and Wang, Xiaolong and Wu, Yi and Keutzer, Kurt and Gonzalez, Joseph E. and Tian, Yuandong , booktitle =

  30. [38]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Exploration via Elliptical Episodic Bonuses , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  31. [39]

    A Study of Global and Episodic Bonuses for Exploration in Contextual

    Henaff, Mika. A Study of Global and Episodic Bonuses for Exploration in Contextual. Proceedings of the 40th International Conference on Machine Learning (ICML) , year =

  32. [40]

    2025 , note =

    Yuan, Mingqi and Castanyer, Roger Creus and Li, Bo and Jin, Xin and Berseth, Glen and Zeng, Wenjun , journal =. 2025 , note =

    2025

  33. [41]

    Neural Computing and Applications , year =

    The Impact of Intrinsic Rewards on Exploration in Reinforcement Learning , author =. Neural Computing and Applications , year =

  34. [42]

    Proceedings of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS) , year =

    Potential-Based Reward Shaping For Intrinsic Motivation , author =. Proceedings of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS) , year =

  35. [43]

    Proceedings of the 42nd International Conference on Machine Learning (ICML) , year =

    [TODO-VERIFY: exact ADOPS title] , author =. Proceedings of the 42nd International Conference on Machine Learning (ICML) , year =

  36. [44]

    Proceedings of the 16th International Conference on Machine Learning (ICML) , pages =

    Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping , author =. Proceedings of the 16th International Conference on Machine Learning (ICML) , pages =

  37. [45]

    Potential-Based Shaping and

    Wiewiora, Eric , journal =. Potential-Based Shaping and

  38. [46]

    Proceedings of the 16th International Conference on Autonomous Agents and Multiagent Systems (AAMAS) , pages =

    Reward Shaping in Episodic Reinforcement Learning , author =. Proceedings of the 16th International Conference on Autonomous Agents and Multiagent Systems (AAMAS) , pages =

  39. [47]

    Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems (AAMAS) , pages =

    Dynamic Potential-Based Reward Shaping , author =. Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems (AAMAS) , pages =

  40. [48]

    Proceedings of the International Conference on Artificial Neural Networks (ICANN) , year =

    Multigrid Reinforcement Learning with Reward Shaping , author =. Proceedings of the International Conference on Artificial Neural Networks (ICANN) , year =

  41. [49]

    2025 , note =

    [TODO-VERIFY: Adamczyk et al.\ 2025 --- exact title] , author =. 2025 , note =

    2025

  42. [50]

    and De Giacomo, Giuseppe , booktitle =

    Brafman, Ronen I. and De Giacomo, Giuseppe , booktitle =. Regular Decision Processes: A Model for Non-

  43. [51]

    arXiv preprint arXiv:2309.14322 , year =

    Small-Scale Proxies for Large-Scale Transformer Training Instabilities , author =. arXiv preprint arXiv:2309.14322 , year =

    arXiv

  44. [52]

    International Conference on Machine Learning (ICML) , year =

    Repeat After Me: Transformers are Better than State Space Models at Copying , author =. International Conference on Machine Learning (ICML) , year =

  45. [53]

    2505.15040 , archivePrefix =

    Smirnov, Ivan and Gu, Shangding , year =. 2505.15040 , archivePrefix =

    arXiv

  46. [54]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Deep Reinforcement Learning at the Edge of the Statistical Precipice , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.