Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agents

T0 review · 5 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Inserting a group-wise softmax layer that confines latent features to a product of simplices makes actor-critic agents learn faster and finish stronger across humanoid, Atari, and robot benchmarks, with no loss in wall-clock speed.

desk verdict A broad, honest empirical study showing simplicial embeddings help actor-critic sample efficiency, but the headline claim overreaches: key hyperparameters are undisclosed and error bars are missing. read the letter →

arxiv 2510.13704 v2 pith:SZVWTJZ4 submitted 2025-10-15 cs.LG cs.AIcs.RO

Johan Obando-Ceron , Walter Mayor , Samuel Lavoie , Scott Fujimoto , Aaron Courville , Pablo Samuel Castro This is my paper · ORCID
classification cs.LGcs.AIcs.RO
keywords simplicialembeddingssampleefficiencyactor-criticrepresentationlearningfeaturecollapsenon-stationarityreinforcement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that simplicial embeddings (SEM)—a lightweight layer that partitions a network's hidden features into groups and applies a softmax within each group—improve both sample efficiency and final performance in modern actor-critic agents. The authors argue that SEM works by keeping representations bounded, sparse, and high-rank even when bootstrapped critics chase moving targets, thereby preventing the feature collapse that degrades learning. They support this with a formal analysis linking non-stationarity to neuron dormancy, a toy CIFAR-10 demonstration, and extensive experiments with FastTD3, FastSAC, and PPO across continuous-control and pixel-based benchmarks. If correct, SEM is a near-zero-cost architectural change that consistently accelerates learning and often raises final returns, without adding runtime overhead.

What carries the argument

Simplicial embeddings (SEM): a layer that partitions a latent vector into L groups of size V and applies a softmax with temperature τ within each group, constraining the output to a product of probability simplices Δ^(V−1) × ... × Δ^(V−1). This transformation enforces boundedness (each block sums to 1), induces sparsity through softmax competition, and promotes group-structured diversity that keeps the covariance rank from deflating, counteracting the neuron dormancy and feature collapse caused by drifting bootstrap targets.

What would settle it

Run a benchmark suite where each baseline (with and without SEM) is given the same per-task hyperparameter optimization budget, including L, V, and τ for SEM; if SEM's sample-efficiency gains shrink or vanish under matched tuning, the claim of consistent improvement collapses. Alternatively, identify a task in the claimed benchmark set where SEM with the paper's fixed configuration yields lower final performance than the baseline across multiple seeds.

Watch

Extended reading notes

Core claim

The central claim is that simplicial embeddings, applied to the actor's or critic's penultimate layer, consistently improve sample efficiency and final performance in actor-critic reinforcement learning. The mechanism identified is the stabilization of bootstrapped value learning: by preserving effective rank, bounding feature norms, and reducing critic disagreement, SEM provides more reliable gradients and prevents representation collapse under non-stationary training. The benefit is strongest when SEM is applied to the actor, and it holds across off-policy (FastTD3, FastSAC) and on-policy (PPO) methods, across humanoid control, IsaacGym, Atari, and multitask robot benchmarks, with no degra

Load-bearing premise

The comparison assumes that the SEM configuration chosen on a five-task HumanoidBench ablation subset—V=64, with L and τ unspecified—transfers fairly to all other algorithms and benchmarks, and that the baseline agents are not under-tuned relative to their SEM-augmented versions.

Editorial extensions

If this is right

  • Inserting SEM into the actor's penultimate layer should improve sample efficiency and final returns for FastTD3, FastSAC, and PPO across humanoid control, IsaacGym, Atari, and multitask robot benchmarks.
  • SEM's gains persist under data-limited settings (fewer environment instances, smaller replay buffers, smaller batch sizes) and when components such as clipped double Q-learning or distributional critics are removed.
  • SEM adds effectively zero wall-clock overhead; reported training times are comparable or slightly faster than baselines.
  • The geometric bias toward sparse, bounded, high-rank representations appears to be a general stabilizer for non-stationary learning, suggesting applicability beyond the specific algorithms and tasks tested.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mechanism is right, SEM may benefit other settings with distribution shift—such as offline RL, continual learning, or sim-to-real transfer—where representation collapse under drift is also a bottleneck.
  • The finding that the actor's penultimate layer matters most suggests that representation-level interventions targeting the policy network specifically, rather than the critic, could yield similar gains in other actor-critic variants.
  • A direct testable extension is to sweep τ and L explicitly: the paper reports V=64 as best but never states the values of L and τ used, so verifying that the fixed configuration transfers without tuning is essential.
  • The toy CIFAR-10 experiment hints that SEM's stabilizing effect is not RL-specific; it may generalize to any supervised or self-supervised setting with non-stationary targets.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes inserting simplicial embeddings (SEM), a group-wise softmax layer, into the penultimate layer of actor and critic networks in actor-critic RL. It claims that SEM consistently improves sample efficiency and final performance across FastTD3, FastTD3-SimBaV2, FastSAC, PPO, and FQL over a wide range of benchmarks, including HumanoidBench, IsaacGym, Atari-10, MT50, and OGBench, with no loss of wall-clock speed. The authors support this with representation diagnostics (effective rank, neuron dormancy, feature norms, TD error, critic disagreement), comparisons to alternative representation regularizers, and ablation studies over SEM parameters and FastTD3 design choices. The paper also includes a formal analysis in App. B linking non-stationarity to neuron dormancy and feature collapse.

Significance. If the claims hold, this is a potentially valuable contribution: a near-zero-cost architectural layer that can be dropped into existing actor-critic agents to improve both sample efficiency and final performance across continuous and discrete control. The empirical coverage is unusually broad for this type of contribution, including 28 h1hand tasks, 20 g1 tasks, IsaacGym, Atari, MT50, and offline-to-online OGBench experiments. The authors also provide several honest limitations and acknowledge the sensitivity of RL to hyperparameters. However, the strength of the stated claims currently exceeds the evidence: the exact SEM configuration is not fully reported (L and τ are missing), the main results are seed-mean curves without confidence intervals or significance tests, and the V=64 selection is made on a subset that later appears in the headline aggregate. These issues are fixable but are load-bearing for the central 'consistent improvement' claim.

major comments (5)
  1. [§2.2, Eq. (3); App. H, Tables 1, 3–5] The SEM configuration used in all experiments is not fully specified. Eq. (3) defines the temperature τ and the module parameters L and V, but no default or chosen value of L or τ appears anywhere in the main text or in the hyperparameter tables. Fig. 7 varies L and V without stating what was used for the remaining experiments. This makes the method irreproducible and leaves sensitivity to L and τ unknown. Please report the exact (L, V, τ) for each algorithm/benchmark and provide a sensitivity study for L and τ analogous to Fig. 7.
  2. [§4, Fig. 3; App. F, Fig. 18] V=64 is selected on a 5-task HumanoidBench subset (h1hand-walk, -stand, -run, -stair, -slide) and the same five tasks are included in the 28-task aggregate of Fig. 18. Thus part of the headline result is in-sample selection rather than a fixed, pre-specified configuration. The paper should either fix V (and L, τ) a priori or demonstrate that V=64 is also best on a held-out subset; otherwise the claim of 'consistent' improvement across the full suite is not independent of the selection procedure.
  3. [§5, Figs. 9–11; App. I, Figs. 15–21] All main results are presented as seed-mean curves without confidence intervals, error bars, or significance tests. Many per-task curves appear to overlap at the end of training (e.g., Fig. 18 h1hand-run and h1hand-cabinet; Fig. 19 several Atari games), so the 'final performance' part of the claim is not statistically supported. Please report per-seed curves or provide bootstrap CIs / IQM with significance testing (e.g., paired tests across seeds) for the aggregate claims.
  4. [Abstract; Table 2] The claim 'without any loss in runtime speed' is contradicted by Table 2: h1hand-walk takes 2:31 h for FastTD3 but 2:42 h with SEM (+7%), and h1hand-stair takes 4:09 h vs. 4:13 h. Other tasks are faster, so the effect is task-dependent and not quantified. Please provide repeated wall-clock measurements with variability estimates or soften the claim to 'without substantial runtime loss' or similar.
  5. [§4/§6 (Limitations); App. H] The comparison uses baseline default hyperparameters unchanged while SEM is inserted without any tuning; the Limitations themselves note that RL agents are 'notably sensitive to these choices.' Because SEM changes feature geometry and gradient scale, some gains could reflect compensating for baseline pathology rather than a general inductive bias. Fig. 8 tests robustness only on the 5-task ablation subset; please extend at least one robustness check to another benchmark (e.g., Atari or IsaacGym) or otherwise provide evidence that the baseline defaults are not disadvantaged by the comparison.
minor comments (5)
  1. [App. B, Theorem 1] The 'formal analysis' is heuristic: Eq. (5) shows a nonzero bias term under drift, but the step from that to increased neuron dormancy is not a rigorous proof. The covariance-deflation and gating arguments are plausible but should be labeled as an analytical motivation rather than a theorem.
  2. [Fig. 19 caption; main text §5] The Atari per-task figure caption says 'PPO' but the legend inside the figure panels says 'FastTD3 + SEM (Actor)'. Please verify which algorithm was used and correct the inconsistency.
  3. [Fig. 17 caption; main text §5] The IsaacGym learning-curve figure caption says 'FastSAC' while the text states 'FastTD3' for the 9 IsaacGym tasks. This makes it unclear which baseline is being reported.
  4. [Captions of Figs. 10 and 11] Typo: 'baseliens' should be 'baselines'.
  5. [§4] The phrase 'see sec 4' appears inside Section 4 itself when discussing the L/V tradeoff; this self-reference should be fixed (e.g., 'see below' or a specific figure number).

Circularity Check

0 steps flagged · score 1.0 of 10

No material circularity: the SEM improvement claim is measured against external baselines; in-sample V=64 selection and undisclosed L/tau are reproducibility/fairness caveats, not definitional reductions.

full rationale

The derivation chain for the paper's central claim is empirical rather than definitional. SEM is defined by the fixed group-wise softmax in Eq. (3); the claimed improvements are established by comparing SEM-augmented FastTD3/FastSAC/PPO against the unmodified baselines under the baselines' default hyperparameters (Section 5, Figs. 9-11, App. I/J), including external benchmarks (Atari, IsaacGym, MT50) that were not used in any ablation. The only close call is that V=64 is selected on a 5-task HumanoidBench subset ('V=64 appears most effective', Section 4) and that subset is contained in the 28-task aggregate of Fig. 18; this is an in-sample hyperparameter selection that weakens that particular aggregate and, together with the undisclosed L and tau (Limitations; hyperparameter tables), is a reproducibility/fairness concern. It is not a circular reduction: no output quantity is defined as a function of the selected hyperparameter, and the headline claim does not rest solely on the 5-task subset. Self-citations to Lavoie et al. (2023) supply the SEM module and SSL evidence, but the RL benefits are newly measured, not inferred from those citations; App. B's formal analysis concerns non-stationarity-induced dormancy and is independent of the empirical gains. 'Bounded feature norms' follows from the softmax definition, but it is offered as a mechanism after the fact, not as the predicted performance target. No step in the paper equates its conclusion to its input by construction.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim is empirical; the only hand-chosen quantities are the SEM hyperparameters V, L, τ (with V=64 selected on the ablation subset and L/τ unspecified). The mechanism narrative rests on the asserted non-stationarity→collapse link (App. B) and the transfer of the CIFAR-10 label-shuffle analogy to RL bootstrapping. No new entities are postulated; SEM is an existing component (Lavoie et al. 2023) reused here.

free parameters (3)
  • V (simplex dimension) = 64
    V=64 was selected in the §4 ablation (Fig. 3: 'V=64 appears most effective') on the same 5 HumanoidBench tasks used for the headline gains, then used across all main experiments without per-domain retuning.
  • L (number of simplices) = not stated
    Fig. 7 shows a sweep over L=1..64 with small L favored, but the L used in the main runs and ablations is never specified in the paper.
  • τ (softmax temperature) = not stated
    τ controls the sparsity of the group-wise softmax in Eq. (3); the default value is not given in any hyperparameter table (Tables 1, 3, 4, 5).
assumptions (5)
  • standard math Standard MDP/Bellman actor–critic framework (Eqs. 1–2)
    Section 2.1; unproved background used throughout the paper.
  • domain assumption Non-stationary bootstrapping causes representation collapse, which limits sample efficiency
    Section 3 + App. B; based on cited literature (Lyle, Kumar, Sokar) plus the paper's own sketch Theorem 1; load-bearing for the mechanism narrative.
  • domain assumption Shuffling CIFAR-10 labels every 20 epochs is a faithful proxy for RL bootstrap dynamics
    Section 3, Fig. 1; motivates why SEM should transfer to RL but provides no direct validation of the analogy.
  • domain assumption Diagnostics (effective rank, Gini, dormant neurons, Cramér distance) measure the claimed mechanism
    Section 4, App. G; correlational evidence underlying the 'why SEM works' story.
  • domain assumption Baseline hyperparameters from Seo et al. (2025) are directly comparable when SEM is inserted
    Section 4: 'we adopted the baseline models' default hyperparameters across all experiments'; if baselines are under-tuned, gains overstate SEM's value.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agents." pith.science (2026). https://pith.science/paper/SZVWTJZ4

@misc{pith2026251013704,
  author       = {Pith},
  title        = {Pith review of: Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SZVWTJZ4}},
  note         = {Machine review of arXiv:2510.13704}
}
read the original abstract

Recent works have proposed accelerating the wall-clock training time of actor-critic methods via the use of large-scale environment parallelization; unfortunately, these can sometimes still require large number of environment interactions to achieve a desired level of performance. Noting that well-structured representations can improve the generalization and sample efficiency of deep reinforcement learning (RL) agents, we propose the use of simplicial embeddings: lightweight representation layers that constrain embeddings to simplicial structures. This geometric inductive bias results in sparse and discrete features that stabilize critic bootstrapping and strengthen policy gradients. When applied to FastTD3, FastSAC, and PPO, simplicial embeddings consistently improve sample efficiency and final performance across a variety of continuous- and discrete-control environments, without any loss in runtime speed.

Figures

Figures reproduced from arXiv: 2510.13704 by the authors.

Figure 1
Figure 1. Training dynamics on CIFAR-10 with stationary vs. non-stationary targets. In the stationary regime (fixed targets), losses decrease smoothly, neuron dormancy and effective rank remains controlled, suggesting stable representation learning. In the non-stationary regime (targets shuffled every 20 epochs), the model exhibits higher variance in losses, increased dormant neuron rates, and reduced effective rank. The addi… view at source ↗
Figure 2
Figure 2. Actor–critic network architecture with SEM. The actor (left) and critic (middle) architec￾tures are modified with a SEM module, which partitions features into groups and applies group-wise softmax (right panel), constraining them to a product of simplices. Representation collapse under such non-stationarity poses a fundamental barrier to stable and ef￾ficient deep RL (see App. A for additional contex). Standard acto… view at source ↗
Figure 3
Figure 3. Average normalized return on 5 HumanoidBench tasks over 6 seeds. Baseline agent (blue, - -) vs. SEM variants applied to actor, critic, or both. Each curve corresponds to an embed￾ding dimension; dim= 64 (green, —) is highlighted. SEM accelerates early learning and improves asymptotic performance, with dim= 64 giving the most stable gains. 2 4 6 8 ×10 4 0.0 2.5 5.0 7.5 h1hand-walk-v0 ×10 2 Episode Return FastTD3 + SE… view at source ↗
Figures from the paper (18 more)
Figure 4
Figure 4. Figure 4: Learning and representation diagnostics on 2 HumanoidBench tasks over 6 seeds. SEM reaches high return earlier, raises actor/critic effective rank, and keeps actor features compact. The Effect of SEM on Learning Dynamics in Deep RL. We empirically evaluate the impact o…
Figure 5
Figure 5. Figure 5: Learning dynamics on 2 HumanoidBench tasks. SEM reaches high return faster, with lower losses, smaller TD error, reduced critic disagreement, and better-calibrated value estimates. 2 4 6 8 ×10 4 0.0 2.5 5.0 7.5 h1hand-walk-v0 ×10 2 Episode Return FastTD3 + SEM (Actor) …
Figure 6
Figure 6. Figure 6: Sparsity, entropy, and action std on 2 HumanoidBench tasks. SEM agents achieve higher returns with sparser features, lower entropy, and more stable action scales. 0 1 2 3 4 5 6 7 8 Environment Steps ×10 4 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Average Normalized Return CRELU Gumb…
Figure 7
Figure 7. Figure 7: Aggregated average return on 5 HumanoidBench tasks. We constrain the encoder’s out￾put of the actor. (left) SEM outperforms alternative methods to impart structure on the encoder’s output. (middle) Effect of varying L. Small L generally leads to better return given eno…
Figure 8
Figure 8. Figure 8: Effect of core design choices on FastTD3 with and without SEM on 5 HumanoidBench tasks. SEM solid green, (green, —) consistently improves sample efficiency and asymptotic return across all settings, showing robustness to both hyperparameter variation and architectural …
Figure 9
Figure 9. Figure 9: SEM on fast actor–critic algorithms. Average normalized return on HumanoidBench with FastTD3 (left), FastTD3–SimBa (middle), and FastSAC (right). SEM, solid green, (green, —), consistently improves sample efficiency and yields higher final performance across all algori…
Figure 10
Figure 10. Figure 10: Performance of PPO with and without SEM across tasks. [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]
Figure 11
Figure 11. Figure 11: Left: Offline-to-online RL results on 3 OGBench tasks (5 seeds) (Park et al., 2025a). Online fine-tuning starts at 1M steps. Right: MTBench MT50 (robotics tasks) comparing FastTD3. Applied SEM accelerates learning and improves return over the baseliens. distributions.…
Figure 12
Figure 12. Figure 12: Environment Visualizations. We evaluate SEM across three benchmark suites such as Isaac Gym, HumanoidBench, and Atari. The first two cover state-based locomotion/manipulation; Atari introduces pixel-based games of varying complexity. D.0.2 HUMANOIDBENCH In our experim…
Figure 13
Figure 13. Figure 13: Effect of core hyperparameters. SEM Actor compared to the baseline across (left) number of parallel environments, (middle) replay buffer size, and (right) batch size. SEM consistently scales better and achieves higher returns. G METRICS To better understand the dynami…
Figure 14
Figure 14. Figure 14: Robustness to design choices. SEM Actor vs. baseline across (left) clipped double Q￾learning, (middle) distributional critic (C51), and (right) exploration noise scale. SEM remains robust, while the baseline is more sensitive. of learning, including representation div…
Figure 15
Figure 15. Figure 15: Learning curves on 9 h1hand tasks. FastTD3+SimbaV2 (blue, - -) vs. + SEM (Actor) (green, —). Curves show the mean episode return across 6 seeds. Axes are independently scaled per subplot for readability. SEM (Actor) consistently accelerates learning and achieves highe…
Figure 16
Figure 16. Figure 16: Learning curves on 9 h1hand tasks. FastSAC (blue, - -) vs. + SEM (Actor) (green, —). Curves show the mean episode return across 6 seeds. Axes are independently scaled per subplot for readability. SEM (Actor) generally accelerates learning and achieves higher final ret…
Figure 17
Figure 17. Figure 17: Learning curves on 9 IsaacGym tasks. FastSAC (blue, - -) vs. + SEM (Actor) (green, —). Curves show the mean episode return across 6 seeds. Axes are independently scaled per subplot for readability. SEM (Actor) generally accelerates learning. 29 [PITH_FULL_IMAGE:figur…
Figure 18
Figure 18. Figure 18: Learning curves on 28 h1hand tasks (Sferrazza et al., 2024). FastTD3 (blue, - -) vs. + SEM (Actor) (green, —). Curves show the mean episode return across 6 seeds. Axes are independently scaled per subplot for readability. SEM (Actor) typically achieves faster learning…
Figure 19
Figure 19. Figure 19: Learning curves on Atari game (Aitchison et al., 2023). PPO (blue, - -) vs. + SEM (Actor) (green, —). Curves show the mean episode return across 3 seeds. SEM (Actor) typically achieves faster learning and equal or higher final return on most tasks. 31 [PITH_FULL_IMAG…
Figure 20
Figure 20. Figure 20: Learning curves on 16 h1 tasks (Sferrazza et al., 2024). FastTD3 (blue, - -) vs. + SEM (Actor) (green, —). Curves show the mean episode return across 6 seeds. Axes are independently scaled per subplot for readability. SEM (Actor) typically achieves faster learning and…
Figure 21
Figure 21. Figure 21: Learning curves on 20 g1 tasks (Sferrazza et al., 2024). FastTD3 (blue, - -) vs. + SEM (Actor) (green, —). Curves show the mean episode return across 6 seeds. Axes are independently scaled per subplot for readability. SEM (Actor) typically achieves faster learning and…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Stable Deep Reinforcement Learning via Isotropic Gaussian Representations

    cs.LG 2026-02 conditional novelty 5.0 of 10

    Enforcing isotropic Gaussian structure on learned representations with the SIGReg regularizer improves stability and performance in PQN and PPO across Atari and Isaac Gym.

Reference graph

Works this paper leans on

121 extracted references · 1 canonical work pages · cited by 1 Pith paper

  1. [1]

    Loss of plasticity in continual deep reinforcement learning

    Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C Machado. Loss of plasticity in continual deep reinforcement learning. In Conference on lifelong learning agents, pp.\ 620--636. PMLR, 2023

    2023

  2. [2]

    Atari-5: Distilling the arcade learning environment down to five games

    Matthew Aitchison, Penny Sweetser, and Marcus Hutter. Atari-5: Distilling the arcade learning environment down to five games. In International Conference on Machine Learning, pp.\ 421--438. PMLR, 2023

    2023

  3. [3]

    Solving rubik's cube with a robot hand

    Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with a robot hand. arXiv preprint arXiv:1910.07113, 2019

    arXiv 1910

  4. [4]

    Diffusion for world modeling: Visual details matter in atari

    Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos J Storkey, Tim Pearce, and Fran c ois Fleuret. Diffusion for world modeling: Visual details matter in atari. Advances in Neural Information Processing Systems, 37: 0 58757--58791, 2024

    2024

  5. [5]

    Unsupervised state representation learning in atari

    Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Marc-Alexandre C \^o t \'e , and R Devon Hjelm. Unsupervised state representation learning in atari. Advances in neural information processing systems, 32, 2019

    2019

  6. [6]

    Neural machine translation by jointly learning to align and translate, 2016

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate, 2016. URL https://arxiv.org/abs/1409.0473

    arXiv 2016

  7. [7]

    Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling

    Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: an evaluation platform for general agents. J. Artif. Int. Res., 47 0 (1): 0 253–279, May 2013. ISSN 1076-9757

    2013

  8. [8]

    A distributional perspective on reinforcement learning

    Marc G Bellemare, Will Dabney, and R \'e mi Munos. A distributional perspective on reinforcement learning. In International conference on machine learning, pp.\ 449--458. PMLR, 2017

    2017

Show all 121 references
  1. [9]

    Interference and generalization in temporal difference learning

    Emmanuel Bengio, Joelle Pineau, and Doina Precup. Interference and generalization in temporal difference learning. In International Conference on Machine Learning, pp.\ 767--777. PMLR, 2020

    2020

  2. [10]

    Estimating or propagating gradients through stochastic neurons for conditional computation, 2013

    Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation, 2013. URL https://arxiv.org/abs/1308.3432

    2013 arXiv

  3. [11]

    Stable gradients for stable learning at scale in deep reinforcement learning

    Roger Creus Castanyer, Johan Obando-Ceron, Lu Li, Pierre-Luc Bacon, Glen Berseth, Aaron Courville, and Pablo Samuel Castro. Stable gradients for stable learning at scale in deep reinforcement learning. arXiv preprint arXiv:2506.15544, 2025

    2025

  4. [12]

    Mico: Improved representations via sampling-based state similarity for markov decision processes

    Pablo Samuel Castro, Tyler Kastner, Prakash Panangaden, and Mark Rowland. Mico: Improved representations via sampling-based state similarity for markov decision processes. Advances in Neural Information Processing Systems, 34: 0 30113--30126, 2021

    2021

  5. [13]

    On the consistency of hyper-parameter selection in value-based deep reinforcement learning

    Johan Samir Obando Ceron, Jo \ a o Guilherme Madeira Ara \'u jo, Aaron Courville, and Pablo Samuel Castro. On the consistency of hyper-parameter selection in value-based deep reinforcement learning. In Reinforcement Learning Conference, 2024 a . URL https://openreview.net/foru...

    2024

  6. [14]

    In value-based deep reinforcement learning, a pruned network is a good network

    Johan Samir Obando Ceron, Aaron Courville, and Pablo Samuel Castro. In value-based deep reinforcement learning, a pruned network is a good network. In International Conference on Machine Learning, pp.\ 38495--38519. PMLR, 2024 b

    2024

  7. [15]

    Mixtures of experts unlock parameter scaling for deep rl

    Johan Samir Obando Ceron, Ghada Sokar, Timon Willi, Clare Lyle, Jesse Farebrother, Jakob Nicolaus Foerster, Gintare Karolina Dziugaite, Doina Precup, and Pablo Samuel Castro. Mixtures of experts unlock parameter scaling for deep rl. In International Conference on Machine Learn...

    2024

  8. [16]

    Implicit quantile networks for distributional reinforcement learning

    Will Dabney, Georg Ostrovski, David Silver, and R \'e mi Munos. Implicit quantile networks for distributional reinforcement learning. In International conference on machine learning, pp.\ 1096--1105. PMLR, 2018 a

    2018

  9. [17]

    Distributional reinforcement learning with quantile regression

    Will Dabney, Mark Rowland, Marc Bellemare, and R \'e mi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018 b

    2018

  10. [18]

    Loss of plasticity in deep continual learning

    Shibhansh Dohare, J Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A Rupam Mahmood, and Richard S Sutton. Loss of plasticity in deep continual learning. Nature, 632 0 (8026): 0 768--774, 2024

    2024

  11. [19]

    Donoho, M

    D.L. Donoho, M. Elad, and V.N. Temlyakov. Stable recovery of sparse overcomplete representations in the presence of noise. IEEE Transactions on Information Theory, 52 0 (1): 0 6--18, 2006. doi:10.1109/TIT.2005.860430

    2006

  12. [20]

    Sample-efficient reinforcement learning by breaking the replay ratio barrier

    Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Marc G Bellemare, and Aaron Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In Deep Reinforcement Learning Workshop NeurIPS 2022, 2022

    2022

  13. [21]

    Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures

    Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning,...

    2018

  14. [22]

    Seed rl: Scalable and efficient deep-rl with accelerated central inference

    Lasse Espeholt, Raphaël Marinier, Piotr Stanczyk, Ke Wang, and Marcin Michalski. Seed rl: Scalable and efficient deep-rl with accelerated central inference. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=rkgvXlrKwH

    2020

  15. [23]

    Addressing function approximation error in actor-critic methods

    Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pp.\ 1587--1596. PMLR, 2018

    2018

  16. [24]

    Off-policy deep reinforcement learning without exploration

    Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp.\ 2052--2062, 2019

  17. [25]

    For sale: State-action representation learning for deep reinforcement learning

    Scott Fujimoto, Wei-Di Chang, Edward Smith, Shixiang Shane Gu, Doina Precup, and David Meger. For sale: State-action representation learning for deep reinforcement learning. Advances in neural information processing systems, 36: 0 61573--61624, 2023

    2023

  18. [26]

    Towards general-purpose model-free reinforcement learning

    Scott Fujimoto, Pierluca D'Oro, Amy Zhang, Yuandong Tian, and Michael Rabbat. Towards general-purpose model-free reinforcement learning. In The Thirteenth International Conference on Learning Representations (ICLR), 2025. URL https://openreview.net/forum?id=R1hIXdST22

    2025

  19. [27]

    N eocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position

    Kunihiko Fukushima. N eocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36: 0 193--202, 1980

    1980

  20. [28]

    Leveraging sparse and shared feature activations for disentangled representation learning

    Marco Fumero, Florian Wenzel, Luca Zancato, Alessandro Achille, Emanuele Rodol \`a , Stefano Soatto, Bernhard Sch \"o lkopf, and Francesco Locatello. Leveraging sparse and shared feature activations for disentangled representation learning. In Thirty-seventh Conference on Neur...

    2023

  21. [29]

    Simplifying deep temporal difference learning

    Matteo Gallici, Mattie Fellows, Benjamin Ellis, Bartomeu Pou, Ivan Masmitja, Jakob Nicolaus Foerster, and Mario Martin. Simplifying deep temporal difference learning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum...

    2025

  22. [30]

    Lucas, David Abel, Prakash Panangaden, and Stefano V Albrecht

    Samuel Garcin, Trevor McInroe, Pablo Samuel Castro, Christopher G. Lucas, David Abel, Prakash Panangaden, and Stefano V Albrecht. Studying the interplay between the actor and critic representations in reinforcement learning. In The Thirteenth International Conference on Learni...

    2025

  23. [31]

    The state of sparse training in deep reinforcement learning

    Laura Graesser, Utku Evci, Erich Elsen, and Pablo Samuel Castro. The state of sparse training in deep reinforcement learning. In International Conference on Machine Learning, pp.\ 7766--7792. PMLR, 2022

    2022

  24. [32]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp.\ 1861--1870. Pmlr, 2018

    2018

  25. [33]

    Mastering atari with discrete world models

    Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In International Conference on Learning Representations, 2020

    2020

  26. [34]

    Mastering diverse domains through world models

    Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023

    2023 arXiv

  27. [35]

    Td-mpc2: Scalable, robust world models for continuous control

    Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpc2: Scalable, robust world models for continuous control. arXiv preprint arXiv:2310.16828, 2023

    2023 arXiv

  28. [36]

    Array programming with numpy

    Charles R Harris, K Jarrod Millman, St \'e fan J Van Der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J Smith, et al. Array programming with numpy. Nature, 585 0 (7825): 0 357--362, 2020

    2020

  29. [37]

    Preventing dimensional collapse in self-supervised learning via orthogonality regularization

    Junlin He, Jinxiao Du, and Wei Ma. Preventing dimensional collapse in self-supervised learning via orthogonality regularization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=Y3FjKSsfmy

    2024

  30. [38]

    Towards the systematic reporting of the energy and carbon footprints of machine learning

    Peter Henderson, Jieru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, and Joelle Pineau. Towards the systematic reporting of the energy and carbon footprints of machine learning. Journal of Machine Learning Research, 21 0 (248): 0 1--43, 2020

    2020

  31. [39]

    Learning sparse representations incrementally in deep reinforcement learning

    J Fernando Hernandez-Garcia and Richard S Sutton. Learning sparse representations incrementally in deep reinforcement learning. arXiv preprint arXiv:1912.04002, 2019

    1912 arXiv

  32. [40]

    Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms

    Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and Jo \ A G o GM Ara \ A s jo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23 0 (274): 0 ...

    2022

  33. [41]

    Matplotlib: A 2d graphics environment

    John D Hunter. Matplotlib: A 2d graphics environment. Computing in science & engineering, 9 0 (03): 0 90--95, 2007

    2007

  34. [42]

    Comparing measures of sparsity

    Niall Hurley and Scott Rickard. Comparing measures of sparsity. IEEE Transactions on Information Theory, 55 0 (10): 0 4723--4741, 2009

    2009

  35. [43]

    Categorical reparameterization with gumbel-softmax, 2017

    Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax, 2017. URL https://arxiv.org/abs/1611.01144

    2017 arXiv

  36. [44]

    Benchmarking massively parallelized multi-task reinforcement learning for robotics tasks

    Vira Joshi, Zifan Xu, Bo Liu, Peter Stone, and Amy Zhang. Benchmarking massively parallelized multi-task reinforcement learning for robotics tasks. arXiv preprint arXiv:2507.23172, 2025

    2025 arXiv

  37. [45]

    Arthur Juliani and Jordan T. Ash. A study of plasticity loss in on-policy deep reinforcement learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=MsUf8kpKTF

    2024

  38. [46]

    Jupyter Notebooks a publishing format for reproducible computational workflows

    Thomas Kluyver , Benjain Ragan-Kelley , Fernando P \'e rez , Brian Granger , Matthias Bussonnier , Jonathan Frederic , Kyle Kelley , Jessica Hamrick , Jason Grout , Sylvain Corlay , Paul Ivanov , Dami \'a n Avila , Safia Abdalla , Carol Willing , and Jupyter Development Team ....

    2016 doi

  39. [47]

    Kooi, Zhao Yang, and Vincent François-Lavet

    Jacob E. Kooi, Zhao Yang, and Vincent François-Lavet. Hadamax encoding: Elevating performance in model-free atari, 2025. URL https://arxiv.org/abs/2505.15345

    2025 arXiv

  40. [48]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. https://www. cs. toronto. edu/kriz/learning-features-2009-TR. pdf, 2009

    2009

  41. [49]

    Implicit under-parameterization inhibits data-efficient deep reinforcement learning

    Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under-parameterization inhibits data-efficient deep reinforcement learning. In International Conference on Learning Representations, 2021 a . URL https://openreview.net/forum?id=O9bnihsFfXU

    2021

  42. [50]

    DR 3: Value-based deep reinforcement learning requires explicit regularization

    Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine. DR 3: Value-based deep reinforcement learning requires explicit regularization. In Deep RL Workshop NeurIPS 2021, 2021 b . URL https://openreview.net/forum?id=LYwOCfpsQ-A

    2021

  43. [51]

    Curl: Contrastive unsupervised representations for reinforcement learning

    Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International conference on machine learning, pp.\ 5639--5650. PMLR, 2020

    2020

  44. [52]

    Simplicial embeddings in self-supervised learning and downstream classification

    Samuel Lavoie, Christos Tsirigotis, Max Schwarzer, Ankit Vani, Michael Noukhovitch, Kenji Kawaguchi, and Aaron Courville. Simplicial embeddings in self-supervised learning and downstream classification. In The Eleventh International Conference on Learning Representations, 2023

    2023

  45. [53]

    Compositional discrete latent code for high fidelity, productive diffusion models, 2025

    Samuel Lavoie, Michael Noukhovitch, and Aaron Courville. Compositional discrete latent code for high fidelity, productive diffusion models, 2025. URL https://arxiv.org/abs/2507.12318

    2025

  46. [54]

    LeCun, B

    Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural Computation, 1 0 (4): 0 541--551, 1989. doi:10.1162/neco.1989.1.4.541

    1989 doi

  47. [55]

    Wurman, Jaegul Choo, Peter Stone, and Takuma Seno

    Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim, Jun Jet Tai, Kaushik Subramanian, Peter R. Wurman, Jaegul Choo, Peter Stone, and Takuma Seno. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning. In The Thirteenth International Conference on L...

    2025

  48. [56]

    Hyperspherical normalization for scalable deep reinforcement learning

    Hojoon Lee, Youngdo Lee, Takuma Seno, Donghu Kim, Peter Stone, and Jaegul Choo. Hyperspherical normalization for scalable deep reinforcement learning. In Forty-second International Conference on Machine Learning, 2025 b . URL https://openreview.net/forum?id=kfYxyvCYQ4

    2025

  49. [57]

    State representation learning for control: An overview

    Timoth \'e e Lesort, Natalia D \' az-Rodr \' guez, Jean-Franois Goudou, and David Filliat. State representation learning for control: An overview. Neural Networks, 108: 0 379--392, 2018

    2018

  50. [58]

    Selective-supervised contrastive learning with noisy labels

    Shikun Li, Xiaobo Xia, Shiming Ge, and Tongliang Liu. Selective-supervised contrastive learning with noisy labels. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 316--325, 2022

    2022

  51. [59]

    Parallel q -learning: Scaling off-policy reinforcement learning under massively parallel simulation

    Zechu Li, Tao Chen, Zhang-Wei Hong, Anurag Ajay, and Pulkit Agrawal. Parallel q -learning: Scaling off-policy reinforcement learning under massively parallel simulation. In International Conference on Machine Learning, pp.\ 19440--19459. PMLR, 2023

    2023

  52. [60]

    Continuous control with deep reinforcement learning

    Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015

    2015 arXiv

  53. [61]

    Neuroplastic expansion in deep reinforcement learning

    Jiashun Liu, Johan Samir Obando Ceron, Aaron Courville, and Ling Pan. Neuroplastic expansion in deep reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025 a . URL https://openreview.net/forum?id=20qZK2T7fa

    2025

  54. [62]

    The courage to stop: Overcoming sunk cost fallacy in deep reinforcement learning

    Jiashun Liu, Johan Obando-Ceron, Pablo Samuel Castro, Aaron Courville, and Ling Pan. The courage to stop: Overcoming sunk cost fallacy in deep reinforcement learning. In Forty-second International Conference on Machine Learning, 2025 b . URL https://openreview.net/forum?id=VzC3BAd9gf

    2025

  55. [63]

    Measure gradients, not activations! enhancing neuronal activity in deep reinforcement learning

    Jiashun Liu, Zihao Wu, Johan Obando-Ceron, Pablo Samuel Castro, Aaron Courville, and Ling Pan. Measure gradients, not activations! enhancing neuronal activity in deep reinforcement learning. arXiv preprint arXiv:2505.24061, 2025 c

    2025

  56. [64]

    The utility of sparse representations for control in reinforcement learning

    Vincent Liu, Raksha Kumaraswamy, Lei Le, and Martha White. The utility of sparse representations for control in reinforcement learning. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intellig...

    2019 doi

  57. [65]

    Precise and dexterous robotic manipulation via human-in-the-loop reinforcement learning

    Jianlan Luo, Charles Xu, Jeffrey Wu, and Sergey Levine. Precise and dexterous robotic manipulation via human-in-the-loop reinforcement learning. Science Robotics, 10 0 (105): 0 eads5033, 2025. doi:10.1126/scirobotics.ads5033. URL https://www.science.org/doi/abs/10.1126/scirobo...

    2025 doi

  58. [66]

    Understanding and preventing capacity loss in reinforcement learning

    Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. In Deep RL Workshop NeurIPS 2021, 2021. URL https://openreview.net/forum?id=5G7fT_tJTt

    2021

  59. [67]

    Learning dynamics and generalization in deep reinforcement learning

    Clare Lyle, Mark Rowland, Will Dabney, Marta Kwiatkowska, and Yarin Gal. Learning dynamics and generalization in deep reinforcement learning. In International conference on machine learning, pp.\ 14560--14581. PMLR, 2022

    2022

  60. [68]

    Understanding plasticity in neural networks

    Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. In International Conference on Machine Learning, pp.\ 23190--23211. PMLR, 2023

    2023

  61. [69]

    Disentangling the causes of plasticity loss in neural networks

    Clare Lyle, Zeyu Zheng, Khimya Khetarpal, Hado van Hasselt, Razvan Pascanu, James Martens, and Will Dabney. Disentangling the causes of plasticity loss in neural networks. In Conference on Lifelong Learning Agents, pp.\ 750--783. PMLR, 2025

    2025

  62. [70]

    Network sparsity unlocks the scaling potential of deep reinforcement learning

    Guozheng Ma, Lu Li, Zilin Wang, Li Shen, Pierre-Luc Bacon, and Dacheng Tao. Network sparsity unlocks the scaling potential of deep reinforcement learning. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=mIomqOskaa

    2025

  63. [71]

    Maddison, Andriy Mnih, and Yee Whye Teh

    Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables, 2017. URL https://arxiv.org/abs/1611.00712

    2017 arXiv

  64. [72]

    Isaac gym: High performance gpu based physics simulation for robot learning

    Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu based physics simulation for robot learning. In Thirty-fifth Conference on Neural Info...

    2021

  65. [73]

    Rapid locomotion via reinforcement learning

    Gabriel B Margolis, Ge Yang, Kartik Paigwar, Tao Chen, and Pulkit Agrawal. Rapid locomotion via reinforcement learning. The International Journal of Robotics Research, 43 0 (4): 0 572--587, 2024

    2024

  66. [74]

    The impact of on-policy parallelized data collection on deep reinforcement learning networks

    Walter Mayor, Johan Obando-Ceron, Aaron Courville, and Pablo Samuel Castro. The impact of on-policy parallelized data collection on deep reinforcement learning networks. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=cnqyzuZhSo

    2025

  67. [75]

    Python for Data Analysis: Data Wrangling with Pandas, NumPy , and IPython

    Wes McKinney. Python for Data Analysis: Data Wrangling with Pandas, NumPy , and IPython . O'Reilly Media, 1 edition, February 2013. ISBN 9789351100065. URL http://www.amazon.com/exec/obidos/redirect?tag=citeulike07-20&path=ASIN/1449319793

    2013

  68. [76]

    Orbit: A unified simulation framework for interactive robot learning environments

    Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu, Nikita Rudin, David Hoeller, Jia Lin Yuan, Ritvik Singh, Yunrong Guo, Hammad Mazhar, et al. Orbit: A unified simulation framework for interactive robot learning environments. IEEE Robotics and Automation Letters, 8 0 (6): 0 374...

    2023

  69. [77]

    No representation, no trust: Connecting representation, collapse, and trust issues in PPO

    Skander Moalla, Andrea Miele, Daniil Pyatko, Razvan Pascanu, and Caglar Gulcehre. No representation, no trust: Connecting representation, collapse, and trust issues in PPO . In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://open...

    2024

  70. [78]

    Learning effective and interpretable semantic models using non-negative sparse embedding

    Brian Murphy, Partha Talukdar, and Tom Mitchell. Learning effective and interpretable semantic models using non-negative sparse embedding. In Martin Kay and Christian Boitet (eds.), Proceedings of COLING 2012 , pp.\ 1933--1950, Mumbai, India, December 2012. The COLING 2012 Org...

    2012

  71. [79]

    Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning

    Michal Nauman, Micha Bortkiewicz, Piotr Mi o \'s , Tomasz Trzci \'n ski, Mateusz Ostaszewski, and Marek Cygan. Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning. In Proceedings of the 41st International Conference on Machi...

    2024

  72. [80]

    Bigger, regularized, optimistic: scaling for compute and sample efficient continuous control

    Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Mi o \'s , and Marek Cygan. Bigger, regularized, optimistic: scaling for compute and sample efficient continuous control. Advances in neural information processing systems, 37: 0 113038--113071, 2024 b

    2024

  73. [81]

    Small batch deep reinforcement learning

    Johan Obando Ceron, Marc Bellemare, and Pablo Samuel Castro. Small batch deep reinforcement learning. Advances in Neural Information Processing Systems, 36: 0 26003--26024, 2023

    2023

  74. [82]

    Oliphant

    Travis E. Oliphant. Python for scientific computing. Computing in Science & Engineering, 9 0 (3): 0 10--20, 2007. doi:10.1109/MCSE.2007.58

    2007 doi

  75. [83]

    OGB ench: Benchmarking offline goal-conditioned RL

    Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. OGB ench: Benchmarking offline goal-conditioned RL . In The Thirteenth International Conference on Learning Representations, 2025 a . URL https://openreview.net/forum?id=M992mjgKzI

    2025

  76. [84]

    Flow q-learning

    Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. arXiv preprint arXiv:2502.02538, 2025 b

    2025 arXiv

  77. [85]

    Empirical design in reinforcement learning

    Andrew Patterson, Samuel Neumann, Martha White, and Adam White. Empirical design in reinforcement learning. Journal of Machine Learning Research, 25 0 (318): 0 1--63, 2024. URL http://jmlr.org/papers/v25/23-0183.html

    2024

  78. [86]

    Data-efficient deep reinforcement learning for dexterous manipulation, 2017

    Ivaylo Popov, Nicolas Heess, Timothy Lillicrap, Roland Hafner, Gabriel Barth-Maron, Matej Vecerik, Thomas Lampe, Yuval Tassa, Tom Erez, and Martin Riedmiller. Data-efficient deep reinforcement learning for dexterous manipulation, 2017. URL https://arxiv.org/abs/1704.03073

    2017 arXiv

  79. [87]

    Sutherland, and Aaron Courville

    Yi Ren, Samuel Lavoie, Mikhail Galkin, Danica J. Sutherland, and Aaron Courville. Improving compositional generalization using iterated learning and simplicial embeddings, 2023. URL https://arxiv.org/abs/2310.18777

    2023 arXiv

  80. [88]

    Learning to walk in minutes using massively parallel deep reinforcement learning

    Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In 5th Annual Conference on Robot Learning, 2021. URL https://openreview.net/forum?id=wK2fDDJ5VcF

    2021

  81. [89]

    iqrl--implicitly quantized representations for sample-efficient reinforcement learning

    Aidan Scannell, Kalle Kujanp \"a \"a , Yi Zhao, Mohammadreza Nakhaei, Arno Solin, and Joni Pajarinen. iqrl--implicitly quantized representations for sample-efficient reinforcement learning. arXiv preprint arXiv:2406.02696, 2024

    2024 arXiv

  82. [90]

    Discrete codebook world models for continuous control

    Aidan Scannell, Mohammadreza Nakhaeinezhadfard, Kalle Kujanp \"a \"a , Yi Zhao, Kevin Sebastian Luck, Arno Solin, and Joni Pajarinen. Discrete codebook world models for continuous control. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  83. [91]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  84. [92]

    Green ai

    Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. Green ai. Communications of the ACM, 63 0 (12): 0 54--63, 2020

    2020

  85. [93]

    Data-efficient reinforcement learning with self-predictive representations

    Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. In The Nineth International Conference on Learning Representations (ICLR), 2021

    2021

  86. [94]

    Bigger, better, faster: Human-level atari with human-level efficiency

    Max Schwarzer, Johan Samir Obando Ceron, Aaron Courville, Marc G Bellemare, Rishabh Agarwal, and Pablo Samuel Castro. Bigger, better, faster: Human-level atari with human-level efficiency. In International Conference on Machine Learning, pp.\ 30365--30380. PMLR, 2023

    2023

  87. [95]

    Fasttd3: Simple, fast, and capable reinforcement learning for humanoid control

    Younggyo Seo, Carmelo Sferrazza, Haoran Geng, Michal Nauman, Zhao-Heng Yin, and Pieter Abbeel. Fasttd3: Simple, fast, and capable reinforcement learning for humanoid control. arXiv preprint arXiv:2505.22642, 2025

    2025 arXiv

  88. [96]

    Humanoidbench: Simulated humanoid benchmark for whole-body locomotion and manipulation

    Carmelo Sferrazza, Dun-Ming Huang, Xingyu Lin, Youngwoon Lee, and Pieter Abbeel. Humanoidbench: Simulated humanoid benchmark for whole-body locomotion and manipulation. arXiv preprint arXiv:2403.10506, 2024

    2024 arXiv

  89. [97]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017. URL https://openreview.net...

    2017

  90. [98]

    Sapg: Split and aggregate policy gradients

    Jayesh Singla, Ananye Agarwal, and Deepak Pathak. Sapg: Split and aggregate policy gradients. In International Conference on Machine Learning, pp.\ 45759--45772. PMLR, 2024

    2024

  91. [99]

    A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning, 2022

    Laura Smith, Ilya Kostrikov, and Sergey Levine. A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning, 2022. URL https://arxiv.org/abs/2208.07860

    2022 arXiv

  92. [100]

    Mind the gap! the challenges of scale in pixel-based deep reinforcement learning

    Ghada Sokar and Pablo Samuel Castro. Mind the gap! the challenges of scale in pixel-based deep reinforcement learning. arXiv preprint arXiv:2505.17749, 2025

    2025

  93. [101]

    The dormant neuron phenomenon in deep reinforcement learning

    Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phenomenon in deep reinforcement learning. In International Conference on Machine Learning, pp.\ 32145--32168. PMLR, 2023

    2023

  94. [102]

    Don't flatten, tokenize! unlocking the key to softmoe's efficacy in deep RL

    Ghada Sokar, Johan Samir Obando Ceron, Aaron Courville, Hugo Larochelle, and Pablo Samuel Castro. Don't flatten, tokenize! unlocking the key to softmoe's efficacy in deep RL . In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview....

    2025

  95. [103]

    Domain randomization for transferring deep neural networks from simulation to the real world

    Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp.\ 23--3...

    2017

  96. [104]

    Neural discrete representation learning, 2018

    Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning, 2018. URL https://arxiv.org/abs/1711.00937

    2018 arXiv

  97. [105]

    Python reference manual

    Guido Van Rossum and Fred L Drake Jr. Python reference manual. Centrum voor Wiskunde en Informatica Amsterdam, 1995

    1995

  98. [106]

    MAD - TD : Model-augmented data stabilizes high update ratio RL

    Claas A Voelcker, Marcel Hussing, Eric Eaton, Amir massoud Farahmand, and Igor Gilitschenski. MAD - TD : Model-augmented data stabilizes high update ratio RL . In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=6RtRsg8ZV1

    2025

  99. [107]

    Piecewise linear parametrization of policies: Towards interpretable deep reinforcement learning

    Maxime Wabartha and Joelle Pineau. Piecewise linear parametrization of policies: Towards interpretable deep reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=hOMVq57Ce0

    2024

  100. [108]

    Mixture of experts in a mixture of RL settings

    Timon Willi, Johan Samir Obando Ceron, Jakob Nicolaus Foerster, Gintare Karolina Dziugaite, and Pablo Samuel Castro. Mixture of experts in a mixture of RL settings. In Reinforcement Learning Conference, 2024. URL https://openreview.net/forum?id=5FFO6RlOEm

    2024

  101. [109]

    Harnessing structures for value-based planning and reinforcement learning

    Yuzhe Yang, Guo Zhang, Zhi Xu, and Dina Katabi. Harnessing structures for value-based planning and reinforcement learning. arXiv preprint arXiv:1909.12255, 2019

    1909 arXiv

  102. [110]

    Offline RL with smooth OOD generalization in convex hull and its neighborhood

    Qingmao Yao, Zhichao Lei, Tianyuan Chen, Ziyue Yuan, Xuefan Chen, Jianxiang Liu, Faguo Wu, and Xiao Zhang. Offline RL with smooth OOD generalization in convex hull and its neighborhood. In The Thirteenth International Conference on Learning Representations, 2025. URL https://o...

    2025

  103. [111]

    Image augmentation is all you need: Regularizing deep reinforcement learning from pixels

    Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=GY6-6sTvGaf

    2021

  104. [112]

    Mastering visual continuous control: Improved data-augmented reinforcement learning

    Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=_SJ-_yyes8

    2022

  105. [113]

    Mastering atari games with limited data

    Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. Advances in neural information processing systems, 34: 0 25476--25488, 2021

    2021

  106. [114]

    Mujoco playground

    Kevin Zakka, Baruch Tabanpour, Qiayuan Liao, Mustafa Haiderbhai, Samuel Holt, Jing Yuan Luo, Arthur Allshire, Erik Frey, Koushil Sreenath, Lueder A Kahrs, et al. Mujoco playground. CoRR, 2025

    2025

  107. [115]

    A study on overfitting in deep reinforcement learning

    Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. A study on overfitting in deep reinforcement learning. arXiv preprint arXiv:1804.06893, 2018

    2018 arXiv

  108. [116]

    Robot parkour learning

    Ziwen Zhuang, Zipeng Fu, Jianren Wang, Christopher G Atkeson, S \"o ren Schwertfeger, Chelsea Finn, and Hang Zhao. Robot parkour learning. In 7th Annual Conference on Robot Learning, 2023. URL https://openreview.net/forum?id=uo937r5eTE

    2023

  109. [117]

    Gini index as sparsity measure for signal reconstruction from compressive samples

    Dornoosh Zonoobi, Ashraf A Kassim, and Yedatore V Venkatesh. Gini index as sparsity measure for signal reconstruction from compressive samples. IEEE Journal of Selected Topics in Signal Processing, 5 0 (5): 0 927--932, 2011

    2011

  110. [118]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  111. [119]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  112. [120]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  113. [121]

    One major source of instability is overestimation bias, which accumulates when bootstrapped critics reinforce overly optimistic targets

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.