Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Sparse networks, created by one random pruning before training, keep improving at model sizes where dense deep reinforcement learning networks collapse.

desk verdict A simple, well-executed empirical study showing static random sparsity lets SimBa networks scale past dense collapse, but the missing per-scale dense baselines leave the headline claim conditional. read the letter →

arxiv 2506.17204 v1 pith:D76L2KUJ submitted 2025-06-20 cs.LG cs.AI

Guozheng Ma , Lu Li , Zilin Wang , Li Shen , Pierre-Luc Bacon , Dacheng Tao This is my paper · ORCID
classification cs.LGcs.AI
keywords deepreinforcementlearningnetworkscalingstaticsparsityone-shotrandompruningplasticitylossparameterefficiencyoptimizationpathologiesSimBa
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Deep reinforcement learning (DRL) models have a scaling problem: past a certain size, larger dense networks stop improving and often get worse, because training pathologies such as plasticity loss and capacity collapse intensify with scale. The paper claims a much simpler fix than periodic reset or new architecture design: randomly remove a fixed percentage of weights once, before training, and leave the sparse topology fixed. On the state-of-the-art SimBa architecture, such sparse networks keep gaining performance at sizes where dense SimBa networks collapse, while using no more learnable parameters than the smaller dense baseline. The same benefit appears across continuous control, visual RL, streaming RL, and Atari-100k, and it requires no extra training computation and no modifications to the RL algorithm.

What carries the argument

The central object is a fixed binary mask $M_l \in \{0,1\}^{n_l \times n_{l-1}}$ per layer, produced once at initialization by random pruning at an overall sparsity level $S$, so that effective weights are $W_l^{\mathrm{eff}} = M_l \odot W_l$ throughout training. Layer-wise sparsity follows the Erdős–Rényi rule, which assigns each layer a sparsity $1 - \frac{n_{l-1} + n_l}{n_{l-1} n_l}$ for fully connected layers (with a convolutional analogue), giving larger layers proportionally more surviving connections. This static sparse topology is the entire intervention: no growing, rewiring, or resetting during training. The paper argues the mask does the work by counteracting the pathologies that dense scaling aggravates—capacity collapse, plasticity loss, unbounded parameter growth, and gradient interference—thereby yielding higher parameter efficiency and continued performance gains at large model sizes.

What would settle it

Run the dense width and depth scaling curves for SimBa on DMC Hard with an independent hyperparameter sweep (learning rate, weight decay, discount, replay ratio) at every model size. If some per-size tuning restores monotone dense improvement beyond about 17M parameters, the claim that sparsity rather than tuning unlocks scaling is refuted; if dense collapse persists across the sweep while sparse networks at the same total size keep improving, the claim is supported.

Watch

Extended reading notes

Core claim

The central discovery is that static network sparsity alone, implemented as one-shot random pruning of a fixed fraction of weights before any training, can unlock scaling potential that dense networks with the same architecture cannot reach. Concretely, keeping the SimBa architecture and its hyperparameters fixed, dense networks degrade once width exceeds about twice the baseline or depth exceeds baseline, whereas sparse versions of those larger networks continue to improve, and at equal total size sparse networks beat dense ones with the same learnable parameter count. The paper attributes this to four mechanisms: larger sparse networks retain higher effective representational rank, keep dormant-neuron ratios and gradient norms at healthy levels, constrain parameter norm growth, and maintain more orthogonal gradients across training samples. It further shows that periodic Reset, which restores plasticity in dense networks, gives sparse networks no benefit, and that the benefit transfers to visual DrQ-v2, streaming Stream AC(λ), and Atari-100k.

Load-bearing premise

The load-bearing premise is that the dense networks' performance collapse at larger sizes is a genuine scaling barrier and not an artifact of the paper holding one fixed SimBa hyperparameter set (learning rate, weight decay, discount, replay ratio) across all model sizes and sparsity levels.

Editorial extensions

If this is right

  • A single fixed random mask applied once before training is sufficient to make large DRL networks scale; no Reset, no dynamic topology, and no RL-algorithm modification is needed.
  • At equal total model size, sparse networks outperform dense networks with the same learnable parameter count, so sparsity improves parameter efficiency while allowing the nominal network to be much larger.
  • The scaling recipe follows directly: increase model size and keep high static sparsity, rather than stopping at the dense model's best size.
  • Because pruning is random and done once, it adds no per-step cost, can be applied before training starts, and is compatible with any RL algorithm or architecture.
  • The benefit is not confined to continuous control: image-based DrQ-v2, streaming Stream AC(λ), and Atari-100k all show sparse scaling gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper only tests unstructured random masks; if the benefit comes from limiting gradient interference, structured or block-sparse masks might give similar gains with better hardware utilization—the paper does not test this combination.
  • The Reset diagnostic suggests sparsity could replace Reset-style interventions in large-scale systems; an implicit consequence is that distributed training pipelines that currently schedule resets might be simplified, though the paper only studies single-agent benchmarks.
  • Since the mask is fixed before any data is seen, the recipe could transfer to settings with scarce or non-stationary data, such as continual RL; the streaming RL experiments are an early hint in that direction.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper asks whether static network sparsity alone, implemented by one-shot random pruning at initialization, can unlock further scaling of deep RL networks beyond the dense SimBa architecture. On six DMC Hard tasks with SAC and DDPG, it reports that dense SimBa networks degrade when scaled in width or depth beyond the default size, while ER-pruned sparse networks at the same or larger total model sizes keep improving, especially when learnable parameter counts are matched to the dense optimum. The paper then analyzes Srank, dormant ratio, gradient norms, parameter norms, simplicity bias, and gradient covariance to argue that sparsity mitigates capacity collapse, plasticity loss, unbounded parameter growth, and gradient interference. It extends the evaluation to visual RL with DrQ-v2, streaming RL with Stream AC(λ), and Atari-100k with DER. The headline comparison is a direct empirical scaling study and does not depend on fitted parameters.

Significance. If the comparison holds, this is a practical and falsifiable finding: a fixed random mask applied before training can push the scaling frontier of actor-critic RL beyond current state-of-the-art architectures without resets, dynamic sparsity, or optimizer changes. The paper's strengths are its breadth (six DMC tasks, two algorithms, visual and streaming setups, five to eight seeds), the public code release, and the use of standard diagnostic metrics. The main risk is interpretive: all configurations share one small-model hyperparameter recipe, so the dense scaling barrier and the sparsity advantage are only established under that fixed recipe. A dense baseline re-tuned per scale is the missing control that determines whether the title-level claim is supportable.

major comments (3)
  1. [Section 3.1 / Appendix B.1 (Tables 2-3), Figure 2] The dense scaling barrier is established only under the SimBa hyperparameter recipe (learning rate 1e-4, weight decay 1e-2, replay ratio 2, and the TD-MPC2 discount heuristic) applied unchanged to every model size from about 4.5M to 208M parameters. No per-scale tuning, or even a small grid over learning rate and weight decay, is reported for the dense 109M and 208M baselines. Figures 5 and 7 show that large dense networks suffer rising dormant ratios and unbounded parameter norms, but those are exactly the symptoms one expects when a learning rate and weight decay tuned for a 17M network are applied to a much larger one. Because the central claim is that sparsity, rather than hyperparameter choice, unlocks scaling, this missing control is load-bearing. Please add per-scale hyperparameter searches for the large dense baselines and report whether any dense configuration closes the gap; if the gap persists, state explicitly that the comparison is a no-tuning comparison and scope the conclusion accordingly.
  2. [Section 3.1 / Figures 1-2] The sparsity levels used for the sparse curves in Figures 1 and 2 are not stated in the main text. Section 3.1 says the sparse configurations maintain the same learnable parameter count as the optimal dense model while increasing total model size, but it does not report the sparsity ratio used at each width and depth scale. Figure 7 shows one 208M point at 92% sparsity, while Figure 3 sweeps sparsity from 0.1 to 0.9 on a 109M network, so the exact relationship between total size, sparsity, and learnable parameters is not transparent. Please provide a table of (width scale, depth scale, total parameters, sparsity, learnable parameters) for every point in the scaling plots; this is needed to verify the parameter-efficiency claim and to make the experiments reproducible.
  3. [Section 4.1 / Section 4.4 / Figure 9] Two diagnostic claims used to support the mechanism analysis are not quantitatively specified. The Srank metric in Section 4.1 is defined with an unspecified threshold tau and an unspecified sample size d, and the claimed theoretical upper bound of 256 is stated without derivation. In Section 4.4, the gradient-interference analysis reports only a qualitative covariance-matrix figure, with the number of sampled points k in Eq. (2) not given and no numerical summary of the correlation strength before and after training. Please report concrete values for tau, d, and k, and add a quantitative measure (for example, mean absolute off-diagonal covariance) so the claims of 'capacity collapse' and 'significantly weaker correlations' can be checked.
minor comments (5)
  1. [Figure 1 caption / Figure 3 caption] The sizes of the baseline networks are inconsistent: Figure 1 refers to a 17M baseline SimBa network while the Figure 3 caption and Section 3 describe the default SimBa network as about 4.5M parameters; please reconcile these numbers.
  2. [Section 2.2 / Appendix B.1] The masking procedure is not fully specified: it is unclear whether masks are applied to biases, layer-normalization gains, or convolutional kernels, and how the ER layer-wise sparsity is computed for residual blocks; please state which tensors are pruned and how learnable parameters are counted.
  3. [Section 4.1 / Figure 4] Please specify the Srank threshold tau and the number of samples d used to form the feature matrix F; without these values the numerical Srank results and the claimed upper bound of 256 cannot be reproduced.
  4. [Appendix C.3 / Figure 18] The Atari-100k experiment does not report the number of seeds or show error bars, despite being used as evidence that the benefits generalize to discrete-action domains; please add this information.
  5. [Figure 7] The legend in Figure 7 lists the large sparse network entry twice; please remove the duplicate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the scaling benefit is a direct empirical measurement, not a fitted or self-cited result.

full rationale

The paper's central claim is an empirical comparison: one-shot random pruning fixes binary masks at initialization (W_eff = M ⊙ W), and then dense vs sparse SimBa networks are trained with the same external hyperparameters and their episode returns are measured. No parameter appearing in the scaling result is fitted from that result, and no metric used in the mechanism analysis (Srank, dormant ratio, simplicity bias score, gradient covariance) is constructed from the episode-return outcome; each is a standard diagnostic imported from prior work (Kumar et al., 2021; Sokar et al., 2023; Lee et al., 2024; Lyle et al., 2023). The only self-citations are Ma et al. (2024), used to choose the visual-RL protocol (scale critic only) and to note that actor plasticity is minimally affected; these choices do not force the observed sparse-vs-dense gap. The fixed-hyperparameter concern in the appendix (learning rate 1e-4, weight decay 1e-2, replay ratio 2 held across sizes; Tables 2-3) is a missing control that threatens whether the dense scaling barrier is fundamental, but it is a correctness/external-validity issue, not circularity, because the reported comparisons still train and measure the stated networks rather than re-deriving the outcome from the input. I therefore exhibit no equation or fitted quantity that reduces the scaling prediction to its inputs.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim is empirical and does not rely on fitted constants, but it does rely on design choices: the sparsity ratio is hand-chosen or swept, the ER allocation is inherited from prior work, and the mechanism analysis uses metrics whose validity is assumed. No new entities are introduced.

free parameters (2)
  • Network sparsity ratio S = 0.1 to 0.9 swept; specific values chosen by parameter-count matching (for example 0.92 for a 208M total network to…
    The central comparisons depend on choosing sparsity levels. In Figures 1 and 2 sparsity is set so the sparse network has the same learnable parameter count as the reference dense model; in Figure 3 and broader setups it is a swept axis. These are experimental design choices, not fitted constants, but the claim depends on them.
  • Srank threshold tau = not reported
    Section 4.1 defines Srank with an indicator function I(sigma_j > tau) but does not report the value of tau. The capacity-collapse comparison depends qualitatively on this threshold, though it is diagnostic rather than load-bearing for the central scaling claim.
assumptions (3)
  • domain assumption SimBa hyperparameters transfer unchanged to all model sizes and sparsity levels.
    Tables 2 and 3 fix learning rate, weight decay, discount, and replay ratio across all configurations. The dense scaling collapse and sparse gains are measured under a single recipe, and dense networks were not re-tuned per scale.
  • domain assumption ER-based layer-wise sparsity allocation is appropriate for all tested tasks and architectures.
    Section 2.2 adopts Erdős-Rényi ratios based on prior work (Mocanu et al. 2018, Evci et al. 2020, Liu et al. 2022) without task-specific validation. If uniform sparsity were used, the scaling benefits might change.
  • domain assumption The diagnostic metrics (dormant ratio, gradient norm, Srank, simplicity bias score, gradient covariance) are valid proxies for the optimization pathologies discussed.
    Section 4 uses these metrics to attribute performance gains to plasticity preservation, capacity retention, regularization, and reduced gradient interference. Their validity is inherited from prior work and not independently established here.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning." pith.science (2026). https://pith.science/paper/D76L2KUJ

@misc{pith2026250617204,
  author       = {Pith},
  title        = {Pith review of: Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D76L2KUJ}},
  note         = {Machine review of arXiv:2506.17204}
}
read the original abstract

Effectively scaling up deep reinforcement learning models has proven notoriously difficult due to network pathologies during training, motivating various targeted interventions such as periodic reset and architectural advances such as layer normalization. Instead of pursuing more complex modifications, we show that introducing static network sparsity alone can unlock further scaling potential beyond their dense counterparts with state-of-the-art architectures. This is achieved through simple one-shot random pruning, where a predetermined percentage of network weights are randomly removed once before training. Our analysis reveals that, in contrast to naively scaling up dense DRL networks, such sparse networks achieve both higher parameter efficiency for network expressivity and stronger resistance to optimization challenges like plasticity loss and gradient interference. We further extend our evaluation to visual and streaming RL scenarios, demonstrating the consistent benefits of network sparsity.

Figures

Figures reproduced from arXiv: 2506.17204 by the authors.

Figure 1
Figure 1. Model scaling trends of ⋆⋆⋆sparse versus •••dense networks on four hardest DMC tasks using SimBa architecture with SAC and DDPG. Beyond a ∼17M baseline SimBa network, dense networks (dashed lines) exhibit degrading performance with increased scale. In contrast, introducing sparsity while increasing model size (solid lines) can unlock further scaling potential. 2022; Sokar et al., 2023), parameter under-utilization (… view at source ↗
Figure 2
Figure 2. Network scaling experiments comparing dense and sparse SimBa architectures trained with SAC and DDPG on DMC Hard tasks. Results demonstrate that appropriate sparsity enables effective model scaling while preserving parameter efficiency. is to investigate whether sparsity can extend DRL scaling boundaries beyond current dense model limitations, we es￾tablish the following experimental protocol to ensure fair and syst… view at source ↗
Figure 3
Figure 3. Scaling via network sparsity on four hardest DMC tasks using SAC and DDPG. For both default SimBa networks (∼ 4.5M parameters, blue lines) and large networks (∼ 109M parameters, orange lines), we systematically explore sparsity ratios from 0.1 to 0.9, with steps of 0.1. Results demonstrate that while default networks suffer from high sparsity, large networks consistently benefit from increased sparsity ratios, highl… view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Analysis of network representation capacity via Srank metric on Humanoid Run using SAC. Network configurations match [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Plasticity measurements of three representative network configurations in the two most challenging tasks with SAC and DDPG. Despite employing the advanced SimBa architecture, large dense critic networks still suffer from rising neuron dormancy and gradient collapse as …
Figure 6
Figure 6. Figure 6: Reset diagnostic comparison for large dense networks and large sparse networks. Despite dense networks relying on Reset operations to recover plasticity, sparse networks maintain learning capability naturally without such remedial interventions. the strength of learnin…
Figure 8
Figure 8. Figure 8: Simplicity bias scores and performance improvements across different sparsity ratios, where performance gains are aver￾aged over large networks in eight scenarios from [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 7
Figure 7. Figure 7 [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 12
Figure 12. Figure 12: Streaming RL network scaling performance across spar￾sity ratios and widths, averaged over 5 seeds after 2M steps. 0.0 0.5 1.0 1.5 2.0 Environment Steps 1e6 0 1k 2k 3k 4k 5k Episode Return Ant-v4 (Width=512) 0.0 0.5 1.0 1.5 2.0 Environment Steps 1e6 0 1k 2k 3k Walker2…
Figure 13
Figure 13. Figure 13: Comparing static sparsity (fixed pruned weights) and SparseInit (trainable zero-initialized weights) in streaming RL. training, SparseInit only zeroes weights at initialization, al￾lowing gradient updates to gradually reduce sparsity during training. Beyond showing th…
Figure 11
Figure 11. Figure 11: Learning curves and critic plasticity for Quadruped Run under varying sparsity levels with a 4x wider critic network. Results in [PITH_FULL_IMAGE:figures/full_fig_p008_11.png]
Figure 14
Figure 14. Figure 14: Model scaling trends of ⋆⋆⋆sparse versus •••dense networks on four hardest DMC tasks using SimBa architecture with SAC and DDPG. The data points in this figure are identical to those shown in [PITH_FULL_IMAGE:figures/full_fig_p015_14.png]
Figure 15
Figure 15. Figure 15: Width scaling experiments comparing dense and sparse networks across all DMC Hard tasks. Results show episode returns for both SAC (top two rows) and DDPG (bottom two rows) implementations on six challenging control tasks. Each data point represents the mean performan…
Figure 16
Figure 16. Figure 16: Depth scaling experiments comparing dense and sparse networks across all DMC Hard tasks. Results show episode returns for both SAC (top two rows) and DDPG (bottom two rows) implementations on six challenging control tasks. Each data point represents the mean performan…
Figure 17
Figure 17. Figure 17: Learning curves of Stream AC(λ) agent on Ant-v4 and Walker2d-v4, evaluated across varying sparsity levels and network widths for both the actor and critic networks. C.3. Atari-100k We conducted Atari experiments on the Atari-100k benchmark (Kaiser et al., 2020), where…
Figure 18
Figure 18. Figure 18: Performance improvements on Atari-100k benchmark when scaling network width (3x default size) with different sparsity levels using Data Efficient Rainbow (DER). Each bar represents the percentage improvement relative to the default dense network configuration. The res…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback

    cs.AI 2026-07 conditional novelty 5.0 of 10

    LEMUR jointly learns a separate reward model for each teacher's preferences and uses them to train a population of multi-objective policies, beating baselines that merge feedback into one reward.

Reference graph

Works this paper leans on

62 extracted references · 28 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Abbas, Z., Zhao, R., Modayil, J., White, A., and Machado, M. C. Loss of plasticity in continual deep reinforcement learning. In Conference on Lifelong Learning Agents, pp.\ 620--636. PMLR, 2023

    2023

  3. [3]

    Y., Ohib, R., Plis, S., and Precup, D

    Arnob, S. Y., Ohib, R., Plis, S., and Precup, D. Single-shot pruning for offline reinforcement learning. arXiv preprint arXiv:2112.15579, 2021

    arXiv 2021

  4. [4]

    Y., Ohib, R., Plis, S

    Arnob, S. Y., Ohib, R., Plis, S. M., Zhang, A., Sordoni, A., and Precup, D. Efficient reinforcement learning by discovering neural pathways. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=WEoOreP0n5

    work page 2024

  5. [5]

    Interference and generalization in temporal difference learning

    Bengio, E., Pineau, J., and Precup, D. Interference and generalization in temporal difference learning. In International Conference on Machine Learning, pp.\ 767--777. PMLR, 2020

    work page 2020

  6. [6]

    Simplicity bias in overparameterized machine learning

    Berchenko, Y. Simplicity bias in overparameterized machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 11052--11060, 2024

    2024

  7. [7]

    Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity

    Bhatt, A., Palenicek, D., Belousov, B., Argus, M., Amiranashvili, A., Brox, T., and Peters, J. Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=PczQtTsTIX

    work page 2024

  8. [8]

    P., and Weinberger, K

    Bjorck, N., Gomes, C. P., and Weinberger, K. Q. Towards deeper deep reinforcement learning with spectral normalization. Advances in neural information processing systems, 34: 0 8242--8255, 2021

    work page 2021

Show all 62 references
  1. [9]

    S., Moitra, S., Gelada, C., Kumar, S., and Bellemare, M

    Castro, P. S., Moitra, S., Gelada, C., Kumar, S., and Bellemare, M. G. Dopamine: A R esearch F ramework for D eep R einforcement L earning. 2018. URL http://arxiv.org/abs/1812.06110

    2018 arXiv

  2. [10]

    Ceron, J. S. O., Courville, A., and Castro, P. S. In value-based deep reinforcement learning, a pruned network is a good network. In Forty-first International Conference on Machine Learning, 2024 a . URL https://openreview.net/forum?id=seo9V9QRZp

    2024

  3. [11]

    Ceron, J. S. O., Sokar, G., Willi, T., Lyle, C., Farebrother, J., Foerster, J. N., Dziugaite, G. K., Precup, D., and Castro, P. S. Mixtures of experts unlock parameter scaling for deep RL . In Forty-first International Conference on Machine Learning, 2024 b . URL https://openr...

    2024

  4. [12]

    Better exploration with optimistic actor critic

    Ciosek, K., Vuong, Q., Loftin, R., and Hofmann, K. Better exploration with optimistic actor critic. Advances in Neural Information Processing Systems, 32, 2019

    2019

  5. [13]

    F., Lan, Q., Rahman, P., Mahmood, A

    Dohare, S., Hernandez-Garcia, J. F., Lan, Q., Rahman, P., Mahmood, A. R., and Sutton, R. S. Loss of plasticity in deep continual learning. Nature, 632 0 (8026): 0 768–774, August 2024. ISSN 1476-4687. doi:10.1038/s41586-024-07711-7. URL http://dx.doi.org/10.1038/s41586-024-07711-7

    2024 doi

  6. [14]

    Elsayed, M., Vasan, G., and Mahmood, A. R. Streaming deep reinforcement learning finally works. arXiv preprint arXiv:2410.14606, 2024

    2024 arXiv

  7. [15]

    Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures

    Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning, pp.\ 1407--1...

    2018

  8. [16]

    S., and Elsen, E

    Evci, U., Gale, T., Menick, J., Castro, P. S., and Elsen, E. Rigging the lottery: Making all tickets winners. In International conference on machine learning, pp.\ 2943--2952. PMLR, 2020

    2020

  9. [17]

    A., Chebotar, Y., Xiao, T., Irpan, A., Levine, S., Castro, P

    Farebrother, J., Orbay, J., Vuong, Q., Ta \" ga, A. A., Chebotar, Y., Xiao, T., Irpan, A., Levine, S., Castro, P. S., Faust, A., et al. Stop regressing: Training value functions via classification for scalable deep rl. arXiv preprint arXiv:2403.03950, 2024

    2024 arXiv

  10. [18]

    Addressing function approximation error in actor-critic methods

    Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pp.\ 1587--1596. PMLR, 2018

    2018

  11. [19]

    J., Gu, S

    Fujimoto, S., Chang, W.-D., Smith, E. J., Gu, S. S., Precup, D., and Meger, D. For SALE : State-action representation learning for deep reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=xZvGrzRq17

    2023

  12. [20]

    D., Lu, C., Jackson, M

    Goldie, A. D., Lu, C., Jackson, M. T., Whiteson, S., and Foerster, J. N. Can learned optimization make reinforcement learning less difficult? arXiv preprint arXiv:2407.07082, 2024

    2024 arXiv

  13. [21]

    Graesser, L., Evci, U., Elsen, E., and Castro, P. S. The state of sparse training in deep reinforcement learning. In International Conference on Machine Learning, pp.\ 7766--7792. PMLR, 2022

    2022

  14. [22]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp.\ 1861--1870. PMLR, 2018

    2018

  15. [23]

    Td-mpc2: Scalable, robust world models for continuous control

    Hansen, N., Su, H., and Wang, X. Td-mpc2: Scalable, robust world models for continuous control. arXiv preprint arXiv:2310.16828, 2023

    2023 arXiv

  16. [24]

    Rainbow: Combining improvements in deep reinforcement learning

    Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., and Silver, D. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  17. [25]

    N., Liu, S., Marculescu, R., and Wang, Z

    Hoang, D. N., Liu, S., Marculescu, R., and Wang, Z. REVISITING PRUNING AT INITIALIZATION THROUGH THE LENS OF RAMANUJAN GRAPH . In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=uVcDssQff_

    2023

  18. [26]

    and Ash, J

    Juliani, A. and Ash, J. T. A study of plasticity loss in on-policy deep reinforcement learning. arXiv preprint arXiv:2405.19153, 2024

    2024 arXiv

  19. [27]

    H., Czechowski, K., Erhan, D., Finn, C., Kozakowski, P., Levine, S., et al

    Kaiser, ., Babaeizadeh, M., Mi os, P., Osi \'n ski, B., Campbell, R. H., Czechowski, K., Erhan, D., Finn, C., Kozakowski, P., Levine, S., et al. Model based reinforcement learning for atari. In International Conference on Learning Representations, 2020

    2020

  20. [28]

    Plasticity loss in deep reinforcement learning: A survey

    Klein, T., Miklautz, L., Sidak, K., Plant, C., and Tschiatschek, S. Plasticity loss in deep reinforcement learning: A survey. arXiv preprint arXiv:2411.04832, 2024

    2024 arXiv

  21. [29]

    Implicit under-parameterization inhibits data-efficient deep reinforcement learning

    Kumar, A., Agarwal, R., Ghosh, D., and Levine, S. Implicit under-parameterization inhibits data-efficient deep reinforcement learning. In International Conference on Learning Representations, 2021

    2021

  22. [30]

    Plastic: Improving input and label plasticity for sample efficient reinforcement learning

    Lee, H., Cho, H., Kim, H., Gwak, D., Kim, J., Choo, J., Yun, S.-Y., and Yun, C. Plastic: Improving input and label plasticity for sample efficient reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  23. [31]

    J., Subramanian, K., Wurman, P

    Lee, H., Hwang, D., Kim, D., Kim, H., Tai, J. J., Subramanian, K., Wurman, P. R., Choo, J., Stone, P., and Seno, T. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning. arXiv preprint arXiv:2410.09754, 2024

    arXiv

  24. [32]

    SNIP : SINGLE - SHOT NETWORK PRUNING BASED ON CONNECTION SENSITIVITY

    Lee, N., Ajanthan, T., and Torr, P. SNIP : SINGLE - SHOT NETWORK PRUNING BASED ON CONNECTION SENSITIVITY . In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=B1VZqjAcYX

    2019

  25. [33]

    R., and Hinton, G

    Lei Ba, J., Kiros, J. R., and Hinton, G. E. Layer normalization. ArXiv e-prints, pp.\ arXiv--1607, 2016

    2016

  26. [34]

    Lewandowski, A., Tanaka, H., Schuurmans, D., and Machado, M. C. Directions of curvature as an explanation for loss of plasticity. Preprint at https://arxiv. org/abs/2312.00246, 2024

    2024 arXiv

  27. [35]

    Continuous control with deep reinforcement learning

    Lillicrap, T. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015

    2015 arXiv

  28. [36]

    Neuroplastic expansion in deep reinforcement learning

    Liu, J., Obando-Ceron, J., Courville, A., and Pan, L. Neuroplastic expansion in deep reinforcement learning. arXiv preprint arXiv:2410.07994, 2024

    2024 arXiv

  29. [37]

    C., Wang, Z., and Pechenizkiy, M

    Liu, S., Chen, T., Chen, X., Shen, L., Mocanu, D. C., Wang, Z., and Pechenizkiy, M. The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training. In International Conference on Learning Representations, 2022. URL https://openreview.ne...

    2022

  30. [38]

    Understanding and preventing capacity loss in reinforcement learning

    Lyle, C., Rowland, M., and Dabney, W. Understanding and preventing capacity loss in reinforcement learning. arXiv preprint arXiv:2204.09560, 2022 a

    2022 arXiv

  31. [39]

    Learning dynamics and generalization in deep reinforcement learning

    Lyle, C., Rowland, M., Dabney, W., Kwiatkowska, M., and Gal, Y. Learning dynamics and generalization in deep reinforcement learning. In International Conference on Machine Learning, pp.\ 14560--14581. PMLR, 2022 b

    2022

  32. [40]

    A., Pascanu, R., and Dabney, W

    Lyle, C., Zheng, Z., Nikishin, E., Pires, B. A., Pascanu, R., and Dabney, W. Understanding plasticity in neural networks. In International Conference on Machine Learning, pp.\ 23190--23211. PMLR, 2023

    2023

  33. [41]

    Normalization and effective learning rates in reinforcement learning

    Lyle, C., Zheng, Z., Khetarpal, K., Martens, J., van Hasselt, H., Pascanu, R., and Dabney, W. Normalization and effective learning rates in reinforcement learning. arXiv preprint arXiv:2407.01800, 2024 a

    2024 arXiv

  34. [42]

    Disentangling the causes of plasticity loss in neural networks

    Lyle, C., Zheng, Z., Khetarpal, K., van Hasselt, H., Pascanu, R., Martens, J., and Dabney, W. Disentangling the causes of plasticity loss in neural networks. arXiv preprint arXiv:2402.18762, 2024 b

    2024 arXiv

  35. [43]

    Revisiting plasticity in visual reinforcement learning: Data, modules and training stages

    Ma, G., Li, L., Zhang, S., Liu, Z., Wang, Z., Chen, Y., Shen, L., Wang, X., and Tao, D. Revisiting plasticity in visual reinforcement learning: Data, modules and training stages. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview....

    2024

  36. [44]

    C., Mocanu, E., Stone, P., Nguyen, P

    Mocanu, D. C., Mocanu, E., Stone, P., Nguyen, P. H., Gibescu, M., and Liotta, A. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9 0 (1): 0 2383, 2018

    2018

  37. [45]

    Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning

    Nauman, M., Bortkiewicz, M., Mi o \'s , P., Trzcinski, T., Ostaszewski, M., and Cygan, M. Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning. In Forty-first International Conference on Machine Learning, 2024 a . URL https:/...

    2024

  38. [46]

    Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control

    Nauman, M., Ostaszewski, M., Jankowski, K., Miłoś, P., and Cygan, M. Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control. In Advances in Neural Information Processing Systems, 2024 b

    2024

  39. [47]

    Parameter, experience, and compute efficient deep reinforcement learning

    Nikishin, E. Parameter, experience, and compute efficient deep reinforcement learning. PhD thesis, Université de Montréal, 2024

    2024

  40. [48]

    The primacy bias in deep reinforcement learning

    Nikishin, E., Schwarzer, M., D’Oro, P., Bacon, P.-L., and Courville, A. The primacy bias in deep reinforcement learning. In International conference on machine learning, pp.\ 16828--16847. PMLR, 2022

    2022

  41. [49]

    R., Mustafa, B., Renggli, C., Pinto, A

    Puigcerver, J., Ruiz, C. R., Mustafa, B., Renggli, C., Pinto, A. S., Gelly, S., Keysers, D., and Houlsby, N. Scalable transfer learning with expert models. In International Conference on Learning Representations, 2020

    2020

  42. [50]

    R., Mustafa, B., and Houlsby, N

    Puigcerver, J., Ruiz, C. R., Mustafa, B., and Houlsby, N. From sparse to soft mixtures of experts. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=jxpsAj7ltE

    2024

  43. [51]

    Schwarzer, M., Ceron, J. S. O., Courville, A., Bellemare, M. G., Agarwal, R., and Castro, P. S. Bigger, better, faster: Human-level atari with human-level efficiency. In International Conference on Machine Learning, pp.\ 30365--30380. PMLR, 2023

    2023

  44. [52]

    The pitfalls of simplicity bias in neural networks

    Shah, H., Tamuly, K., Raghunathan, A., Jain, P., and Netrapalli, P. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33: 0 9573--9585, 2020

    2020

  45. [53]

    C., Pechenizkiy, M., and Stone, P

    Sokar, G., Mocanu, E., Mocanu, D. C., Pechenizkiy, M., and Stone, P. Dynamic sparse training for deep reinforcement learning. arXiv preprint arXiv:2106.04217, 2021

    2021 arXiv

  46. [54]

    S., and Evci, U

    Sokar, G., Agarwal, R., Castro, P. S., and Evci, U. The dormant neuron phenomenon in deep reinforcement learning. In International Conference on Machine Learning, pp.\ 32145--32168. PMLR, 2023

    2023

  47. [55]

    RL x2: Training a sparse deep reinforcement learning model from scratch

    Tan, Y., Hu, P., Pan, L., Huang, J., and Huang, L. RL x2: Training a sparse deep reinforcement learning model from scratch. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=DJEEqoAq7to

    2023

  48. [56]

    Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018

    2018 arXiv

  49. [57]

    Mujoco: A physics engine for model-based control

    Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp.\ 5026--5033. IEEE, 2012

    2012

  50. [58]

    P., Hessel, M., and Aslanides, J

    Van Hasselt, H. P., Hessel, M., and Aslanides, J. When to use parametric models in reinforcement learning? Advances in Neural Information Processing Systems, 32, 2019

    2019

  51. [59]

    Vasan, G., Elsayed, M., Azimi, A., He, J., Shariar, F., Bellinger, C., White, M., and Mahmood, A. R. Deep policy gradient methods without batch updates, target networks, or replay buffers. arXiv preprint arXiv:2411.15370, 2024

    2024 arXiv

  52. [60]

    A., Lange, R

    Vischer, M. A., Lange, R. T., and Sprekeler, H. On lottery tickets and minimal task representations in deep reinforcement learning. arXiv preprint arXiv:2105.01648, 2021

    2021 arXiv

  53. [61]

    D., Huang, F., and Xu, H

    Xu, G., Zheng, R., Liang, Y., Wang, X., Yuan, Z., Ji, T., Luo, Y., Liu, X., Yuan, J., Hua, P., Li, S., Ze, Y., III, H. D., Huang, F., and Xu, H. Drm: Mastering visual reinforcement learning through dormant ratio minimization. In The Twelfth International Conference on Learning...

    2024

  54. [62]

    Mastering visual continuous control: Improved data-augmented reinforcement learning

    Yarats, D., Fergus, R., Lazaric, A., and Pinto, L. Mastering visual continuous control: Improved data-augmented reinforcement learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=_SJ-_yyes8

    2022

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.