REVIEW 4 major objections 5 minor 13 references
On the Interplay Between Sparsity and Training in Deep Reinforcement Learning
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Sparse network layouts change an agent's performance, and the best layout depends on whether hidden weights are fixed or learned.
desk verdict A genuinely new fixed-vs-learned sparsity comparison, but the central ranking reversal is asserted without a statistical test. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is a binary mask $M \in \{0,1\}^{d \times n}$ applied to the hidden-layer weight matrix $\Phi$ by element-wise multiplication, $M \odot \Phi$. The mask fixes the sparse topology before learning and is held constant across training regimes, so the same connection structure is tested once with hidden weights frozen at random initialization and once with hidden weights learned. Capacity is controlled by giving every architecture the same number of active connections and the same hidden-layer dimensionality, with sparsity fixed at about 91 percent. This isolates the sparse structure as the only architectural variable, making any performance gap attributable to the interplay between topology and whether weights are fixed or learned.
What would settle it
Run the same capacity-controlled comparison in Breakout and Space Invaders with a Spatial variant that shares weights across kernels (a true convolution) or uses a different receptive-field size, and check whether it overtakes the random and prediction-based baselines under fixed hidden weights; if it does, the central spatial-sparsity claim is refuted. A second check is to repeat the fixed-versus-learned sweep on a third MinAtar game, such as Freeway or Asterix, and see whether the architecture ordering still reverses.
Extended reading notes
Core claim
The paper's central discovery is an interaction: the sparse connectivity pattern that yields the highest return is different when hidden weights are fixed than when they are learned. With fixed random hidden weights, a random sparse network can match a fully connected network on Breakout, while a spatially biased network performs worst; in Space Invaders, the spatially biased network leads. When hidden weights are learned end-to-end, prediction-based and dense networks benefit the most on Breakout, while on Space Invaders random and dense networks nearly triple their performance and spatial connectivity barely improves. At fixed capacity and sparsity, these ordering changes show that the utility of a sparse architecture comes from the combination of connectivity and whether weight magnitudes are tuned to the task.
Load-bearing premise
The load-bearing premise is that the Spatial baseline, a hand-coded set of nearest-neighbor connections without weight sharing using 36 or 54 inputs per feature, fairly represents what a spatially biased, convolutional-style architecture would do; if a different receptive field or weight sharing would change the ranking, the paper's conclusion that spatial sparsity is not the most performant would not generalize.
Editorial extensions
If this is right
- Architecture rankings for sparse RL networks should report whether hidden weights are fixed or learned; the best topology in one regime is not reliably the best in the other.
- A fully connected network's advantage is not uniform: with fixed weights, sparse random connectivity can match dense networks on Breakout, so large capacity is not always buying better returns.
- Spatial sparsity is not automatically the right inductive bias in spatially structured RL domains; its performance is environment-dependent and can be the worst among the tested architectures.
- Sparse masks generated under one training regime (such as L1-regularization trained end-to-end) carry an implicit dependency: L1-regularized sparsity only performs competitively when the weights are learned, not when they are frozen.
- The fixed-versus-learned distinction is a meaningful axis for future sparse architecture selection, since the same connectivity can gain or lose several fold in average return depending on the regime.
Reading between the lines
- If the interaction generalizes beyond MinAtar, then transferring sparse architectures from one training pipeline to another, such as from a learned-weight benchmark to a fixed-weight deployment, could silently discard the inductive bias that made the architecture work.
- The surprisingly strong performance of random sparse connectivity under frozen weights suggests that, in that regime, what matters most may be the number and dimensionality of features rather than any specific pairing of inputs; a test that varies feature count while holding connections random would clarify this.
- A direct extension would be to compare the hand-coded Spatial baseline against a weight-sharing convolutional variant at equal capacity; if weight sharing changes the ranking, then the relevant inductive bias is not merely nearest-neighbor connectivity but shared filters.
- The paper's two-domain comparison leaves open whether the observed interactions are stable across games; re-running the same fixed-versus-learned sweep on the full MinAtar suite would establish whether the reversal is a general property of sparse RL architectures.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies whether the relative performance of sparse DQN architectures depends on whether the hidden-layer weights are fixed to random values or learned end-to-end. Using MinAtar Breakout and Space-Invaders, it compares Random, Spatial, Predictive, L1-Reg, and Dense architectures at roughly 91% sparsity, with 30 independent trials per condition. The paper claims that sparse structure has a significant effect on performance when capacity is controlled, that the best architecture reverses between the fixed-weight and learned-weight regimes, that spatially biased sparsity is not the most performant in spatially structured domains, and that L1-Reg is competitive only when the hidden weights are learned. The evidence is entirely empirical, with no statistical tests reported.
Significance. If the central interaction is real, it is an important result for deep RL practice: it would caution against transferring architecture rankings between fixed-feature and end-to-end training regimes and against assuming that spatially biased sparsity is always preferable in image-based domains. The manuscript is transparent about hyperparameters, uses a reasonable number of trials, and provides visualizations of the generated masks, which are useful for interpretation. However, the paper's headline claims are currently supported only by mean learning curves and qualitative language, and at least one confound (per-architecture step-size tuning) is present in a key comparison, so the significance of the findings is conditional on additional analysis.
major comments (4)
- [§4.3–4.5, Figures 1–2] The paper's central claim is a rank reversal between the fixed-weight and learned-weight regimes, but the text supports this claim only with qualitative statements such as 'significantly higher', 'significant margin', and 'not statistically distinguishable', without any hypothesis test, confidence interval, or description of what the plotted error bars represent. Since these are means over 30 trials of a high-variance RL procedure, an apparent interaction can arise from sampling noise; please report inferential statistics for the key pairwise comparisons (for example, Dense vs Predictive and Predictive vs Random in learned Breakout; Spatial vs Random and Predictive vs Random in fixed and learned Space-Invaders) and, ideally, a test or effect-size measure for the fixed-vs-learned interaction itself.
- [Tables 2–3, §4.4] In Space-Invaders with learned hidden weights, Spatial and L1-Reg use step-size 1e-5 while Predictive and Random use 1e-4. A smaller Adam step-size can by itself reduce learning speed and final performance, so the conclusion that Spatial is 'not useful' and that L1-Reg trails in the learned regime is partially confounded by optimizer tuning. Please either adopt a common step-size for all architectures in each regime or demonstrate that the ranking is stable across the swept step-sizes, for example by showing learning curves or final returns for the alternative settings.
- [§4.5, Appendix A.2] The L1-Reg baseline is generated by running DQN with L1 regularization on the same environment and then evaluating the resulting mask in the learned-weight condition. The mask is therefore optimized for the exact task used in the evaluation, which makes the comparison with Random, Predictive, and Spatial partly self-referential; this is acknowledged in §4.5, but the manuscript should either report how the conclusion changes when L1-Reg masks are generated on a different task or environment, or restrict the claims about L1-Reg accordingly.
- [§4.2, Figures 5 and 8, §4.4] The claim that 'spatially-biased sparsity is not useful' in spatially structured domains depends on the Spatial baseline being a fair proxy for convolutional-style spatial bias. This baseline uses fixed nearest-neighbor receptive fields without weight sharing, whereas the success of convolutions is often attributed to weight sharing and translation equivariance; with a different receptive-field size or shared weights the ranking could change. The paper flags this as future work in §4.4, but since the Spatial result is an explicit headline finding, the limitation should be stated directly alongside the claim and the conclusion should be narrowed accordingly.
minor comments (5)
- [§4.2] The text says that 'Each architecture has the same number of learnable parameters' but immediately notes that Dense has nearly ten times the active hidden-layer weights; please clarify that capacity is controlled at the level of hidden-layer dimensionality and sparsity fraction rather than total active parameters.
- [References] The references list Silver et al. 2017a and 2017b with the same title ('Mastering the game of go without human knowledge'); this duplication should be corrected.
- [§2.2, Eq. (4)] There is a typographical spacing error in the sentence following Eq. (4): 'In machine learning,L1-regularization' should have a space after the comma.
- [Figures 1–2] The main-text figures do not appear to include error bars or a description of their construction; if error bars are omitted from the figures, please state this explicitly in the figure captions or text.
- [General] No code or data release is indicated; for an empirical study of this kind, providing code or precomputed learning curves would substantially improve reproducibility.
Circularity Check
Main fixed-vs-learned comparison is self-contained; L1-Reg baseline is self-referential but disclosed and non-central.
-
fitted input called prediction
[Section 4.5, 'Case Study: Learned Hidden Weights and Topology' (paragraph beginning 'Why do we observe that L1-Reg...')]
"Recall that in order to generate L1-sparsity, we trained a dense network with a regularized loss function. This means that L1-sparsity is optimized for a network that is learned end-to-end through back-propagation. For this reason, L1-Reg yields high performance when the network is trained, which is precisely what Figure 2 shows. In other words, the learning task that generated L1-sparsity is the same task to approximate the action-values in DQN."
The L1-Reg mask is fitted by training a DQN with L1 regularization on the same environment and the same action-value objective used in the subsequent evaluation. Reporting that this baseline 'performs better than Random and statistically on par with Dense' in the learned-weights condition is therefore partly an in-sample report of the mask-generation procedure rather than an independent test of a fixed sparse topology. The paper explicitly acknowledges this ('the same task to approximate the action-values in DQN'), so the step is disclosed. It does not make the main fixed-vs-learned reversal circular because that reversal is driven by Random, Spatial, Predictive, and Dense, whose masks do not come from the evaluated objective.
full rationale
The paper's central claim is an empirical ranking comparison across sparse DQN architectures under two training regimes, not a derivation from a fitted quantity. Random, Spatial, and Predictive masks are generated by procedures external to the evaluated DQN objective (random assignment, hand-coded neighborhoods, and Prediction Adapted Networks respectively), so the observed reversal in which architecture performs best under fixed versus learned hidden weights is independent evidence. The Predictive baseline does cite prior work by a co-author (Martin & Modayil, 2021), but the current paper's conclusion about predictive sparsity rests on the new Figure 2 results, not on that citation, so the self-citation is not load-bearing. The only self-referential element is the L1-Reg baseline: its mask is produced by optimizing the same DQN/L1 loss used in the learned-weights evaluation, making its learned-regime performance partly a re-test of the fitting procedure. This is acknowledged in Section 4.5 and does not drive the paper's main interplay claim. Missing inferential statistics are a correctness/support concern, not circularity.
Assumptions & free parameters
free parameters (5)
- Step-size for each architecture/domain/learning regime =
Breakout frozen: 0.1; Breakout learned: 1e-4; Space-Invaders frozen: 1e-4 for most; Space-Invaders learned: 1e-4 or…
- L1 regularization coefficient =
Breakout 2.5e-5, Space-Invaders 2e-5
- L1 zeroing threshold =
average hidden weight magnitude
- Sparsity level =
91%
- Predictive mask generation parameters =
k=9 neighbors, n=16 pre-activations per neighborhood, update frequency 1000 steps
assumptions (5)
- domain assumption MinAtar Breakout and Space-Invaders have spatial dependence structure that a nearest-neighbor receptive field can exploit.
- domain assumption Controlling the number of active connections controls network capacity.
- domain assumption A single hidden ReLU layer suffices to learn action-values in these domains.
- domain assumption The sparse structure is fixed during learning and is not affected by optimization.
- domain assumption Standard DQN with target network and Adam is a reliable base learner.
Cite this review
Pith. "Pith review of On the Interplay Between Sparsity and Training in Deep Reinforcement Learning." pith.science (2026). https://pith.science/paper/247ZCOIH
@misc{pith2026250116729,
author = {Pith},
title = {Pith review of: On the Interplay Between Sparsity and Training in Deep Reinforcement Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/247ZCOIH}},
note = {Machine review of arXiv:2501.16729}
}
read the original abstract
We study the benefits of different sparse architectures for deep reinforcement learning. In particular, we focus on image-based domains where spatially-biased and fully-connected architectures are common. Using these and several other architectures of equal capacity, we show that sparse structure has a significant effect on learning performance. We also observe that choosing the best sparse architecture for a given domain depends on whether the hidden layer weights are fixed or learned.
Figures
Figures from the paper (8 more)
Reference graph
Works this paper leans on
-
[1]
On the convergence of bounded agents.arXiv preprint arXiv:2307.11044,
David Abel, André Barreto, Hado van Hasselt, Benjamin Van Roy, Doina Precup, and Satinder Singh. On the convergence of bounded agents.arXiv preprint arXiv:2307.11044,
-
[7]
Discovering sensor space: Constructing spatial embeddings that explain sensor correlations
Joseph Modayil. Discovering sensor space: Constructing spatial embeddings that explain sensor correlations. In 2010 IEEE 9th International Conference on Development and Learning, pp. 120–125. IEEE,
work page 2010
-
[10]
The quest for a common model of the intelligent decision maker
Richard S Sutton. The quest for a common model of the intelligent decision maker. arXiv preprint arXiv:2202.13252,
-
[11]
Are Sparse Neural Networks Better Hard Sample Learners?
Qiao Xiao, Boqian Wu, Lu Yin, Christopher Neil Gadzinski, Tianjin Huang, Mykola Pechenizkiy, and Decebal Constantin Mocanu. Are sparse neural networks better hard sample learners?arXiv preprint arXiv:2409.09196,
-
[13]
11 Under review as submission to TMLR A Experimental Details A.1 Trials, Initial Conditions and Environment Dynamics To evaluate performance, we compute the average return incurred by each DQN agent over 30 independent trials. More specifically, atrial refers to a random seed used to (1) randomly initialize the DQN weights at the beginning of learning, (2...
work page 2019
-
[2009]
Learning Sparse Representations Incrementally in Deep Reinforcement Learning
J Fernando Hernandez-Garcia and Richard S Sutton. Learning sparse representations incrementally in deep reinforcement learning. arXiv preprint arXiv:1912.04002,
work page Pith review arXiv 1912
-
[2010]
Towards model-free RL algorithms that scale well with unstructured data
Joseph Modayil and Zaheer Abbas. Towards model-free RL algorithms that scale well with unstructured data. arXiv preprint arXiv:2311.02215,
-
[2015]
Junjie Liu, Zhe Xu, Runbin Shi, Ray CC Cheung, and Hayden KH So. Dynamic Sparse Training: Find efficient sparse network from scratch with trainable masked layers.arXiv preprint arXiv:2005.06870,
arXiv 2005
Show all 13 references
-
[2018]
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,
-
[2019]
Adapting the function approximation architecture in online reinforce- ment learning
9 Under review as submission to TMLR John D Martin and Joseph Modayil. Adapting the function approximation architecture in online reinforce- ment learning. arXiv preprint arXiv:2106.09776,
-
[2020]
Mastering the game of go without human knowledge
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354–359, 2017a. David Silver, Julian Schrittwieser...
-
[2022]
Taylor, Mykola Pechenizkiy, and Decebal Constantin Mocanu
Bram Grooten, Ghada Sokar, Shibhansh Dohare, Elena Mocanu, Matthew E. Taylor, Mykola Pechenizkiy, and Decebal Constantin Mocanu. Automatic noise filtering with dynamic sparse training in deep reinforce- ment learning. InProceedings of the 2023 International Conference on Auton...
2023
-
[2024]
MinAtar: An Atari-inspired testbed for thorough and reproducible reinforce- ment learning experiments.arXiv preprint arXiv:1903.03176,
Kenny Young and Tian Tian. MinAtar: An Atari-inspired testbed for thorough and reproducible reinforce- ment learning experiments.arXiv preprint arXiv:1903.03176,
1903 arXiv
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.