{"id":"04c38924-e0d5-4f0a-9fab-17403bdee9e4","arxiv_id":"2506.17204","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"One-shot random pruning at initialization lets deep RL networks keep improving at model sizes where dense networks collapse in performance.","lead":"This paper shows that randomly deleting a fixed fraction of a network's weights before training helps large reinforcement learning agents learn better, while the same larger dense networks get worse as they grow. A generalist should care because it offers a simple, parameter-efficient recipe for scaling RL models, a setting where bigger networks usually hurt.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dense scaling barrier may be a fixed-hyperparameter artifact: SimBa defaults are never re-tuned for 109M-208M networks, so the sparsity advantage lacks a tuned dense baseline.","rationale":"The reader's weakest assumption is also the point I would bet the argument on. The paper's empirical case is broad (DMC, visual DrQ-v2, streaming Stream AC(lambda), Atari-100k) and internally consistent; Figure 6's Reset diagnosis and Figure 5's plasticity metrics provide independent support that large dense SimBa networks suffer a real optimization pathology, not just a plateau. That support makes me reluctant to call the central claim unsound. But none of those diagnostics replaces a tuned dense baseline. Plasticity loss can be induced by a bad learning-rate/weight-decay combination, and the parameter-norm growth in Figure 7 is a signature of the fixed weight decay being too weak at scale. The paper explicitly states it keeps SimBa's hyperparameters consistently across all experiments (Tables 2 and 3), so the dense scaling limit has only been demonstrated under one hyperparameter configuration. The proposed grid search is expensive but feasible on the two most informative tasks. If the dense barrier survives a reasonable tuning grid, the conditional verdict can be upgraded; if it does not, the central claim is materially weakened. Because this is exactly the condition the reader flagged, I keep the verdict unchanged and agree with the reader's weakest-assumption analysis.","tokens_in":21624,"tokens_out":6622,"duration_ms":76735,"concrete_test":"Run Width Scale=4 and Depth Scale=2 dense SimBa SAC on Humanoid Walk and Dog Run with a grid over learning rate {3e-5, 1e-4, 3e-4} x weight decay {1e-3, 1e-2, 3e-2} x replay ratio {1, 2}, using 8 seeds and 1M environment steps, and compare the best dense result against the sparse network at the same total model size. Also run the same grid on the sparse network to verify the comparison is not confounded by a different sensitivity to learning rate. If no dense configuration closes the gap, the scaling barrier is robust; if one does, the headline claim needs qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is that the dense scaling barrier in Figure 2 is a property of dense networks, not of the fixed small-model hyperparameters used to train them. The paper fixes learning rate (1e-4), weight decay (1e-2), replay ratio (2), and discount heuristic across all model sizes and sparsity levels (Appendix B.1, Tables 2-3). Large dense networks show unbounded parameter-norm growth (Figure 7) and plasticity loss (Figure 5), but both symptoms are exactly what one would expect from an undertuned learning rate/weight decay combination on a much larger network. If dense 109M/208M networks were re-tuned per scale, some configuration might avoid the collapse, in which case 'sparsity unlocks scaling' would reduce to 'sparse networks happen to work well with the hyperparameters chosen for the 17M baseline.' The central causal claim - that sparsity, rather than per-scale tuning, is what unlocks scaling - depends on this control, and the paper does not report it. The Reset diagnostic in Figure 6 is suggestive but not decisive, because Reset is not a hyperparameter search. This is a missing control, not an internal inconsistency.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks whether static network sparsity alone, implemented by one-shot random pruning at initialization, can unlock further scaling of deep RL networks beyond the dense SimBa architecture. On six DMC Hard tasks with SAC and DDPG, it reports that dense SimBa networks degrade when scaled in width or depth beyond the default size, while ER-pruned sparse networks at the same or larger total model sizes keep improving, especially when learnable parameter counts are matched to the dense optimum. The paper then analyzes Srank, dormant ratio, gradient norms, parameter norms, simplicity bias, and gradient covariance to argue that sparsity mitigates capacity collapse, plasticity loss, unbounded parameter growth, and gradient interference. It extends the evaluation to visual RL with DrQ-v2, streaming RL with Stream AC(λ), and Atari-100k with DER. The headline comparison is a direct empirical scaling study and does not depend on fitted parameters.","tokens_in":21843,"tokens_out":8582,"duration_ms":93796,"significance":"If the comparison holds, this is a practical and falsifiable finding: a fixed random mask applied before training can push the scaling frontier of actor-critic RL beyond current state-of-the-art architectures without resets, dynamic sparsity, or optimizer changes. The paper's strengths are its breadth (six DMC tasks, two algorithms, visual and streaming setups, five to eight seeds), the public code release, and the use of standard diagnostic metrics. The main risk is interpretive: all configurations share one small-model hyperparameter recipe, so the dense scaling barrier and the sparsity advantage are only established under that fixed recipe. A dense baseline re-tuned per scale is the missing control that determines whether the title-level claim is supportable.","major_comments":[{"comment":"The dense scaling barrier is established only under the SimBa hyperparameter recipe (learning rate 1e-4, weight decay 1e-2, replay ratio 2, and the TD-MPC2 discount heuristic) applied unchanged to every model size from about 4.5M to 208M parameters. No per-scale tuning, or even a small grid over learning rate and weight decay, is reported for the dense 109M and 208M baselines. Figures 5 and 7 show that large dense networks suffer rising dormant ratios and unbounded parameter norms, but those are exactly the symptoms one expects when a learning rate and weight decay tuned for a 17M network are applied to a much larger one. Because the central claim is that sparsity, rather than hyperparameter choice, unlocks scaling, this missing control is load-bearing. Please add per-scale hyperparameter searches for the large dense baselines and report whether any dense configuration closes the gap; if the gap persists, state explicitly that the comparison is a no-tuning comparison and scope the conclusion accordingly.","section":"Section 3.1 / Appendix B.1 (Tables 2-3), Figure 2"},{"comment":"The sparsity levels used for the sparse curves in Figures 1 and 2 are not stated in the main text. Section 3.1 says the sparse configurations maintain the same learnable parameter count as the optimal dense model while increasing total model size, but it does not report the sparsity ratio used at each width and depth scale. Figure 7 shows one 208M point at 92% sparsity, while Figure 3 sweeps sparsity from 0.1 to 0.9 on a 109M network, so the exact relationship between total size, sparsity, and learnable parameters is not transparent. Please provide a table of (width scale, depth scale, total parameters, sparsity, learnable parameters) for every point in the scaling plots; this is needed to verify the parameter-efficiency claim and to make the experiments reproducible.","section":"Section 3.1 / Figures 1-2"},{"comment":"Two diagnostic claims used to support the mechanism analysis are not quantitatively specified. The Srank metric in Section 4.1 is defined with an unspecified threshold tau and an unspecified sample size d, and the claimed theoretical upper bound of 256 is stated without derivation. In Section 4.4, the gradient-interference analysis reports only a qualitative covariance-matrix figure, with the number of sampled points k in Eq. (2) not given and no numerical summary of the correlation strength before and after training. Please report concrete values for tau, d, and k, and add a quantitative measure (for example, mean absolute off-diagonal covariance) so the claims of 'capacity collapse' and 'significantly weaker correlations' can be checked.","section":"Section 4.1 / Section 4.4 / Figure 9"}],"minor_comments":[{"comment":"The sizes of the baseline networks are inconsistent: Figure 1 refers to a 17M baseline SimBa network while the Figure 3 caption and Section 3 describe the default SimBa network as about 4.5M parameters; please reconcile these numbers.","section":"Figure 1 caption / Figure 3 caption"},{"comment":"The masking procedure is not fully specified: it is unclear whether masks are applied to biases, layer-normalization gains, or convolutional kernels, and how the ER layer-wise sparsity is computed for residual blocks; please state which tensors are pruned and how learnable parameters are counted.","section":"Section 2.2 / Appendix B.1"},{"comment":"Please specify the Srank threshold tau and the number of samples d used to form the feature matrix F; without these values the numerical Srank results and the claimed upper bound of 256 cannot be reproduced.","section":"Section 4.1 / Figure 4"},{"comment":"The Atari-100k experiment does not report the number of seeds or show error bars, despite being used as evidence that the benefits generalize to discrete-action domains; please add this information.","section":"Appendix C.3 / Figure 18"},{"comment":"The legend in Figure 7 lists the large sparse network entry twice; please remove the duplicate.","section":"Figure 7"}],"recommendation":"major_revision","confidential_remarks":"The paper is worth another round. The observed sparse-versus-dense gap is real under the stated fixed-recipe protocol; the risk is interpretation. The dense re-tuning control and a precise table of sparse configurations should be requested before the title-level claim is accepted. I would not reject on the basis of the mechanism section, but the headline should be scoped to the no-tuning protocol if the control confirms the gap."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this paper reports that a single fixed random mask, applied at init, lets large SimBa networks keep improving where dense ones collapse. The effect is consistent across six DMC tasks, two algorithms, and visual/streaming setups. If you work on RL scaling or sparse training, this is worth your time.\n\nWhat's genuinely new: prior sparse RL work used dynamic topologies (SET, RigL, gradual magnitude pruning, growth-based methods). Here they just prune once at init with ER layer ratios and train. That simplicity is the point, and it holds up as a recipe. Their parameter-efficiency comparison is also fair: sparse large networks are matched to dense networks on learnable parameter count, so the gains aren't just from more parameters. The mechanism section is reasonable—dormant ratios, Srank, gradient interference—though none of those metrics are new.\n\nThe soft spot is the load-bearing one: dense baselines are never re-tuned per model size. All scales share the SimBa hyperparameters (LR 1e-4, weight decay 1e-2, replay ratio 2). Large dense networks show parameter-norm growth and plasticity loss, but those symptoms are exactly what you'd expect from an undertuned dense network. The Reset diagnostic in Figure 6 is suggestive—Reset helps dense but not sparse—but Reset is not a hyperparameter search. So the claim 'sparsity unlocks scaling' is partly an artifact of fixing dense hyperparameters to the small-model optimum. It's a missing control, not an internal contradiction. I'd like to see at least a learning-rate sweep for the dense 109M/208M configurations.\n\nOther gaps are minor: Srank threshold τ is never specified, several key figures lack error bars (though the appendix has them for individual tasks), and the code link is vague—'GitHub/gtb' in the abstract is not a usable URL.\n\nBottom line: this is a practical empirical paper, not a theoretical breakthrough. The broad consistency of the results and the honesty about Atari-100k's limits in the appendix make it worth engaging. I'd send it to review, with a request to add per-scale dense tuning or explicitly reframe the claim as 'works with fixed hyperparameters.' If the authors can close that hole, the result becomes a strong recipe.","headline":"A simple, well-executed empirical study showing static random sparsity lets SimBa networks scale past dense collapse, but the missing per-scale dense baselines leave the headline claim conditional.","tokens_in":22372,"tokens_out":4109,"would_cite":true,"duration_ms":40468,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sparse networks, created by one random pruning before training, keep improving at model sizes where dense deep reinforcement learning networks collapse.","keywords":["deep reinforcement learning","network scaling","static sparsity","one-shot random pruning","plasticity loss","parameter efficiency","optimization pathologies","SimBa"],"falsifier":"Run the dense width and depth scaling curves for SimBa on DMC Hard with an independent hyperparameter sweep (learning rate, weight decay, discount, replay ratio) at every model size. If some per-size tuning restores monotone dense improvement beyond about 17M parameters, the claim that sparsity rather than tuning unlocks scaling is refuted; if dense collapse persists across the sweep while sparse networks at the same total size keep improving, the claim is supported.","tokens_in":21431,"feed_emoji":"🧠","tokens_out":10776,"duration_ms":98767,"temperature":0.7,"pith_summary":"Deep reinforcement learning (DRL) models have a scaling problem: past a certain size, larger dense networks stop improving and often get worse, because training pathologies such as plasticity loss and capacity collapse intensify with scale. The paper claims a much simpler fix than periodic reset or new architecture design: randomly remove a fixed percentage of weights once, before training, and leave the sparse topology fixed. On the state-of-the-art SimBa architecture, such sparse networks keep gaining performance at sizes where dense SimBa networks collapse, while using no more learnable parameters than the smaller dense baseline. The same benefit appears across continuous control, visual RL, streaming RL, and Atari-100k, and it requires no extra training computation and no modifications to the RL algorithm.","feed_headline":"One-shot random pruning unlocks deep RL scaling beyond dense limits","feed_subtitle":"A fixed mask chosen before training keeps larger networks learning instead of collapsing, in control and visual RL.","key_machinery":"The central object is a fixed binary mask $M_l \\in \\{0,1\\}^{n_l \\times n_{l-1}}$ per layer, produced once at initialization by random pruning at an overall sparsity level $S$, so that effective weights are $W_l^{\\mathrm{eff}} = M_l \\odot W_l$ throughout training. Layer-wise sparsity follows the Erdős–Rényi rule, which assigns each layer a sparsity $1 - \\frac{n_{l-1} + n_l}{n_{l-1} n_l}$ for fully connected layers (with a convolutional analogue), giving larger layers proportionally more surviving connections. This static sparse topology is the entire intervention: no growing, rewiring, or resetting during training. The paper argues the mask does the work by counteracting the pathologies that dense scaling aggravates—capacity collapse, plasticity loss, unbounded parameter growth, and gradient interference—thereby yielding higher parameter efficiency and continued performance gains at large model sizes.","core_discovery":"The central discovery is that static network sparsity alone, implemented as one-shot random pruning of a fixed fraction of weights before any training, can unlock scaling potential that dense networks with the same architecture cannot reach. Concretely, keeping the SimBa architecture and its hyperparameters fixed, dense networks degrade once width exceeds about twice the baseline or depth exceeds baseline, whereas sparse versions of those larger networks continue to improve, and at equal total size sparse networks beat dense ones with the same learnable parameter count. The paper attributes this to four mechanisms: larger sparse networks retain higher effective representational rank, keep dormant-neuron ratios and gradient norms at healthy levels, constrain parameter norm growth, and maintain more orthogonal gradients across training samples. It further shows that periodic Reset, which restores plasticity in dense networks, gives sparse networks no benefit, and that the benefit transfers to visual DrQ-v2, streaming Stream AC(λ), and Atari-100k.","pith_inferences":["The paper only tests unstructured random masks; if the benefit comes from limiting gradient interference, structured or block-sparse masks might give similar gains with better hardware utilization—the paper does not test this combination.","The Reset diagnostic suggests sparsity could replace Reset-style interventions in large-scale systems; an implicit consequence is that distributed training pipelines that currently schedule resets might be simplified, though the paper only studies single-agent benchmarks.","Since the mask is fixed before any data is seen, the recipe could transfer to settings with scarce or non-stationary data, such as continual RL; the streaming RL experiments are an early hint in that direction."],"forward_implications":["A single fixed random mask applied once before training is sufficient to make large DRL networks scale; no Reset, no dynamic topology, and no RL-algorithm modification is needed.","At equal total model size, sparse networks outperform dense networks with the same learnable parameter count, so sparsity improves parameter efficiency while allowing the nominal network to be much larger.","The scaling recipe follows directly: increase model size and keep high static sparsity, rather than stopping at the dense model's best size.","Because pruning is random and done once, it adds no per-step cost, can be applied before training starts, and is compatible with any RL algorithm or architecture.","The benefit is not confined to continuous control: image-based DrQ-v2, streaming Stream AC(λ), and Atari-100k all show sparse scaling gains."],"supporting_citations":[{"why":"Supplies the SimBa architecture baseline that the paper scales, and the simplicity-bias score used as a diagnostic.","marker":"Lee et al. 2024"},{"why":"Provides the static sparse training recipe (one-shot random pruning) that is the paper's entire intervention.","marker":"Liu et al. 2022"},{"why":"Supplies the Erdős–Rényi layer-wise sparsity rule for fully connected layers.","marker":"Mocanu et al. 2018"},{"why":"Extends the ER rule to convolutional layers and supplies the RigL dynamic-sparse baseline contrasted with static pruning.","marker":"Evci et al. 2020"},{"why":"Defines primacy bias and the Reset intervention used as a diagnostic to show sparse networks retain plasticity.","marker":"Nikishin et al. 2022"},{"why":"Defines the dormant neuron ratio used to measure plasticity loss in the paper's diagnostics.","marker":"Sokar et al. 2023"},{"why":"Provides the gradient-interference and plasticity measurement methodology, including gradient covariance matrices.","marker":"Lyle et al. 2023"},{"why":"Prior demonstration that gradual magnitude pruning improves value-based RL scaling, which the paper contrasts with static random pruning.","marker":"Ceron et al. 2024a"}],"fun_headline_variants":["Random pruning unlocks larger deep RL without collapse","Sparse networks outscale dense ones in deep reinforcement learning","One-shot sparsity unlocks deep RL scaling potential","Fixed random mask boosts deep RL scaling beyond dense"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the dense networks' performance collapse at larger sizes is a genuine scaling barrier and not an artifact of the paper holding one fixed SimBa hyperparameter set (learning rate, weight decay, discount, replay ratio) across all model sizes and sparsity levels.","fun_headline_variants_meta":{"raw":{"variants":["Random pruning unlocks larger deep RL without collapse","Sparse networks outscale dense ones in deep reinforcement learning","One-shot sparsity unlocks deep RL scaling potential","Fixed random mask boosts deep RL scaling beyond dense"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000254,"raw_usage":{"total_tokens":1523,"prompt_tokens":853,"completion_tokens":670,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":469,"completion_tokens_details":{"reasoning_tokens":610}},"tokens_in":469,"tokens_out":670,"duration_ms":7359,"temperature":1.0,"reasoning_tokens":610,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:10:19.794254+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the dense width and depth scaling curves for SimBa on DMC Hard with an independent hyperparameter sweep (learning rate, weight decay, discount, replay ratio) at every model size. If some per-size tuning restores monotone dense improvement beyond about 17M parameters, the claim that sparsity rather than tuning unlocks scaling is refuted; if dense collapse persists across the sweep while sparse networks at the same total size keep improving, the claim is supported.","supporting_citations":[{"cited_title":"C., Wang, Z., and Pechenizkiy, M","cited_arxiv_id":null,"evidence_quote":"Provides the static sparse training recipe (one-shot random pruning) that is the paper's entire intervention."}],"review_version":2}