Pith. sign in

REVIEW 4 major objections 4 minor 33 references

The paper claims that replacing layerwise sparsity selection with fixed half-density masking on an augmented score tensor makes strong-lottery-ticket extraction more reliable.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 02:20 UTC pith:CKG5GAGV

load-bearing objection A clever score-space trick that removes layerwise sparsity tuning, but the headline accuracy gain over edge-popup evaporates when sparsity is matched. the 4 major comments →

arxiv 2607.20555 v1 pith:CKG5GAGV submitted 2026-07-15 cs.LG math.OC

Double-Scoring: Reliable Extraction of Strong Lottery Tickets

Bryce A. Christopherson , Jack Baretz , Darian Colgrove , Salah Dandan This is my paper
classification cs.LG math.OC
keywords strong lottery ticketslottery ticket hypothesisedge-popupscore spacelayerwise sparsitypruning at initializationstraight-through estimatorfrozen-weight training
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper targets a practical bottleneck in the strong lottery ticket hypothesis: such tickets are known to exist, but finding them reliably is hard. It argues that the obstacle is not the frozen weights or the optimization dynamics of edge-popup, but the parameterization of sparsity: users must choose a density for every layer, and bad choices sharply degrade accuracy. The proposed fix, double-scoring, appends an auxiliary score tensor of the same shape to each weight tensor and applies a fixed-density (1/2) top-k mask to the concatenation, then restricts the mask back to the original coordinates. The paper proves this preserves representational access to every original-coordinate mask and is exactly edge-popup on a zero-padded network, and it reports substantially higher strong-ticket accuracy and lower sensitivity to sparsity settings on controlled MLP and ConvNet benchmarks.

Core claim

On the paper's own terms, the central discovery is that a dense layer need not be searched over a family of layerwise sparsity levels. If each layer is at least half zero, then every masked subnetwork can be realized by a mask with exactly the half-density number of ones. Doubling the score tensors simulates the needed zeros without touching the weights: the mask produced by taking the top half of the concatenated scores and restricting to original coordinates can realize any original mask, and the score updates coincide exactly with edge-popup run on an augmented zero-padded network. Empirically, the effective original sparsity emerges from competition with auxiliary scores, and this induce

What carries the argument

The central object is the augmented score tensor bS_t = (S_t, T_t), formed by concatenating two score tensors of the same shape as each weight tensor. A hard top-k mask at fixed density 1/2 is applied to |bS_t|, and the result is restricted to the original coordinates. The zero-weight/zero-score observation — changing mask entries on coordinates where the weight is zero does not change the masked tensor — carries the argument: it lets every original-coordinate mask be represented at fixed half density (Propositions 5.1–5.3), and Theorem H.6 shows the straight-through dynamics are identical to edge-popup on a zero-padded network. The auxiliary scores act as a competitive reservoir, not as add

Load-bearing premise

The load-bearing premise is that straight-through score optimization over randomly initialized doubled scores will actually converge to good masks, and that the accuracy gain comes from the induced effective sparsity rather than from enforcing the requested original sparsity; the paper proves representability but not convergence, and its own ablation shows that projecting to exact requested sparsity makes the method worse than edge-popup.

What would settle it

Run double-scoring and fixed-density edge-popup on the same task and initialization, then evaluate both at the same achieved original-coordinate sparsity; if edge-popup at that achieved sparsity matches or beats double-scoring, the claimed advantage from augmented competition collapses. A second check: if freezing the auxiliary scores at their random initialization changes the outcome, the straight-through equivalence in Theorem H.6 would be violated.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Edge-popup-style extraction no longer requires a layerwise sparsity sweep; a single global density of 1/2 over the augmented scores is enough for representability.
  • High-sparsity strong tickets can be extracted with roughly the same runtime as standard training, up to a constant factor from the doubled score tensors.
  • The extracted masks are simultaneously strong and trainable in the tested settings: they begin with high accuracy and remain competitive after masked weight training.
  • The advantage is tied to induced effective sparsity; projecting the final mask to the exact requested sparsity substantially degrades accuracy, so applications requiring exact sparsity must calibrate the nominal augmented density.
  • Sensitivity to sparsity hyperparameters drops markedly: the paper reports an edge-popup sweep spanning roughly 18.6% to 84.4% accuracy while the double-score variants stay near 80% with standard deviation below 0.2 percentage points.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • I infer that the paper's deeper claim, if true, is that the practical barrier to strong tickets is the parameterization of mask search rather than existence or convergence; the theoretical results are representational only, and the paper itself states they do not guarantee that straight-through optimization finds globally optimal masks.
  • The effective-sparsity mechanism suggests a general recipe: any mask-search method that fixes density can be freed by embedding weights in a larger score space, and the paper notes the choice of which coordinates count as original is arbitrary.
  • A testable extension is to control induced sparsity by varying the auxiliary-to-original score ratio; the appendix observes that extra auxiliary capacity tends to pull the induced sparsity toward 50%.
  • I would be cautious about extrapolating the empirical margin beyond the reported MLP and no-BatchNorm ConvNet settings, since the paper's matched CIFAR-10 comparison shows a retrospectively chosen edge-popup density can slightly outperform double-scoring.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. Double-scoring is a modification of edge-popup for strong lottery ticket extraction: for each layer, two score tensors are concatenated and a fixed-density top-k mask is computed on the augmented tensor; the mask is then restricted to the original weights. The paper proves (Prop. 5.1–5.3, Thm H.6) that every original-coordinate mask remains representable by some half-density augmented mask and that double-scoring is exactly edge-popup on a zero-padded network. Empirically, it reports improved strong-ticket accuracy over edge-popup and pruning baselines on FashionMNIST MLPs at requested 90/95% sparsity, lower sensitivity to sparsity choices, and competitive performance when masks are subsequently trained. A CIFAR-10 ConvNet matched-sparsity sanity check is reported in Appendix G. Section 7 explicitly limits the theoretical results to representational guarantees and notes that the method induces effective sparsity rather than enforcing an exact budget.

Significance. The representational propositions are elementary but correct, and the paper follows good empirical practice in several respects: it reports achieved sparsity, shares random initializations across methods, includes code in the supplementary material, reports a matched-sparsity CIFAR-10 check, and candidly acknowledges in Appendix G that DoubleScore is not uniformly superior to a retrospectively well-chosen edge-popup density. If the headline empirical claims held, the paper would make a practically useful contribution by removing the layerwise density search from edge-popup-style extraction while preserving representational access to all masks. The method has no fitted constants and the effective sparsity is measured rather than tuned. However, the central empirical claim of substantial improvement over fixed-density edge-popup is not supported once sparsity is controlled; the plausible remaining contribution is a weaker one of hyperparameter robustness and competitive accuracy.

major comments (4)
  1. [§6.1, Table 1; §6.4, Table 8; App. G, Tables 9–10] The abstract's central claim that double-scoring 'substantially improves strong-ticket extraction over fixed-density edge-popup' is not supported at matched sparsity. In Table 1, DoubleScore-Augmented achieves 82.74% at 81.3% achieved sparsity vs EdgePopup 77.73% at 90.0%; at requested 95%, 78.03% at 90.0% vs 60.61% at 95.0%. When original sparsity is controlled, the advantage disappears: Table 8's Aug-x1-Projected at 90.0% achieved sparsity is 70.16% vs SingleScore-Abs 73.88%; matched CIFAR-10 fixed edge-popup is 49.08% vs DoubleScore 45.73% (Table 10), and Table 9 shows no consistent edge. The paper's own App. G acknowledges this, but the abstract and conclusion still assert dominance. The accuracy gain in Table 1 is therefore attributable to the uncontrolled effective-sparsity difference, not to improved mask search.
  2. [§4 (Algorithm 1), §6, App. B] The representational theory (Prop. 5.3, Thm H.6) is stated for a fixed augmented-space density of exactly 1/2, but the high-sparsity experiments (requested 90/95%) use a 'targetable augmented variant' in which a single global augmented-space density is supplied. Algorithm 1 and App. B do not specify how the requested sparsity maps to the augmented density, nor do they extend Prop. 5.3 to general densities. Thus the main empirical results are for a variant outside the proven representability guarantee. Specify the variant precisely and either prove a coverage result for general k or restrict the theoretical claims accordingly.
  3. [§5, Thm H.6, Remark H.7] Thm H.6 and Remark H.7 show that auxiliary scores receive zero gradient and double-scoring is exactly edge-popup on a zero-padded network. The method therefore does not optimize the enlarged score space in the usual sense: the auxiliary coordinates are a fixed random thresholding reservoir, as confirmed by Aug-x1-FrozenAux (Table 8: 79.87 vs 80.34). Consequently, the theory provides no mechanism for accuracy gain at fixed original sparsity; the empirical gain over edge-popup must come from the induced density shift. The abstract's explanatory phrase 'augmented score-space competition' should be qualified to reflect this.
  4. [§6.3, Fig. 2] The hyperparameter-stability comparison repeats the same confound. In §6.3, DoubleScore-Augmented is reported at 80.1% with effective sparsity about 82.9% (Table 8), while the fixed edge-popup baseline is 73.5% at 90.0% achieved sparsity; the unconstrained oracle is explicitly denser, and the constrained oracle is described but its numeric value is not reported. A fair stability comparison should hold achieved sparsity fixed. As presented, the stability claim is also entangled with the density advantage.
minor comments (4)
  1. [§5] The statement after Prop. 5.1 is called Corollary 5.2 in the text but 'Theorem 5.2' in the proof header and later references; renumber consistently.
  2. [App. G] Table 9 reports DoubleScore-Augmented 46.68±2.21 at nominal 90%, while Table 10 reports 45.73±1.17 for the same setting; clarify whether these are different runs/protocols or reconcile the discrepancy.
  3. [Notation] The paper alternates between 'EdgePopup' and 'edge-popup'; choose one convention. Also, Algorithm 1's output includes bias masks, but the body sometimes refers only to weights.
  4. [App. H] There is an unfinished sentence: 'This is not technically needed for the basic double-scoring construction; but instead it records...' — rephrase for clarity.

Circularity Check

0 steps flagged

No significant circularity: the representational claims are self-contained proofs and the empirical comparisons are independent measurements with limitations disclosed.

full rationale

The paper's derivation chain is self-contained. Propositions 5.1 and 5.3 establish representability by explicit combinatorial constructions (adding ones on zero-weight coordinates), and Theorem H.6 proves exact equivalence to edge-popup on a zero-padded network under the stated straight-through estimator; neither theorem assumes the conclusion it derives. The method's effective sparsity is measured and reported, not fitted to accuracy and then re-reported as a prediction. The central empirical claim is benchmarked against external baselines (edge-popup, SNIP, GraSP, IMP, SET, RigL, Movement) on shared initializations, which is independent evidence rather than a self-citation chain. The paper explicitly limits its theory: Section 7 and Remark 4.1 state that the results are representational and do not guarantee that straight-through score optimization finds optimal masks, and Appendix G concedes that matched edge-popup can be competitive (e.g., fixed matched edge-popup 49.08 vs. DoubleScore 45.73 at matched CIFAR-10 sparsity). The unmatched-sparsity issue in Table 1 is a fair-comparison/correctness concern, not circularity: the comparison is not a fitted-input-then-prediction chain, and the paper's own matched experiments are reported. No load-bearing self-citations or imported uniqueness theorems appear; the inheritance of Ramanujan et al.'s monotonicity result (Corollary H.8) is explicitly attributed to prior work and is not needed for the representability claims. The speculative appendices are explicitly labelled as not supporting the main claims. Therefore, no reduction of a claimed result to its own inputs was found.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The method has no fitted constants and introduces no new physical or conceptual entity: the auxiliary score coordinates are additional trainable parameters, not a separately postulated mechanism. The central claim rests on the elementary zero-weight masking observation, on the straight-through gradient assumption, and on the empirical premise that score optimization will succeed.

axioms (4)
  • standard math Mask entries on zero-weight coordinates can be changed freely without changing the masked tensor.
    Used in Prop 5.1 and 5.3: W⊙M is unchanged when the mask flips on entries where W=0.
  • domain assumption The straight-through estimator for the hard top-k mask is a valid gradient proxy.
    Required by Algorithm 1 and Theorem H.6 to equate double-scoring with edge-popup on a zero-augmented network and to train the scores at all.
  • domain assumption Strong lottery tickets exist in sufficiently overparameterized random networks.
    The paper relies on Malach et al., Orseau et al., and Pensia et al. for existence; the contribution is extraction, not existence.
  • domain assumption Random score initialization and Adam optimization in the enlarged score space will find high-accuracy masks in practice.
    Not guaranteed by the representational theory; Section 7 states that optimization may still fail. All empirical gains depend on this premise.

pith-pipeline@v1.3.0-alltime-deepseek · 35118 in / 16328 out tokens · 153868 ms · 2026-08-02T02:20:41.880454+00:00 · methodology

0 comments
read the original abstract

The lottery ticket hypothesis proposes that large random neural networks contain sparse subnetworks that can match the performance of dense models after comparable training. A stronger version asserts that sufficiently overparameterized random networks contain subnetworks that are already accurate before any weight training. Existing theory establishes that such strong lottery tickets exist, but reliable extraction remains difficult. We revisit edge-popup, a frozen-weight score-training method for extracting strong tickets, and identify layerwise sparsity selection as a central bottleneck. We introduce double-scoring, an augmented score-space parameterization that replaces a layerwise sparsity search with optimization over enlarged score tensors. We prove that fixed-density masking in an augmented score space preserves access to all original-coordinate masks, and we show that the resulting method can be interpreted as edge-popup on a zero-augmented network. In controlled experiments, double-scoring substantially improves strong-ticket extraction over fixed-density edge-popup and pruning-at-initialization baselines, improves on the performance of rewound sparse-training topologies, and exhibits markedly lower sensitivity to sparsity hyperparameters. Ablations show that the gain is not merely due to additional trainable score parameters, but is tied to the augmented score-space competition that induces the effective original sparsity.

Figures

Figures reproduced from arXiv: 2607.20555 by Bryce A. Christopherson, Darian Colgrove, Jack Baretz, Salah Dandan.

Figure 1
Figure 1. Figure 1: Weak-ticket training after strong-ticket extraction. Each method first extracts a binary [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Hyperparameter stability at requested 90% sparsity on FashionMNIST with a linear MLP. Left: edge-popup exhibits large variance across scalar and layerwise sparsity choices, while the double-score variants are stable across seeds. Right: the constrained oracle provides a same￾regime high-sparsity reference, while the unconstrained validation-oracle edge-popup configuration achieves high accuracy only by sel… view at source ↗
Figure 3
Figure 3. Figure 3: Strong-ticket extraction accuracy across requested sparsity levels. DoubleScore [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Achieved sparsity of extracted masks. Methods that impose a fixed global density match the [PITH_FULL_IMAGE:figures/full_fig_p015_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Masked weight-training curves at requested [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Scalar keep-density sweep for edge-popup at requested 90% sparsity. Performance varies substantially with the density parameter, illustrating that fixed-density edge-popup requires nontrivial sparsity selection even in this controlled benchmark. The full hyperparameter-stability comparison, including constrained and unconstrained validation-selected references, appears in [PITH_FULL_IMAGE:figures/full_fig… view at source ↗
Figure 7
Figure 7. Figure 7: Ablation accuracy versus achieved sparsity. The augmented variants obtain high accuracy [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Auxiliary score-capacity ablation. Increasing auxiliary width improves accuracy but also [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Matched CIFAR-10 ConvNet hyperparameter stability. Left: [PITH_FULL_IMAGE:figures/full_fig_p021_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Performance depends sharply on the density choice, both globally and layerwise, and [PITH_FULL_IMAGE:figures/full_fig_p022_10.png] view at source ↗
Figure 10
Figure 10. Figure 10: The k-selection bottleneck: A network with a single width 32 hidden layer was trained using edge-popup to approximate a sine function across layerwise density tuples. Left: The mean￾squared errors of the resulting masked networks are plotted above over a two-layer grid of the possible layerwise densities. Right: A heat map of the same figure with contour lines. As is apparent, the landscape is jagged, hig… view at source ↗
Figure 11
Figure 11. Figure 11: A network with a single width 8 hidden layer was trained using [PITH_FULL_IMAGE:figures/full_fig_p024_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Left: test accuracy as a function of k-tuples for a 50% sparsity two-layer network trained on the scikit-learn digits dataset. The best performance begins to appear around the center of each k interval. Right: A heat map of test losses for the same network across k-tuples. The best-performing region concentrates near the fixed-density regime suggested by Theorem 5.1, consistent with the claim that the spa… view at source ↗
Figure 13
Figure 13. Figure 13: The fixed-density procedure tracks the outcome of an exhaustive search over layerwise [PITH_FULL_IMAGE:figures/full_fig_p028_13.png] view at source ↗
Figure 13
Figure 13. Figure 13: Comparison between the proposed fixed-density method and brute-force layerwise density [PITH_FULL_IMAGE:figures/full_fig_p029_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Approximation error trends are broadly consistent with the logarithmic [PITH_FULL_IMAGE:figures/full_fig_p029_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Loss curves for training a network with one hidden layer of width 8 trained to approximate [PITH_FULL_IMAGE:figures/full_fig_p030_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Two identical networks were trained for 20 epochs on the MNIST Fashion data set. The [PITH_FULL_IMAGE:figures/full_fig_p031_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Observationally, it often appears to be the case that the obtained strong lottery tickets [PITH_FULL_IMAGE:figures/full_fig_p032_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Two identical networks were trained for 20 epochs on the MNIST Fashion dataset with [PITH_FULL_IMAGE:figures/full_fig_p033_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: By changing the initial k-value and additional capacity in the score tensors, it is possible to fine-tune the sparsity of the mask obtained via double-scoring. The initial k-value determines the sparsity of the mask at initialization, and the ratio of the number of parameters in the score tensors and weight tensors controls the speed of its deviation from this value. Though it is unclear why, it appears t… view at source ↗
Figure 20
Figure 20. Figure 20: By repeatedly performing Algorithm 1 and applying the obtained mask to the weights of [PITH_FULL_IMAGE:figures/full_fig_p034_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Training with simultaneous masking and weight modification produces no meaningful [PITH_FULL_IMAGE:figures/full_fig_p035_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Function approximation using masked polynomial compositions. The [PITH_FULL_IMAGE:figures/full_fig_p036_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Similar Tasks Have Similar Masks: Well-performing masks are identified in a small [PITH_FULL_IMAGE:figures/full_fig_p038_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: A data set of doubled mask popup scores was generated for a very small test network [PITH_FULL_IMAGE:figures/full_fig_p039_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: We train a transformer-based policy that generates sequences of mask tokens (“programs”) [PITH_FULL_IMAGE:figures/full_fig_p041_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: For each true class, the most frequently generated programs (top- [PITH_FULL_IMAGE:figures/full_fig_p041_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: Top: The best-achieved loss over generations and the distribution of training trajectories [PITH_FULL_IMAGE:figures/full_fig_p043_27.png] view at source ↗
Figure 28
Figure 28. Figure 28: Performance of dream-based subnet selection compared to a random classical baseline. [PITH_FULL_IMAGE:figures/full_fig_p044_28.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 7 linked inside Pith

  1. [1]

    A survey on mixture of experts.arXiv preprint arXiv:2407.06204, 2024

    Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts.arXiv preprint arXiv:2407.06204, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    Towards understanding the mixture-of-experts layer in deep learning.Advances in neural information processing systems, 35:23049–23062, 2022

    Zixiang Chen, Yihe Deng, Yue Wu, Quanquan Gu, and Yuanzhi Li. Towards understanding the mixture-of-experts layer in deep learning.Advances in neural information processing systems, 35:23049–23062, 2022

    2022

  3. [3]

    Rigging the lottery: Making all tickets winners

    Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. InInternational conference on machine learning, pages 2943–2952. PMLR, 2020

    2020

  4. [4]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks

    Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. 2018. doi: 10.48550/ARXIV .1803.03635. URL https://arxiv.org/abs/ 1803.03635

    Pith/arXiv arXiv doi:10.48550/arxiv.1803.03635 2018

  5. [5]

    Linear mode connectivity and the lottery ticket hypothesis

    Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 ofPro- ceedings of Machine Learning Research, pages 3259–3269. PMLR, 13–18 Jul 2020. URL https:...

    2020

  6. [6]

    Mixture of experts models

    Isobel Claire Gormley and Sylvia Frühwirth-Schnatter. Mixture of experts models. InHandbook of mixture analysis, pages 271–307. Chapman and Hall/CRC, 2019

    2019

  7. [7]

    Transfer learning: a friendly introduction.Journal of Big Data, 9(1):102, 2022

    Asmaul Hosna, Ethel Merry, Jigmey Gyalmo, Zulfikar Alom, Zeyar Aung, and Mohammad Ab- dul Azim. Transfer learning: a friendly introduction.Journal of Big Data, 9(1):102, 2022

    2022

  8. [8]

    Snip: Single-shot network pruning based on connection sensitivity.arXiv preprint arXiv:1810.02340, 2018

    Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity.arXiv preprint arXiv:1810.02340, 2018

    Pith/arXiv arXiv 2018

  9. [9]

    Proving the lottery ticket hypothesis: Pruning is all you need, 2020

    Eran Malach, Gilad Yehudai, Shai Shalev-Shwartz, and Ohad Shamir. Proving the lottery ticket hypothesis: Pruning is all you need, 2020. URLhttps://arxiv.org/abs/2002.00585

    Pith/arXiv arXiv 2020

  10. [10]

    Mixture of experts: a literature survey.Artificial Intelligence Review, 42:275–293, 2014

    Saeed Masoudnia and Reza Ebrahimpour. Mixture of experts: a literature survey.Artificial Intelligence Review, 42:275–293, 2014

    2014

  11. [11]

    Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science.Nature communications, 9(1):2383, 2018

    Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science.Nature communications, 9(1):2383, 2018

    2018

  12. [12]

    Practical and theoretical aspects of mixture-of- experts modeling: An overview.Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(4):e1246, 2018

    Hien D Nguyen and Faicel Chamroukhi. Practical and theoretical aspects of mixture-of- experts modeling: An overview.Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(4):e1246, 2018. 10

    2018

  13. [13]

    Logarithmic pruning is all you need

    Laurent Orseau, Marcus Hutter, and Omar Rivasplata. Logarithmic pruning is all you need. Advances in Neural Information Processing Systems, 33:2925–2934, 2020

    2020

  14. [14]

    Optimal lottery tickets via subset sum: Logarithmic over-parameterization is sufficient

    Ankit Pensia, Shashank Rajput, Alliot Nagle, Harit Vishwakarma, and Dimitris Papailiopou- los. Optimal lottery tickets via subset sum: Logarithmic over-parameterization is sufficient. Advances in neural information processing systems, 33:2599–2610, 2020

    2020

  15. [15]

    What’s hidden in a randomly weighted neural network?, 2019

    Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari. What’s hidden in a randomly weighted neural network?, 2019. URL https: //arxiv.org/abs/1911.13299

    Pith/arXiv arXiv 2019

  16. [16]

    Movement pruning: Adaptive sparsity by fine-tuning.Advances in neural information processing systems, 33:20378–20389, 2020

    Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine-tuning.Advances in neural information processing systems, 33:20378–20389, 2020

    2020

  17. [17]

    The bitter lesson, 2019.URL http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019

    Rich Sutton. The bitter lesson, 2019.URL http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019

    2019

  18. [18]

    Pruning neural networks without any data by iteratively conserving synaptic flow.Advances in Neural Information Processing Systems, 33:6377–6389, 2020

    Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow.Advances in Neural Information Processing Systems, 33:6377–6389, 2020

    2020

  19. [19]

    Picking winning tickets before training by preserving gradient flow.arXiv preprint arXiv:2002.07376, 2020

    Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow.arXiv preprint arXiv:2002.07376, 2020

    Pith/arXiv arXiv 2002

  20. [20]

    A survey of transfer learning.Journal of Big data, 3:1–40, 2016

    Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. A survey of transfer learning.Journal of Big data, 3:1–40, 2016

    2016

  21. [21]

    Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis

    Jiancheng Yang, Rui Shi, and Bingbing Ni. Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis. In2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 191–195. IEEE, 2021

    2021

  22. [22]

    Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification.Scientific Data, 10(1):41, 2023

    Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification.Scientific Data, 10(1):41, 2023

    2023

  23. [23]

    The super weight in large language models.arXiv preprint arXiv:2411.07191, 2024

    Mengxia Yu, De Wang, Qi Shan, Colorado J Reed, and Alvin Wan. The super weight in large language models.arXiv preprint arXiv:2411.07191, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    Twenty years of mixture of experts

    Seniha Esen Yuksel, Joseph N Wilson, and Paul D Gader. Twenty years of mixture of experts. IEEE transactions on neural networks and learning systems, 23(8):1177–1193, 2012

    2012

  25. [25]

    A comprehensive survey on transfer learning.Proceedings of the IEEE, 109(1): 43–76, 2020

    Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning.Proceedings of the IEEE, 109(1): 43–76, 2020. A Reproducibility and code All code necessary to reproduce the experiments is included in the anonymous supplementary material. B Experimental details All experiments ...

    2020

  26. [26]

    The batch size is 512

    Score variables are trained with learning rate 10−2, while weight-training baselines use learning rate 10−3. The batch size is 512. We set Python, NumPy, and PyTorch random seeds for each run and disable cuDNN benchmarking. Datasets and assets used. Mask densities and achieved sparsity.For a requested sparsity s, the corresponding keep density is k= 1−s ....

  27. [27]

    With this choice, k1 ranges over an interval centered at 1 2 of width sz 1−s

    One convenient choice is k= 1−s+sz 2 1−s(1−z) = 1 2 , which is the midpoint of the admissible interval above. With this choice, k1 ranges over an interval centered at 1 2 of width sz 1−s. The nontrivial masking proportions on the nonzero entries are precisely 1 n(1−s) , 2 n(1−s) , . . . ,n(1−s)−1 n(1−s) . Hence, it is enough for the interval of possible k...

  28. [28]

    Since this is wasteful in practice (one can simply check these trivial masks without any score training), this is likely undesirable. 25 The content of Theorem H.3 is that, on a sufficiently sparse tensor, selector-based scoring allows one to simulate the full family of nontrivial masking levels while keeping the target density fixed at k= 1

  29. [29]

    This has a particularly clean consequence when one is interested only in representability rather than in the bookkeeping of scored zero coordinates

    Thus, the apparent freedom in the density parameter is, in this regime, largely illusory: one may trade a search over densities for a suitable choice of scored coordinates. This has a particularly clean consequence when one is interested only in representability rather than in the bookkeeping of scored zero coordinates. This is the content of Theorem 5.1 ...

  30. [30]

    Assume that the backward pass for TopKMask uses the usual straight-through estimator, i.e

    classicaledge-popupapplied to the augmented tensors( cWt, bbt)with scores( bSt, bht); 2.double-scoring applied to the original tensors (Wt, bt) with score pairs (St, Tt) and (ft, gt), where the effective masks are obtained by restricting TopKMask(bSt; 1/2)and TopKMask( bft; 1/2)to the original coordinates. Assume that the backward pass for TopKMask uses t...

  31. [31]

    the masked parameters used by the two procedures agree on the original coordinates, and hence the network outputs and losses are identical

  32. [32]

    the gradients of the original score coordinates agree in the two procedures

  33. [33]

    similar tasks, similar masks

    the gradients of the auxiliary score coordinates are zero in both procedures. Consequently, after restricting to the original coordinates,double-scoring produces exactly the same score iterates, masks, network outputs, and losses as classical edge-popup on the augmented zero-padded network. 27 Proof. We compare gradients first with respect to the magnitud...