Pith. sign in

REVIEW 4 major objections 6 minor 41 references

Adding a single learned permutation per layer to a fixed structured sparse pattern restores unstructured-level accuracy, and the paper explains the gain as restored expressivity measured by linear regions.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Learned per-layer permutations bring structured sparse training (block, N:M, diagonal) close to unstructured sparse-training accuracy at 90-95% sparsity, with inference speedups up to 2.9x.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection The learned-shuffle recipe works in practice, but the expressivity proof is invalid: the span computation undercounts regions for diagonal/block patterns. the 4 major comments →

arxiv 2510.14812 v2 pith:GX4RTL7C submitted 2025-10-16 cs.LG

SHUFFLESPARSE: Learned Shuffles for Structured Sparse Networks

Abhishek Tyagi , Arjun Iyer , Liam Young , William H Renninger , Christopher Kanan , Yuhao Zhu This is my paper
classification cs.LG MSC 68T07
keywords structured sparsitydynamic sparse traininglearned permutationslinear regionsexpressivityN:M sparsityblock sparsitytransformer pruning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SHUFFLESPARSE asks why structured sparsity—block, N:M, or diagonal masks that GPUs can run fast—lags behind unstructured dynamic sparse training in accuracy, especially at 90–95% sparsity. The paper's answer is that the gap is expressivity, not optimization: a fixed structure restricts which directions each layer can slice the input space along, so the depth-multiplicative growth of linear regions stalls. It proposes learning one permutation per layer, so the same sparse pattern is applied in a different learned orientation at each depth, and proves that this restores dense-like expressive growth after a short, quantified warm-up. Empirically, the recipe pulls block, N:M, and diagonal sparse training nearly up to unstructured accuracy on ImageNet-1K and WikiText-103 while keeping structured speedups, with under 8.7% inference overhead and no extra matrix multiplication at inference.

Core claim

On the paper's own terms, the central discovery is that a single full-rank, per-layer reorientation—instantiated as a learned permutation—recovers the expressivity structured sparsity removes. In the linear-region counting framework, an axis-aligned structure with directional rank cap r_struct caps the effective dimension k_l at that value, so the per-layer combinatorial factor in the region-count bound is bounded and depth no longer multiplies regions; with one permutation per layer, the span budget u_l = min(d0, u_{l-1} + r_struct) grows by r_struct each layer, and after ceil(d0/r_struct) layers the per-layer factor matches the dense bound. Composed with dynamic sparse training (PA-DST), t

What carries the argument

The central object is a column permutation matrix P learned per sparsified layer, applied as y = S(P x), where S obeys a fixed accelerator-friendly sparse pattern. During training P is represented as a soft doubly-stochastic matrix optimized with a Lipschitz row/column sparsity penalty that drives it toward a true permutation; at inference it disappears into precomputed index maps, so the structured sparse GEMM reads and writes reordered indices without an extra shuffle pass. The theoretical carrier is the span-budget recursion u_l = min(d0, u_{l-1} + r_struct) together with a master lower bound on the number of linear regions; the recursion converts the structure's rank cap into a count of

Load-bearing premise

The theoretical restoration claim assumes that inside every current linear region, the sparse rows of each layer can be arranged so their restricted normals stay in sufficiently general position to keep adding r_struct fresh independent directions; this subspace-general-position assumption is adopted rather than proved for block, N:M, and diagonal masks with learned permutations.

What would settle it

For the small network from the paper's worked example (d0=4, width 8, depth 3, block size 2), exhaustively evaluate permutations and exactly count linear regions; if no permutation sequence reaches the claimed product bound 37·163·163, the expressive-restoration theorem fails. More generally, search random structured masks and learned permutations for a depth where the span budget saturates but region counts stop growing multiplicatively.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • At 90–95% sparsity, permutation-augmented structure matches unstructured dynamic sparse training on ImageNet-1K and WikiText-103 while training up to about 1.21x faster and inferring up to about 2.9x faster.
  • The same permutation primitive applies across block, N:M, and diagonal patterns and across from-scratch dynamic sparse training and one-shot pruning, so the fix is pattern-agnostic rather than kernel-specific.
  • Because the permutation is absorbed as index reordering, inference adds no extra matrix multiply and less than 8.7% wall-clock overhead.
  • The theory predicts dense-like expressive growth after ceil(d0/r_struct) layers; in the worked example with alternating widths 1024/4096, mixing restores the dense per-layer factor after about four blocks, while unmixed structure stays capped.
  • The permutation-structured family is closed under transposition, so backward passes remain in the same family and sparse-to-sparse training acceleration is preserved.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If expressive restoration is the correct mechanism, cheap full-rank depth-varying reorderings—random shuffles, hash-based index maps, or butterfly-style mixers—should recover most of the accuracy gain, so the learned part may be a convenience rather than a necessity.
  • The warm-up length depends inversely on the structure's rank fraction, which suggests N:M patterns with dense groups (larger alpha) should need only very few layers to saturate the span budget; this is a direct, testable prediction of the theory.
  • A further testable extension is to count linear regions exactly on small ReLU nets and compare with the claimed product bound; if learned permutations do not attain it, the empirical accuracy gain would be coming from training dynamics rather than from restored expressivity.
  • The transfer to one-shot 2:4 pruning hints that learned permutations could become a post-hoc reordering stage for pretrained models, potentially a general alternative to fine-tuning the sparsity mask itself.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes PA-DST (presented under the name SHUFFLESPARSE in the arXiv metadata), which augments structured sparse layers—diagonal, block, and N:M—with a learned or fixed permutation matrix per layer, while dynamically updating the structured mask during training. The central thesis is twofold: (i) a theoretical claim that structured sparsity caps the number of linear regions at a per-layer factor determined by a 'directional rank' r_struct, and that a single learned permutation per layer restores dense-like, depth-multiplicative expressivity after O(d0/r_struct) layers; (ii) an empirical claim that this permutation-augmented structured DST reaches near-unstructured dynamic-sparse accuracy on ViT-B/16, ViT-L/16, Mixer-S/16, and GPT-2 Small/Medium at 90–95% sparsity, at modest inference overhead. The paper also reports ablations on random vs learned permutations and row vs column permutations.

Significance. If the theoretical result and the empirical claims were both valid, this would be a practically valuable contribution: a single, hardware-friendly primitive that recovers much of the accuracy lost to structured sparsity, backed by a clear combinatorial story and by extensive experiments on vision and language transformers. The empirical observation that learned or even fixed permutations consistently improve structured DST baselines is credible and worth reporting, as is the careful treatment of inference re-indexing to avoid explicit permutation multiplies. However, the theoretical argument is not merely incomplete; it is invalid for the canonical structured families it targets. The definitional inconsistency in Eq. (8)–(9), together with the unproved subspace-general-position assumption in Eq. (1), means the paper's central explanation for the gap is unsupported. Furthermore, the empirical results show substantial residual gaps to unstructured baselines at the extreme sparsities emphasized by the paper. The practical finding may survive as a useful empirical contribution, but the manuscript in its current form does not establish its main claims.

major comments (4)
  1. [§3.4, Eqs. (8)–(9)] The instantiation r_struct = K, B, or 2b+1 is inconsistent with the definition r_l = dim(span(A_l)). For a K-diagonal matrix—including K=1—the admissible row normals across rows include the coordinate directions e_1,...,e_{d_{l-1}}, so span(A_l) = R^{d_{l-1}}, not a K-dimensional subspace. The same holds for banded and block patterns, because different rows have nonzeros in different coordinate blocks; only a tied support where every row shares one fixed set of columns gives the claimed cap. A single diagonal layer with d0=4 and n=4 rows realizes 16 activation regions (axis-aligned hyperplanes), equal to the dense single-layer bound, whereas Eq. (9) with s=1 predicts a per-layer factor of only 1+4=5. Hence the 'stall' in Eq. (9), the restored bound Eq. (10), and the overhead formula Eq. (11) are not valid for the paper's flagship families.
  2. [§3.5, Eq. (10)] Even if r_struct were reinterpreted as per-row support width rather than the span of all admissible row normals, the span-budget recursion u_l = min(d0, u_{l-1}+r_struct) does not describe the architecture. For K=1, a layer SΠ with S diagonal and Π a permutation is a monomial matrix: each output coordinate is a scaled copy of one input coordinate, and a depth-L composition of such layers remains coordinate-wise; it cannot 'regain dense-like expressivity'. For K>1, banded and block row normals collectively span the full input space already at the first layer, so the idea that each layer injects only r_struct fresh directions until saturation is not a valid model. The claimed restoration after ceil(d0/r_struct) layers is therefore not established.
  3. [§3.1, Eq. (1)] The master lower bound is stated under a subspace-general-position (SGP) assumption, but no proof is supplied that the hyperplane arrangements induced by block, N:M, or diagonal masks—with or without learned permutations—satisfy SGP inside every current linear region. For axis-aligned patterns the hyperplanes are parallel in groups, so the generic arrangement count can overcount; for the permutation-augmented families the issue is not resolved. This is not a minor technicality: Eq. (1) is the foundation for Eq. (5), Eq. (9), and Eq. (10), so all numerical bounds in §3 and Appendix C should be regarded as unproven.
  4. [§6.1, Tables 11–12] The abstract and conclusion state that PA-DST 'matches' or attains 'unstructured-level accuracy'. The raw numbers do not support this at extreme sparsity. On ViT-B/16 at 95% sparsity, DynaDiag+PA-DST reaches 70.12% vs RigL's 71.50% (a 1.38-point gap); on GPT-2 Medium at 90% sparsity, DynaDiag+PA-DST has PPL 54.26 vs RigL's 51.76 (a 2.50-point gap), and even at 60% the gap is about 1.2 PPL. The claim should be weakened to 'reduces the structured-vs-unstructured gap' rather than 'matches unstructured baselines' or 'achieves unstructured-level accuracy'.
minor comments (6)
  1. [Title/Abstract] The arXiv metadata names the method SHUFFLESPARSE, while the body consistently uses PA-DST. Please reconcile the naming.
  2. [Abstract vs. full text] The arXiv abstract promises a one-shot 2:4 pruning experiment on LLaMA-2 7B with a 4.6-point improvement, but no such experiment appears in the provided main text or appendices. Either add the experiment or remove the claim.
  3. [Typographical issues] There are typos including 'upto' (§1), 'acorss' (Fig. 6 caption), and 'permutaitons' (Table 12 header).
  4. [Fig. 2] The caption says blue, green, and red lines correspond to unstructured, structured, and structured+permutation methods, but the color scheme is not always easy to distinguish in the reproduced figure; consider using distinct markers as well.
  5. [§6.1/Tables 11–12] Paired asymptotic McNemar tests are reported for both accuracy and perplexity; McNemar tests are not the standard significance test for perplexity differences. Please report standard errors/bootstrap intervals for PPL.
  6. [Appendix C] The worked example (Appendix C.1) uses 'Block-B' with all rows lying in the same B-dimensional subspace. This is the same incorrect assumption as in §3.4; the example should be revised or removed if the theoretical claims are corrected.

Circularity Check

1 steps flagged

The expressivity theorem encodes its conclusion: per-row nonzero count K is substituted for the directional rank cap dim span(A_l), so the predicted stall and permutation-driven recovery are built into the parameterization.

specific steps
  1. self definitional [Sec. 3.4, Eq. (8) and the instantiations feeding Eq. (9)-(11)]
    "Define the directional rank cap at layer ℓ by rℓ := dim(span(Aℓ)) ≤ dℓ−1. (8) ... For each sparsity structure, we instantiate rstruct as follows: for Diagonal-K, rstruct =K, hence s= min{d0, K}; for Block-B, rstruct =B, hence s= min{d0, B}; and for N:M with a tied group template, rstruct =αd0 with α=N/M."

    Under Eq. (8) itself, r_l is dim span(A_l). For K-diagonal, banded, and block patterns the row normals include the standard coordinate directions (K=1: they are e_1..e_d0), so span(A_l)=R^{d0}, i.e., r_l=d0, not K. A single ReLU diagonal layer with d0=4 already attains the dense bound of 16 regions. Replacing r_l by the per-row nonzero count K makes Eq. (9)'s stall, Eq. (10)'s restored bound, and Eq. (11)'s ceil(d0/K) overhead algebraic consequences of the chosen K. The claimed loss of structure expressivity and its recovery by a permutation are therefore injected through the definition of r_struct, not derived from the span of admissible normals.

full rationale

The experimental program is not circular: PA-DST accuracy is measured against external unstructured DST baselines (RigL, SET, CHT, MEST), and gains are reported on externally defined structures (SRigL, PixelatedBFly, DSB) as well as the authors' DynaDiag. The self-citation to prior DynaDiag work is used to explain a speed advantage, not to establish the central accuracy claim, so it is not load-bearing. However, the paper's theoretical derivation of the expressivity gap is circular in the sense that its key quantity r_struct is defined by Eq. (8) as dim span(A_l) but instantiated as per-row nnz (K, B, 2b+1). For the flagship diagonal/band/block families this makes the 'directional cap' equal to a chosen pattern parameter, so the predicted stall, the restored dense-like bound, and the warm-up overhead reduce by construction rather than following from the span of the admissible row normals. The empirical gains may be real, but the central formal claim that one learned permutation per layer provably restores lost expressivity is, as written, supported by the definition of the cap rather than by a derivation from it. This is a partial, theory-level circularity; score 6 rather than higher because the experimental comparisons are external and self-contained.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The central method introduces no new physical entities. The main uncharged assumptions are the unproved counting template and the SGP attainability of sparse+permuted masks; these carry the theoretical expressivity claim. The method also relies on two under-specified permutation-learning hyperparameters (lambda, delta) that shape the learned shuffles.

free parameters (2)
  • permutation-penalty weight lambda (Eq. 13) = not reported
    Controls how strongly the soft doubly-stochastic matrix is pushed toward a true permutation; no value or schedule is given in the text, and the learned permutations depend on its balance with the task loss.
  • soft-to-hard permutation threshold delta = 0.22
    Chosen by monitoring permutation-penalty curves (Fig. 5); defines when a layer's soft permutation is frozen, affecting training time and the final permutation.
axioms (4)
  • domain assumption Eq. (1): NLR(f) >= product over layers of sum_{j=0}^{k_l} C(n_l,j) under subspace-general position (SGP)
    Master lower-bound template stated without proof in Sec. 3.1; all subsequent structured-vs-dense expressivity comparisons use it as the counting engine.
  • domain assumption Masked row normals from block/N:M/diagonal sparsity, after per-layer permutations, can be kept in SGP inside every current region
    Needed for the span budget u_l to translate into actual linear-region multiplicativity; attainability is asserted, not demonstrated.
  • ad hoc to paper NLR expressivity is the principal cause of the structured-vs-unstructured accuracy gap
    Introduction claims the shortfall stems from a loss of expressivity, but no experiment varies NLR while holding other factors fixed.
  • domain assumption Inference-time re-indexing is exactly equivalent to a permutation multiply and adds negligible cost
    Sec. 4.3; true for exact index maps, but measured overheads are only reported for the specific kernels used.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of SHUFFLESPARSE: Learned Shuffles for Structured Sparse Networks." pith.science (2026). https://pith.science/paper/GX4RTL7C

@misc{pith2026251014812,
  author       = {Pith},
  title        = {Pith review of: SHUFFLESPARSE: Learned Shuffles for Structured Sparse Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GX4RTL7C}},
  note         = {Machine review of arXiv:2510.14812}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Structured weight sparsity accelerates training and inference on modern GPUs, but it trails unstructured dynamic sparse training (DST) in accuracy especially at extreme sparsity. We pinpoint the reason for this difference in performance to a lack of expressivity: a dense layer can implement any pattern of non-zero weights, whereas structured patterns are restricted to only a small set of weight configurations. We introduce SHUFFLESPARSE, a single permutation primitive that applies uniformly across DST-from-scratch and one-shot pruning, and across N:M, block etc. We close most of this gap by learning a single permutation matrix jointly with the structured weight matrix. When used on three different types of structures (block, N:M, and diagonal), SHUFFLESPARSE is able to reduce the structured-vs-unstructured accuracy gap on ViT-B16 (ImageNet-1K) and GPT-2 (WikiText-103) at 90-95% sparsity, while adding minimal inference overhead (< 8.7% inference overhead) and preserving any training acceleration the host structure provides. The same permutation formulation transfers to one-shot 2:4 pruning of pretrained LLMs, where it improves zero-shot accuracy by 4.6 points on LLaMA-2 7B. Together, these results establish learned permutations as a general tool for recovering unstructured-level accuracy from structured sparsity patterns.

Figures

Figures reproduced from arXiv: 2510.14812 by Abhishek Tyagi, Arjun Iyer, Christopher Kanan, Liam Young, William H Renninger, Yuhao Zhu.

Figure 1
Figure 1. Figure 1: Overview of the PA-DST training approach. The training starts by initializing the target [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Accuracy vs sparsity of networks listed in Sec. 5. The plots in blue, green, and red correspond [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Impact of permutations on the training and inference time for a ViT-B/16 model. We observe [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Distance of permutations to an identity matrix (no permutation) for a ViT-B/16 at 90% sparsity [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Tracking the loss of permutation matrices based on Eqn. 14. We plot the loss for every 10 [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: We show the epoch count of each layer in a ViT-B/16 network when the corresponding [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 11 linked inside Pith

  1. [1]

    What is the state of neural network pruning? Proceedings of machine learning and systems, 2: 0 129--146, 2020

    Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? Proceedings of machine learning and systems, 2: 0 129--146, 2020

    2020

  2. [2]

    Structured pruning is all you need for pruning cnns at initialization

    Yaohui Cai, Weizhe Hua, Hongzheng Chen, G Edward Suh, Christopher De Sa, and Zhiru Zhang. Structured pruning is all you need for pruning cnns at initialization. arXiv preprint arXiv:2203.02549, 2022

    Pith/arXiv arXiv 2022

  3. [3]

    Trends in the dollar training cost of machine learning systems

    Ben Cottier. Trends in the dollar training cost of machine learning systems. Epoch. January, 31: 0 2023, 2023

    2023

  4. [4]

    Kaleidoscope: An efficient, learnable representation for all structured linear maps

    Tri Dao, Nimit S Sohoni, Albert Gu, Matthew Eichhorn, Amit Blonder, Megan Leszczynski, Atri Rudra, and Christopher R \'e . Kaleidoscope: An efficient, learnable representation for all structured linear maps. arXiv preprint arXiv:2012.14966, 2020

    Pith/arXiv arXiv 2012

  5. [5]

    Pixelated butterfly: Simple and efficient sparse training for neural network models

    Tri Dao, Beidi Chen, Kaizhao Liang, Jiaming Yang, Zhao Song, Atri Rudra, and Christopher Re. Pixelated butterfly: Simple and efficient sparse training for neural network models. arXiv preprint arXiv:2112.00029, 2021

    Pith/arXiv arXiv 2021

  6. [6]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.\ 248--255. Ieee, 2009

    2009

  7. [7]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    Pith/arXiv arXiv 2010

  8. [8]

    Rigging the lottery: Making all tickets winners

    Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International conference on machine learning, pp.\ 2943--2952. PMLR, 2020

    2020

  9. [9]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks

    Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635, 2018

    Pith/arXiv arXiv 2018

  10. [10]

    Learning both weights and connections for efficient neural network

    Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015

    2015

  11. [11]

    Accelerated sparse neural training: A provable and efficient method to find n: m transposable masks

    Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner, Joseph Naor, and Daniel Soudry. Accelerated sparse neural training: A provable and efficient method to find n: m transposable masks. Advances in neural information processing systems, 34: 0 21099--21111, 2021

    2021

  12. [12]

    Training your sparse neural network better with any mask

    Ajay Kumar Jaiswal, Haoyu Ma, Tianlong Chen, Ying Ding, and Zhangyang Wang. Training your sparse neural network better with any mask. In International Conference on Machine Learning, pp.\ 9833--9844. PMLR, 2022

    2022

  13. [13]

    Exposing and exploiting fine-grained block structures for fast and accurate sparse training

    Peng Jiang, Lihan Hu, and Shihui Song. Exposing and exploiting fine-grained block structures for fast and accurate sparse training. Advances in Neural Information Processing Systems, 35: 0 38345--38357, 2022

    2022

  14. [14]

    Dynamic sparse training with structured sparsity

    Mike Lasby, Anna Golubeva, Utku Evci, Mihai Nica, and Yani Ioannou. Dynamic sparse training with structured sparsity. arXiv preprint arXiv:2305.02299, 2023

    Pith/arXiv arXiv 2023

  15. [15]

    Towards optimal structured cnn pruning via generative adversarial learning

    Shaohui Lin, Rongrong Ji, Chenqian Yan, Baochang Zhang, Liujuan Cao, Qixiang Ye, Feiyue Huang, and David Doermann. Towards optimal structured cnn pruning via generative adversarial learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 2790--2799, 2019

    2019

  16. [16]

    On improving deep learning generalization with adaptive sparse connectivity

    Shiwei Liu, Decebal Constantin Mocanu, and Mykola Pechenizkiy. On improving deep learning generalization with adaptive sparse connectivity. arXiv preprint arXiv:1906.11626, 2019

    Pith/arXiv arXiv 1906

  17. [17]

    Alphapruning: Using heavy-tailed self regularization theory for improved layer-wise pruning of large language models

    Haiquan Lu, Yefan Zhou, Shiwei Liu, Zhangyang Wang, Michael W Mahoney, and Yaoqing Yang. Alphapruning: Using heavy-tailed self regularization theory for improved layer-wise pruning of large language models. arXiv preprint arXiv:2410.10912, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    Ai beats humans for the first time in physical skill game

    Ioanna Lykiardopoulou. Ai beats humans for the first time in physical skill game. https://www.newscientist.com/article/2402645, 12 2023. Accessed: 20 01 2025

    arXiv 2023

  19. [19]

    Autoshufflenet: Learning permutation matrices via an exact lipschitz continuous penalty in deep convolutional neural networks

    Jiancheng Lyu, Shuai Zhang, Yingyong Qi, and Jack Xin. Autoshufflenet: Learning permutation matrices via an exact lipschitz continuous penalty in deep convolutional neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp.\ 608--616, 2020

    2020

  20. [20]

    Building a large annotated corpus of english: The penn treebank

    Mitch Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19 0 (2): 0 313--330, 1993

    1993

  21. [21]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016

    Pith/arXiv arXiv 2016

  22. [22]

    Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science

    Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9 0 (1): 0 2383, 2018

    2018

  23. [23]

    Variational dropout sparsifies deep neural networks

    Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In International conference on machine learning, pp.\ 2498--2507. PMLR, 2017

    2017

  24. [24]

    Pruning convolutional neural networks for resource efficient inference

    Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440, 2016

    Pith/arXiv arXiv 2016

  25. [25]

    On the number of linear regions of deep neural networks

    Guido Mont \'u far, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. Advances in neural information processing systems, 27, 2014

    2014

  26. [26]

    Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization

    Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning, pp.\ 4646--4655. PMLR, 2019

    2019

  27. [27]

    Cusparse library

    Maxim Naumov, L Chien, Philippe Vandermersch, and Ujval Kapasi. Cusparse library. In GPU Technology Conference, volume 12, 2010

    2010

  28. [28]

    Channel permutations for n: m sparsity

    Jeff Pool and Chong Yu. Channel permutations for n: m sparsity. Advances in neural information processing systems, 34: 0 13316--13327, 2021

    2021

  29. [29]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  30. [30]

    Game-playing deepmind ai can beat top humans at chess, go and poker

    Matthew Sparkes. Game-playing deepmind ai can beat top humans at chess, go and poker. https://thenextweb.com/news/, 11 2023. Accessed: 20 01 2025

    2023

  31. [31]

    Pruning neural networks without any data by iteratively conserving synaptic flow

    Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in neural information processing systems, 33: 0 6377--6389, 2020

    2020

  32. [32]

    Mlp-mixer: An all-mlp architecture for vision

    Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. Advances in neural information processing systems, 34: 0 24261--24272, 2021

    2021

  33. [33]

    Dynamic sparse training of diagonally sparse networks

    Abhishek Tyagi, Arjun Iyer, William H Renninger, Christopher Kanan, and Yuhao Zhu. Dynamic sparse training of diagonally sparse networks. arXiv preprint arXiv:2506.11449, 2025

    Pith/arXiv arXiv 2025

  34. [34]

    Mest: Accurate and fast memory-economic sparse training framework on the edge

    Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, et al. Mest: Accurate and fast memory-economic sparse training framework on the edge. Advances in Neural Information Processing Systems, 34: 0 20838--20850, 2021

    2021

  35. [35]

    Universal structural patterns in sparse recurrent neural networks

    Xin-Jie Zhang, Jack Murdoch Moore, Gang Yan, and Xiang Li. Universal structural patterns in sparse recurrent neural networks. Communications Physics, 6 0 (1): 0 243, 2023

    2023

  36. [36]

    Epitopological learning and cannistraci-hebb network shape intelligence brain-inspired theory for ultra-sparse advantage in deep learning

    Yingtao Zhang, Jialin Zhao, Wenjing Wu, Alessandro Muscoloni, and Carlo Vittorio Cannistraci. Epitopological learning and cannistraci-hebb network shape intelligence brain-inspired theory for ultra-sparse advantage in deep learning. In The Twelfth International Conference on Learning Representations, 2024

    2024

  37. [37]

    Brain-inspired sparse training enables transformers and llms to perform as fully connected

    Yingtao Zhang, Jialin Zhao, Wenjing Wu, Ziheng Liao, Umberto Michieli, and Carlo Vittorio Cannistraci. Brain-inspired sparse training enables transformers and llms to perform as fully connected. arXiv preprint arXiv:2501.19107, 2025

    arXiv 2025

  38. [38]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  39. [39]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  40. [40]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  41. [41]

    winning tickets,

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.