Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Dynamic Sparse Training of Diagonally Sparse Networks

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A sparse training method confined to diagonal weight patterns matches unstructured accuracy at moderate sparsity and converts the structure into real GPU speedups in both training and inference.

desk verdict Genuinely new diagonal-sparsity DST method with real hardware speedups, but the abstract's 'par with unstructured' claim overreaches at high sparsity — the paper's own tables show significant gaps vs RigL at 95% ViT-B/16 and 90% GPT-2. read the letter →

arxiv 2506.11449 v1 pith:Q2ERSVSM submitted 2025-06-13 cs.LG

Abhishek Tyagi , Arjun Iyer , William H Renninger , Christopher Kanan , Yuhao Zhu This is my paper
classification cs.LG
keywords dynamicsparsetrainingdiagonalsparsitystructuredsparse-to-sparsevisiontransformersGPUaccelerationblockmatricesTopKmaskselection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DynaDiag is a dynamic sparse training method that restricts every weight matrix to a set of full diagonals, chosen by a differentiable TopK over learnable importance scores, and it keeps that diagonal pattern through both forward and backward passes. Because transposing a diagonal mask yields another diagonal mask, the backward pass stays sparse instead of densifying, and the selected diagonals are repacked into block-sparse (BCSR) blocks for custom GPU kernels. The paper's central claim is that this structured pattern performs on par with unstructured dynamic sparse training at moderate sparsities (60–80%) while producing, at 90% sparsity in Vision Transformers, up to 3.13× inference and 1.59× training speedups over equivalent unstructured layers without a statistically significant accuracy loss. At sparsities of 80% and above the purely diagonal mask shows an accuracy gap that the authors close by adding low-rank unstructured adapters (LoRA-FA) outside the mask. If the claim holds, structured sparsity stops being a compromise: hardware-friendly training and inference no longer require giving up the accuracy of unstructured masks in the regimes most models actually use.

What carries the argument

The central machinery is the diagonal-sparse factorization of a weight matrix as a sum of permutation matrices times diagonal value vectors, gated by a learnable importance vector passed through a temperature-controlled softmax TopK. Transposition invariance of the pseudo-diagonal mask is what carries training acceleration: the transposed weight used in backpropagation remains diagonal, so sparse gradients can be formed without a dense pass. The final hardware step converts the selected diagonals to Block Compressed Sparse Row (BCSR) format, using a Jaccard-plus-proximity heuristic to cluster rows from nearby diagonals into dense blocks that map onto GPU tensor-core operations. These pieces together turn an abstract sparsity pattern into a method that is differentiable, dynamic, and hardware-aware.

What would settle it

Train ViT-B/16 on ImageNet-1K at 95% sparsity with DynaDiag and with unstructured DST over multiple seeds and compare Top-1 accuracy with a paired McNemar test: the paper's own Table 1 and Appendix Table 10 already report a significant gap (69.54 vs 71.68 for CHT and 71.50 for RigL, p=0.0352), which would refute the parity claim in the high-sparsity regime unless the LoRA-FA adapters are counted as part of the sparse model.

Watch

Extended reading notes

Core claim

The discovery is that a transposable structured sparsity pattern—full diagonals of a weight matrix—is expressive enough to make sparse-to-sparse training competitive with unstructured DST while remaining computationally sparse end-to-end. The paper represents a layer as $W_K = \sum_{j=1}^K \tilde\alpha_j P_j \operatorname{diag}(V_j)$, where each $P_j$ places a $V_j$ vector on a diagonal and $\tilde\alpha = \operatorname{TopK}(\alpha)$ masks in the $K = (1-S)MN/\min(M,N)$ diagonals that a sparsity budget $S$ allows. The importance scores $\alpha$ are trained with a temperature-annealed softmax TopK, so diagonal positions are learned rather than fixed. The authors prove that a pseudo-diagonal mask stays pseudo-diagonal under transposition, which lets the backward pass multiply by the transposed sparse weight without breaking the pattern; the diagonals are then reordered into dense blocks (BCSR) and executed on GPU tensor cores. On this basis the paper reports accuracy statistically indistinguishable from unstructured baselines such as RigL at 60–80% sparsity on ImageNet-1K, and the speedups above.

Load-bearing premise

The load-bearing premise is that limiting every layer to a small set of full diagonals—about $(1-S)MN/\min(M,N)$ of them—preserves enough expressivity to match the accuracy of unconstrained sparse masks at the same sparsity budget, particularly at the high sparsities the paper emphasizes.

Editorial extensions

If this is right

  • Structured DST no longer has to sacrifice training speed: DynaDiag keeps sparsity in both forward and backward passes, so sparse-to-sparse training converts directly into GPU time saved.
  • At 60–80% sparsity the diagonal mask is statistically at parity with unstructured DST methods, making structured sparsity a drop-in replacement in the regime where unstructured masks give no hardware benefit anyway.
  • At 90% sparsity on ViT models, the reported wall-clock numbers mean a practitioner can get a 3.13× faster online inference and 1.59× faster training while staying within measurement error of the unstructured baseline's accuracy.
  • At extreme sparsity (99–99.9%) DynaDiag outperforms unstructured RigL, suggesting the fixed full-row and full-column coverage of diagonal masks prevents the layer-collapse failure mode that hurts unstructured DST.
  • The high-sparsity accuracy gap is shown to be recoverable: adding rank-6 LoRA-FA adapters (about 1.67% more parameters) lets DynaDiag exceed RigL accuracy, indicating the diagonal pattern captures most but not all of the needed connectivity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next experiment the paper does not run: replacing the diagonal pattern with any other transpose-closed family (e.g., anti-diagonals or symmetric band patterns) would isolate whether the gains come from diagonals per se or from transposability combined with block packing.
  • The reported speedups are tied to custom CUDA kernels and BCSR packing measured on A100 GPUs; porting the same diagonal pattern to standard sparse libraries or consumer hardware could shrink or erase the 3.13×/1.59× figures, so the speedup is a property of the implementation as much as the pattern.
  • The LoRA-FA fix points to a general hybrid recipe—a structured diagonal backbone for most weights plus a small unstructured residual—that could be trained jointly rather than post-hoc; the paper does not claim this.
  • The small-world interpretation suggests a scaling prediction: as matrices widen, the number of diagonals needed for a given accuracy should grow more slowly than the number of unstructured nonzeros, so diagonal sparsity should become increasingly attractive for very large layers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes DynaDiag, a dynamic sparse training (DST) method that restricts non-zero weights to a set of learnable diagonals. Diagonals are selected through a differentiable TopK mechanism on a trainable importance vector, and the resulting matrices are converted to Block CSR (BCSR) format to enable GPU acceleration in both forward and backward passes. The authors evaluate DynaDiag on ImageNet-1K with ViT-B/L/H and MLP-Mixer and on WikiText-103 with GPT-2 Small/Medium, comparing against unstructured DST baselines (RigL, SET, MEST, CHT, CHTs) and structured baselines (SRigL, DSB, PixelatedBFly, and a DiagHeur variant). They report that DynaDiag matches unstructured sparsity accuracy while achieving up to 3.13x inference and 1.59x training speedups, and they provide ablations on temperature schedules, sparsity schedules, sparsity distributions, and a check that BCSR conversion does not change accuracy.

Significance. If the central claims held, this would be a valuable contribution: a structured DST method that preserves sparse computation in both forward and backward passes, converts diagonal structure into real GPU speedups, and matches the accuracy of unstructured DST. The experimental breadth is a genuine strength, including multiple model families, sparsity levels, paired statistical tests, ablations, and a BCSR-vs-direct accuracy check in Table 8. However, the headline parity claim is contradicted by the paper's own results at the high sparsity levels it emphasizes, and the speedup comparison baseline is stated inconsistently. The significance is therefore contingent on a substantial reframing of the claims rather than on the current abstract.

major comments (4)
  1. [Abstract; §4.2.1, Table 1; §4.2.2, Table 2; §4.3.1; Table 10] The abstract's claim that DynaDiag 'performs at par with unstructured sparsity' and achieves 90%-sparse ViT accuracy 'without sacrificing model performance' is not supported at high sparsity. In Table 1, ViT-B/16 at 95% sparsity gives 69.54 for DynaDiag versus 71.50 for RigL and 71.68 for CHT, and Table 10 reports p=0.0352 versus RigL, i.e., significantly worse. In Table 2, GPT2-S at 90% sparsity gives perplexity 56.33 versus RigL's 53.76 (p=0.0401 in Table 11), and GPT2-M gives 54.87 versus 51.76 (p=0.0136). Section 4.3.1 explicitly concedes a 'performance gap between RigL and DynaDiag at sparsities ≥ 80%' and closes that gap only by adding unstructured LoRA-FA parameters outside the diagonal mask (Fig. 5). The parity claim must be narrowed to the regimes where the data actually support it, or the paper must provide a version of DynaDiag that achieves parity without unstructured extras.
  2. [Abstract; §4.2.3; Fig. 4] The training-speedup baseline is stated inconsistently. The abstract says '1.59x speedup in training on a GPU compared to equivalent unstructured layers,' while §4.2.3 says 'training speedup of 1.59× compared to the dense equivalent at 90% sparsity.' Fig. 4's caption, in turn, says the diagonal-sparse kernel is compared to 'standard sparse kernels.' These are different comparisons. The paper should specify the exact baseline used for each reported speedup; if the 1.59x figure is versus dense, then the abstract's comparison to unstructured sparse layers is not supported by the reported measurements.
  3. [§4.1; Appendix E, Tables 9-11] The statistical testing procedure is reported inconsistently. Section 4.1 says McNemar tests compare the top-performing method at each sparsity level against all others, with bolding indicating no significant difference from the best. However, Tables 9-11 report p-values comparing each method specifically with RigL, not with the column best. For example, in Table 1 at ViT-B/16 60%, the best method is CHTs (79.88), but DynaDiag's p=0.0654 in Table 10 is versus RigL, not versus CHTs, so the bolding in Table 1 cannot be interpreted as the caption states. Additionally, for language experiments, Table 11 reports McNemar p-values for perplexity comparisons without defining the paired binary outcomes that McNemar's test requires. Please clarify the testing protocol or replace it with tests that support the stated bolding decisions.
  4. [Appendix B, Theorem 2] The claimed universal-approximation justification is not a valid proof. Lemma 1 establishes that each diagonal mask has at least one active entry per row and column, but Theorem 2 then concludes that universal approximation is retained because 'the assumptions required for universal approximation remain valid.' Full input-output coverage is neither necessary nor sufficient for the universal approximation property, and the subsequent rank-preservation argument does not repair the gap. If this appendix is intended as theoretical support for the method, it needs to be rewritten or stated explicitly as intuition; as written, it overstates what is proven.
minor comments (4)
  1. [§3.1, Eq. (1)] The notation is confusing: N is used both for the original column dimension of W ∈ R^{M×N} and then redefined as N = min(M,N). Please use a distinct symbol such as N_min for the smaller dimension.
  2. [Table 1; Table 16; §4.2.1] There are small typos: 'SRigl' in Table 1 should be 'SRigL', 'blocks.8.mlp.fc2.linear.weigh' in Table 16 is missing a 't', and §4.2.1 uses the phrase 'To the best of our knowledge, we are the first to show a DST method's performance training large and huge variants' which should be reworded for clarity.
  3. [§4.3.1, Fig. 5] The claim that LoRA-FA parameters are 'distributed across the weight matrix in an unstructured pattern' would be strengthened by a quantitative comparison with a random or diagonal baseline; the current figure alone does not establish that the pattern is the cause of the accuracy recovery.
  4. [§4.2.3] The text says 'Our PyTorch implementation does not exploit CUDA kernel optimizations' immediately after describing custom CUDA kernels; please clarify which components are implemented in PyTorch and which use the custom kernels, since this affects how the reported speedups should be interpreted.

Circularity Check

0 steps flagged · score 1.0 of 10

No load-bearing circularity: DynaDiag's accuracy and speedup claims are empirical measurements against external baselines; the diagonal parameterization (Eqs. 2-5) is learned from data labels, and no fitted parameter is renamed as a prediction.

full rationale

The paper's core claims are empirical rather than derived. Eq. 2-3 define a diagonal parameterization, Eq. 4-5 define a learnable TopK gate over diagonal offsets, and the reported accuracies/perplexities are measured on held-out test sets and compared with independently published baselines (RigL, SET, MEST, CHT, CHTs, SRigL, DSB, PixelatedBFly). No equation in the paper defines the target accuracy in terms of the fitted parameters, so the 'at par with unstructured sparsity' claim is not an identity or a fit renamed as a prediction. The closest thing to a self-referential move is Appendix I.1, where the authors compute the small-world factor sigma of their own trained networks and treat sigma >= 1 as confirmation of the small-world inspiration; this is a post-hoc diagnostic, not a load-bearing step in the argument, and it does not determine the accuracy or speedup results. Appendix B's universal-approximation and rank-preservation arguments are heuristic and contain a real mathematical gap (a sum of K diagonals has rank at most K, not min(m,n)), but invalid proof is a soundness concern, not a circularity. The paper also discloses the regime where the parity claim weakens (Table 1: ViT-B/16 at 95% sparsity, 69.54 vs RigL 71.50, p=0.0352 in Table 10; GPT2-M at 90%, p=0.0136) and Sec. 4.3.1 concedes a performance gap at sparsities >=80% that is closed only with unstructured LoRA-FA parameters; that concession undercuts the strength of the abstract's parity claim but does not make the derivation circular.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical or ontological entities are introduced. The 'pseudo-diagonal' is a formal definition (Eq. 1) rather than an invented object, and alpha is a standard auxiliary parameter vector. The free parameters listed are choices that affect reported accuracy and speedup numbers without being justified by derivation.

free parameters (4)
  • TopK temperature T schedule = cosine schedule, start/end values not reported
    Eq. 5 uses temperature in the softmax TopK; a cosine decay from high to low is chosen (Sec. 3.2, Apdx F.3); this is a hand-tuned balance of exploration vs exploitation.
  • Blocking similarity weight alpha = alpha < 0.5 (exact value not reported)
    Eq. 6 blends Jaccard index with diagonal proximity in the BCSR reordering; set by hand to favor diagonal structure over raw overlap.
  • l1 coefficient on alpha = not reported
    An l1 regularization on the importance vector alpha is applied to encourage sparsity in Sec. 3.2, but the coefficient value is not given.
  • LoRA-FA rank = rank 6 for ViT-B/16 at 80% sparsity
    Used in Sec. 4.3.1 to close the accuracy gap to RigL, requiring 1.67% extra parameters; this is a post-hoc choice rather than a derived quantity.
assumptions (4)
  • domain assumption Softmax-based TopK (Eq. 5) provides a faithful, optimizable relaxation of hard TopK diagonal selection.
    Underlies the claim that gradient descent on alpha discovers good diagonal sets; an alternative TopK (Sander et al.) was tested and discarded for being too slow.
  • standard math Diagonal masks with k >= 1 cover every row and every column (Lemma 1).
    Used for the full-coverage and universal approximation claims in Apdx. B; verified for the modulo construction, though the lemma's phrasing about 'exactly once' is loose for rectangular matrices.
  • ad hoc to paper Universal approximation is retained by diagonally sparse layers given full coverage and enough width and depth (Theorem 2).
    Apdx. B proves this only by citing full coverage plus a nonlinearity, which is an informal assertion; the cited references (Cybenko, Hornik) are missing from the bibliography.
  • ad hoc to paper Jaccard-plus-proximity BCSR reordering yields dense-enough blocks for GPU acceleration.
    Apdx. D modifies SmaT's Jaccard blocking with a diagonal proximity term; the effective alpha is hand-picked <0.5, and speedup evidence is provided only for one matrix size (Fig. 7).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dynamic Sparse Training of Diagonally Sparse Networks." pith.science (2026). https://pith.science/paper/Q2ERSVSM

@misc{pith2026250611449,
  author       = {Pith},
  title        = {Pith review of: Dynamic Sparse Training of Diagonally Sparse Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q2ERSVSM}},
  note         = {Machine review of arXiv:2506.11449}
}
read the original abstract

Recent advances in Dynamic Sparse Training (DST) have pushed the frontier of sparse neural network training in structured and unstructured contexts, matching dense-model performance while drastically reducing parameter counts to facilitate model scaling. However, unstructured sparsity often fails to translate into practical speedups on modern hardware. To address this shortcoming, we propose DynaDiag, a novel structured sparse-to-sparse DST method that performs at par with unstructured sparsity. DynaDiag enforces a diagonal sparsity pattern throughout training and preserves sparse computation in forward and backward passes. We further leverage the diagonal structure to accelerate computation via a custom CUDA kernel, rendering the method hardware-friendly. Empirical evaluations on diverse neural architectures demonstrate that our method maintains accuracy on par with unstructured counterparts while benefiting from tangible computational gains. Notably, with 90% sparse linear layers in ViTs, we observe up to a 3.13x speedup in online inference without sacrificing model performance and a 1.59x speedup in training on a GPU compared to equivalent unstructured layers. Our source code is available at https://github.com/horizon-research/DynaDiag/.

Figures

Figures reproduced from arXiv: 2506.11449 by the authors.

Figure 1
Figure 1. Comparing the inference (left) and training [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of different sparsity patterns from the [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Training with DynaDiag and diagonal sparsity. TopK induces sparsity in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Inference and training time of a ViT-Base at dif [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Performance of ViT-Base at 80% sparsity fine [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 7
Figure 7. Figure 7: Speedup obtained using our custom CUDA implementation while doing matrix-matrix multiplication with matrices [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Comparing the three different temperature schedules which affect the amount of non-zeros present at a training [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9: Comparing the inference speedup (left), training speedup (center), and Top-1 accuracy (right) of sparse training [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SHUFFLESPARSE: Learned Shuffles for Structured Sparse Networks

    cs.LG 2025-10 conditional novelty 5.0 of 10

    Learned per-layer permutations bring structured sparse training (block, N:M, diagonal) close to unstructured sparse-training accuracy at 90-95% sparsity, with inference speedups up to 2.9x.

Reference graph

Works this paper leans on

57 extracted references · 40 canonical work pages · cited by 1 Pith paper

  1. [1]

    and Albert, R

    Barab \'a si, A.-L. and Albert, R. Emergence of scaling in random networks. science, 286 0 (5439): 0 509--512, 1999

    1999

  2. [2]

    J., Frankle, J., and Guttag, J

    Blalock, D., Gonzalez Ortiz, J. J., Frankle, J., and Guttag, J. What is the state of neural network pruning? Proceedings of machine learning and systems, 2: 0 129--146, 2020

    work page 2020

  3. [3]

    Structured Pruning is All You Need for Pruning CNNs at Initialization

    Cai, Y., Hua, W., Chen, H., Suh, G. E., De Sa, C., and Zhang, Z. Structured pruning is all you need for pruning cnns at initialization. arXiv preprint arXiv:2203.02549, 2022

    work page Pith review arXiv 2022

  4. [4]

    Sparsity Winning Twice: Better Robust Generalization from More Efficient Training

    Chen, T., Zhang, Z., Wang, P., Balachandra, S., Ma, H., Wang, Z., and Wang, Z. Sparsity winning twice: Better robust generalization from more efficient training. arXiv preprint arXiv:2202.09844, 2022

    work page Pith review arXiv 2022

  5. [5]

    Which layer is learning faster? a systematic exploration of layer-wise convergence rate for deep neural networks

    Chen, Y., Yuille, A., and Zhou, Z. Which layer is learning faster? a systematic exploration of layer-wise convergence rate for deep neural networks. In The Eleventh International Conference on Learning Representations, 2023

    work page 2023

  6. [6]

    Trends in the dollar training cost of machine learning systems

    Cottier, B. Trends in the dollar training cost of machine learning systems. Epoch. January, 31: 0 2023, 2023

    work page 2023

  7. [7]

    Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models

    Dao, T., Chen, B., Liang, K., Yang, J., Song, Z., Rudra, A., and Re, C. Pixelated butterfly: Simple and efficient sparse training for neural network models. arXiv preprint arXiv:2112.00029, 2021

    work page Pith review arXiv 2021

  8. [8]

    Imagenet: A large-scale hierarchical image database

    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.\ 248--255. Ieee, 2009

    2009

Show all 57 references
  1. [9]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    2010 arXiv

  2. [10]

    S., and Elsen, E

    Evci, U., Gale, T., Menick, J., Castro, P. S., and Elsen, E. Rigging the lottery: Making all tickets winners. In International conference on machine learning, pp.\ 2943--2952. PMLR, 2020

    2020

  3. [11]

    and Carbin, M

    Frankle, J. and Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635, 2018

    2018 arXiv

  4. [12]

    Learning both weights and connections for efficient neural network

    Han, S., Pool, J., Tran, J., and Dally, W. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015

    2015

  5. [13]

    Accelerating transformer pre-training with 2: 4 sparsity

    Hu, Y., Zhao, K., Huang, W., Chen, J., and Zhu, J. Accelerating transformer pre-training with 2: 4 sparsity. arXiv preprint arXiv:2404.01847, 2024

    2024 arXiv

  6. [14]

    Accelerated sparse neural training: A provable and efficient method to find n: m transposable masks

    Hubara, I., Chmiel, B., Island, M., Banner, R., Naor, J., and Soudry, D. Accelerated sparse neural training: A provable and efficient method to find n: m transposable masks. Advances in neural information processing systems, 34: 0 21099--21111, 2021

    2021

  7. [15]

    K., Ma, H., Chen, T., Ding, Y., and Wang, Z

    Jaiswal, A. K., Ma, H., Chen, T., Ding, Y., and Wang, Z. Training your sparse neural network better with any mask. In International Conference on Machine Learning, pp.\ 9833--9844. PMLR, 2022

    2022

  8. [16]

    Top-kast: Top-k always sparse training

    Jayakumar, S., Pascanu, R., Rae, J., Osindero, S., and Elsen, E. Top-kast: Top-k always sparse training. Advances in Neural Information Processing Systems, 33: 0 20744--20754, 2020

    2020

  9. [17]

    Advancing dynamic sparse training by exploring optimization opportunities

    Ji, J., Li, G., Yin, L., Qin, M., Yuan, G., Guo, L., Liu, S., and Ma, X. Advancing dynamic sparse training by exploring optimization opportunities. In Forty-First International Conference on Machine Learning, 2024

    2024

  10. [18]

    Exposing and exploiting fine-grained block structures for fast and accurate sparse training

    Jiang, P., Hu, L., and Song, S. Exposing and exploiting fine-grained block structures for fast and accurate sparse training. Advances in Neural Information Processing Systems, 35: 0 38345--38357, 2022

    2022

  11. [19]

    and Hinton, G

    Krizhevsky, A. and Hinton, G. Learning multiple layers of features from tiny images. Technical Report TR-2009, University of Toronto, 2009

    2009

  12. [20]

    Accurate neural network pruning requires rethinking sparse optimization

    Kuznedelev, D., Kurtic, E., Iofinova, E., Frantar, E., Peste, A., and Alistarh, D. Accurate neural network pruning requires rethinking sparse optimization. arXiv preprint arXiv:2308.02060, 2023

    2023 arXiv

  13. [21]

    S., Bernaschi, M., Nutt, W., Silvestri, F., and Vella, F

    Labini, P. S., Bernaschi, M., Nutt, W., Silvestri, F., and Vella, F. Blocking sparse matrices to leverage dense-specific multiplication. In 2022 IEEE/ACM Workshop on Irregular Applications: Architectures and Algorithms (IA3), pp.\ 19--24. IEEE, 2022

    2022

  14. [22]

    Crafting papers on machine learning

    Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp.\ 1207--1216, Stanford, CA, 2000. Morgan Kaufmann

    2000

  15. [23]

    Dynamic sparse training with structured sparsity

    Lasby, M., Golubeva, A., Evci, U., Nica, M., and Ioannou, Y. Dynamic sparse training with structured sparsity. arXiv preprint arXiv:2305.02299, 2023

    2023 arXiv

  16. [24]

    Lee, N., Ajanthan, T., and Torr, P. H. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810.02340, 2018

    2018 arXiv

  17. [25]

    Towards optimal structured cnn pruning via generative adversarial learning

    Lin, S., Ji, R., Yan, C., Zhang, B., Cao, L., Ye, Q., Huang, F., and Doermann, D. Towards optimal structured cnn pruning via generative adversarial learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 2790--2799, 2019

    2019

  18. [26]

    and Wang, Z

    Liu, S. and Wang, Z. Ten lessons we have learned in the new" sparseland. A short handbook for sparse neural network researchers, 2023

    2023

  19. [27]

    C., and Pechenizkiy, M

    Liu, S., Mocanu, D. C., and Pechenizkiy, M. On improving deep learning generalization with adaptive sparse connectivity. arXiv preprint arXiv:1906.11626, 2019

    1906 arXiv

  20. [28]

    W., and Yang, Y

    Lu, H., Zhou, Y., Liu, S., Wang, Z., Mahoney, M. W., and Yang, Y. Alphapruning: Using heavy-tailed self regularization theory for improved layer-wise pruning of large language models. arXiv preprint arXiv:2410.10912, 2024

    2024 arXiv

  21. [29]

    Ai beats humans for the first time in physical skill game

    Lykiardopoulou, I. Ai beats humans for the first time in physical skill game. https://www.newscientist.com/article/2402645, 12 2023. Accessed: 20 01 2025

    2023

  22. [30]

    Marcus, M., Santorini, B., and Marcinkiewicz, M. A. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19 0 (2): 0 313--330, 1993

    1993

  23. [31]

    Pointer sentinel mixture models

    Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016

    2016 arXiv

  24. [32]

    A., Pool, J., Stosic, D., Stosic, D., Venkatesh, G., Yu, C., and Micikevicius, P

    Mishra, A., Latorre, J. A., Pool, J., Stosic, D., Stosic, D., Venkatesh, G., Yu, C., and Micikevicius, P. Accelerating sparse deep neural networks. arXiv preprint arXiv:2104.08378, 2021

    2021 arXiv

  25. [33]

    C., Mocanu, E., Stone, P., Nguyen, P

    Mocanu, D. C., Mocanu, E., Stone, P., Nguyen, P. H., Gibescu, M., and Liotta, A. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9 0 (1): 0 2383, 2018

    2018

  26. [34]

    Variational dropout sparsifies deep neural networks

    Molchanov, D., Ashukha, A., and Vetrov, D. Variational dropout sparsifies deep neural networks. In International conference on machine learning, pp.\ 2498--2507. PMLR, 2017

    2017

  27. [35]

    Pruning convolutional neural networks for resource efficient inference

    Molchanov, P., Tyree, S., Karras, T., Aila, T., and Kautz, J. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440, 2016

    2016 arXiv

  28. [36]

    Importance estimation for neural network pruning

    Molchanov, P., Mallya, A., Tyree, S., Frosio, I., and Kautz, J. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 11264--11272, 2019

    2019

  29. [37]

    and Wang, X

    Mostafa, H. and Wang, X. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning, pp.\ 4646--4655. PMLR, 2019

    2019

  30. [38]

    S., Besta, M., Vella, F., and Hoefler, T

    Okanovic, P., Kwasniewski, G., Labini, P. S., Besta, M., Vella, F., and Hoefler, T. High performance unstructured spmm computation using tensor cores. In SC24: International Conference for High Performance Computing, Networking, Storage and Analysis, pp.\ 1--14. IEEE, 2024

    2024

  31. [39]

    Language models are unsupervised multitask learners

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  32. [40]

    E., Puigcerver, J., Djolonga, J., Peyr \'e , G., and Blondel, M

    Sander, M. E., Puigcerver, J., Djolonga, J., Peyr \'e , G., and Blondel, M. Fast, differentiable and sparse top-k: a convex analysis perspective. In International Conference on Machine Learning, pp.\ 29919--29936. PMLR, 2023

    2023

  33. [41]

    Game-playing deepmind ai can beat top humans at chess, go and poker

    Sparkes, M. Game-playing deepmind ai can beat top humans at chess, go and poker. https://thenextweb.com/news/, 11 2023. Accessed: 20 01 2025

    2023

  34. [42]

    Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695, 2023

    2023 arXiv

  35. [43]

    L., and Ganguli, S

    Tanaka, H., Kunin, D., Yamins, D. L., and Ganguli, S. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in neural information processing systems, 33: 0 6377--6389, 2020

    2020

  36. [44]

    K., Joyce, K

    Telesford, Q. K., Joyce, K. E., Hayasaka, S., Burdette, J. H., and Laurienti, P. J. The ubiquity of small-world networks. Brain connectivity, 1 0 (5): 0 367--375, 2011

    2011

  37. [45]

    O., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung, J., Steiner, A., Keysers, D., Uszkoreit, J., et al

    Tolstikhin, I. O., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung, J., Steiner, A., Keysers, D., Uszkoreit, J., et al. Mlp-mixer: An all-mlp architecture for vision. Advances in neural information processing systems, 34: 0 24261--24272, 2021

    2021

  38. [46]

    Picking winning tickets before training by preserving gradient flow

    Wang, C., Zhang, G., and Grosse, R. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.07376, 2020

    2002 arXiv

  39. [47]

    Watts, D. J. and Strogatz, S. H. Collective dynamics of `small-world' networks. nature, 393 0 (6684): 0 440--442, 1998

    1998

  40. [48]

    and Busato, F

    Yamaguchi, T. and Busato, F. Accelerating matrix multiplication with block sparse format and nvidia tensor cores. NVIDIA Developer Technical Blog, https://developer. nvidia. com/blog/accelerating-matrixmultiplication-with-block-sparse-format-and-nvidia-tensor-cores, 2021

    2021

  41. [49]

    Pruning before training may improve generalization, provably

    Yang, H., Liang, Y., Guo, X., Wu, L., and Wang, Z. Pruning before training may improve generalization, provably. arXiv preprint arXiv:2301.00335, 2023 a

    2023 arXiv

  42. [50]

    Global vision transformer pruning with hessian-aware saliency

    Yang, H., Yin, H., Shen, M., Molchanov, P., Li, H., and Kautz, J. Global vision transformer pruning with hessian-aware saliency. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 18547--18557, 2023 b

    2023

  43. [51]

    Width & depth pruning for vision transformers

    Yu, F., Huang, K., Wang, M., Cheng, Y., Chu, W., and Cui, L. Width & depth pruning for vision transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.\ 3143--3151, 2022

    2022

  44. [52]

    Mest: Accurate and fast memory-economic sparse training framework on the edge

    Yuan, G., Ma, X., Niu, W., Li, Z., Kong, Z., Liu, N., Gong, Y., Zhan, Z., He, C., Jin, Q., et al. Mest: Accurate and fast memory-economic sparse training framework on the edge. Advances in Neural Information Processing Systems, 34: 0 20838--20850, 2021

    2021

  45. [53]

    Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning

    Zhang, L., Zhang, L., Shi, S., Chu, X., and Li, B. Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning. arXiv preprint arXiv:2308.03303, 2023 a

    2023 arXiv

  46. [54]

    M., Yan, G., and Li, X

    Zhang, X.-J., Moore, J. M., Yan, G., and Li, X. Universal structural patterns in sparse recurrent neural networks. Communications Physics, 6 0 (1): 0 243, 2023 b

    2023

  47. [55]

    Zhang, Y., Zhao, J., Wu, W., Muscoloni, A., and Cannistraci, C. V. Epitopological learning and cannistraci-hebb network shape intelligence brain-inspired theory for ultra-sparse advantage in deep learning. In The Twelfth International Conference on Learning Representations, 2024

    2024

  48. [56]

    Zhang, Y., Zhao, J., Wu, W., Liao, Z., Michieli, U., and Cannistraci, C. V. Brain-inspired sparse training enables transformers and llms to perform as fully connected. arXiv preprint arXiv:2501.19107, 2025

    2025

  49. [57]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.