Pith. sign in

REVIEW 3 major objections 5 minor 60 references

CoNNect: Connectivity-Based Regularization for Structural Pruning

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces CoNNect, a differentiable regularizer maximizing normalized input-to-output path weight, claiming it approximates L0 pruning, keeps only sparse connected structures stable, and improves one-shot structural pruning in…

desk verdict The normalized path-product regularizer is a real idea with credible small-scale evidence, but the headline large-scale results use an unnormalized variant that the theory does not cover, leaving the paper's central claims overbroad. read the letter →

arxiv 2502.00744 v2 pith:6WIUY6G5 submitted 2025-02-02 cs.LG

Christian Franssen , Jinyang Jiang , Yijie Peng , Bernd Heidergott This is my paper · ORCID
classification cs.LG
keywords neuralnetworkpruningstructuralL0regularizationconnectivityregularizerSynFlowone-shotLLMdifferentiable
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CoNNect is a differentiable regularizer that replaces the L0 norm in sparse neural-network training with a connectivity objective: it maximizes the total normalized weight of all input-to-output paths. The paper proves that for networks with more than two layers, any global minimizer of this objective has at most $|V_1| + |V_K| + K - 3$ non-zero weights, mirroring the sparsity that L0 regularization would enforce. Because the regularizer is the logarithm of the path sum, a network that is about to disconnect sends the penalty to infinity, which prevents layer collapse during pruning. The authors show that CoNNect improves classical magnitude and SynFlow pruning after training, and that plugging it into DepGraph and LLM-pruner improves one-shot structural pruning of ResNet-56, VGG-19, and LLaMA-7B.

What carries the argument

The central object is the connectivity matrix $\phi(W)=\sum_{k=1}^K (\theta(W))^k$, where $(\theta(W))_{i,j}$ is the weight $W_{i,j}$ divided by the sum of absolute weights in its layer. The CoNNect regularizer is $R(W)=-\phi_{tot}(W)$, the negative sum of all entries connecting input layer $V_1$ to output layer $V_K$; equivalently, it is the negative sum of products of normalized weights over all input-to-output paths. This path-product form is what makes the objective prefer one strong path over many weak parallel paths, and the log transform turns imminent disconnection into an infinite penalty.

What would settle it

Train a small fully connected network with $K=4$, one input, and one output on a task that needs two separate features and measure the number of non-zero internal weights at convergence under $-\log(\phi_{tot}(W))$; if it consistently exceeds the $K-3$ bound, the L0-approximation claim fails, and for the large-scale variant one can check scale-invariance by multiplying all weights by a constant and seeing whether the unnormalized path product changes by that constant raised to $K-1$.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a single scalar computed by one forward pass, the sum over all input-to-output paths of products of layer-wise normalized weights $\phi_{tot}(W)$, acts as an L0 surrogate for pruning. Theorem 3.1 shows that minimizing $-\phi_{tot}(W)$ forces all but $|V_1|+|V_K|+K-3$ weights to zero, and Theorem 3.3 (proof omitted) asserts that the only stable stationary points are global minimizers, so gradient descent cannot get stuck in a disconnected or overly dense configuration. Using $-\log(\phi_{tot}(W))$ makes disconnection infinitely costly, which is what prevents layer collapse. The paper further claims that this connectivity signal transfers to structured pruning through channel scaling factors and improves the importance scores of one-shot pruners.

Load-bearing premise

The load-bearing premise is that the layer-wise normalized objective of Eq. (2) is what training actually optimizes, together with the unproved stability theorem; the large-scale experiments instead use unnormalized absolute weights, so it is open whether the L0-approximation and stable-stationary-point guarantees carry over.

Editorial extensions

If this is right

  • Networks trained with the log-CoNNect penalty can be hard-pruned to very high ratios without layer collapse; in the paper's small MLP experiments this finds the two relevant input features that magnitude-only and L1-trained networks miss.
  • The same objective extends to channels by inserting learnable scaling factors, so CoNNect gives a structured importance score $I_{k,c}$ for pruning whole channels.
  • Plugged into DepGraph, CoNNect improves one-shot structural pruning of ResNet-56 and VGG-19, with larger gains at extreme 16x speed-up targets.
  • Plugged into LLM-pruner, CoNNect improves zero-shot perplexity and classification after pruning 20% and 40% of LLaMA-7B (and LLaMA-13B in the appendix), with or without LoRA fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same connectivity score could be computed at initialization for structured one-shot pruning of untrained models, giving a data-free group-importance signal.
  • The path-product formulation connects CoNNect to path-norm theory; one could derive closed-form sensitivity of $\phi_{tot}$ to whole modules such as attention heads or MLP blocks and use that as a group score for transformer pruning.
  • For tasks requiring genuinely parallel features, a natural extension is a multi-path variant that keeps a small number of top paths rather than a single path, which would trade the L0-style bound against representational capacity.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces CoNNect, a differentiable regularizer based on the sum of weighted input-to-output paths in a neural network, and claims that it approximates L0 regularization while guaranteeing maximally connected network structures and avoiding layer collapse. Section 3.3.1 defines a layer-normalized connectivity measure θ(W) and proves (or sketches proofs of) three results: Theorem 3.1 bounding the number of nonzero weights at minimizers of -φ_tot, Lemma 3.2 characterizing stationary points, and Theorem 3.3 claiming all stable stationary points are global minimizers. Section 3.3.2 extends the idea to channel-level scaling. Experiments cover unstructured pruning of small MLPs, channel pruning of GNNs, and integration into DepGraph and LLM-pruner for ResNet-56, VGG-19, and LLaMA-7B pruning.

Significance. If the theory and experiments are properly connected, CoNNect would be a useful addition to the pruning toolbox: the regularizer is simple, differentiable, computationally cheap (a single forward pass), and the small-scale experiments show meaningful improvements over L1 and L2 regularization with ablations over initializations and coefficients. The integration with DepGraph and LLM-pruner addresses practically important settings. However, the theoretical guarantees are established only for a normalized version of the connectivity measure, while the large-scale experiments switch to an unnormalized version, and Theorem 3.3 is explicitly left unproved. The paper therefore currently overstates the extent to which its advertised guarantees apply to its headline results.

major comments (3)
  1. [Section 4.2 vs. Eq. (2); Tables 1-2] The theoretical results in Theorems 3.1-3.3 are derived for the layer-normalized θ(W) defined in Eq. (2), where each edge weight is divided by the sum of weights in its layer. In Section 4.2, the paper explicitly redefines (θ(W))_{i,j} = |W_{i,j}| for the DepGraph and LLM-pruner integrations and sets biases to |b|. This unnormalized variant is outside the theory: with raw weights, φ_tot(W) is a sum of products of unscaled weights, so multiplying all weights by a constant c scales every path by c^{K-1}, making -log φ_tot(W) unbounded below and the minimization in Theorem 3.1 not well-posed. Moreover, the L0-type sparsity preference fails qualitatively; for a simple 3-layer 2-2-1 network, the dense all-ones configuration gives φ_tot=2 and R=-log 2≈-0.69, while the sparse single-path configuration gives φ_tot=1 and R=0, so the unnormalized regularizer actually prefers the dense network. Consequently, the ResNet-56, VGG-19, and LLaMA-7B results in Tables 1-2 are produced by a heuristic connectivity score, not by the regularizer whose guarantees are advertised in the abstract. The paper must either extend the theory to the unnormalized objective or explicitly re-scope the claims made for these experiments.
  2. [Section 3.3.1, Theorem 3.3] Theorem 3.3 is a load-bearing claim for the paper's assertion that CoNNect is a 'well-behaved regularizer' with no stable stationary points except global optima, yet its proof is omitted in the text: the sentence 'we the proof is omitted as it follows directly form the previous observation' is not a proof. In addition, the notion of 'stable stationary point' is never formally defined (e.g., isolated local minimum, infinitesimal stability under gradient flow, or other). Since Lemma 3.2's proof is itself only an induction sketch, the combined chain of reasoning is not sufficiently verifiable. A complete proof of Theorem 3.3 and a precise definition of stability are required.
  3. [Section 3.3.1, Theorem 3.1 and Appendix A.1] The claim that CoNNect 'approximates L0 regularization' rests on Theorem 3.1, but the theorem's bound is |V1|+|VK|+K-3 nonzero weights, whereas an L0-constrained network with connectivity has K-1 nonzero weights. The difference can be large for networks with many input or output nodes, and the paper's justification ('negligible in practice when most input nodes contribute...') is an empirical assumption, not a proven property. Furthermore, the proof in Appendix A.1 argues that a maximizer can be assumed to have a single path with all mass, but this is asserted rather than derived; the subsequent calculation shows only that shifting mass within the first layer preserves φ_tot=1, not that every maximizer has the claimed form. The theorem may be true, but the proof as written is a sketch and should be completed or clearly labeled as such.
minor comments (5)
  1. [Section 3.3.1] There is a typo in the sentence introducing Theorem 3.3: 'we the proof is omitted as it follows directly form the previous observation' should be 'the proof is omitted as it follows directly from the previous observation.'
  2. [Section 3.3.1, Eq. (6)] Equation (6) contains an unbalanced parenthesis: '− log( φtot(W )) )' has an extra closing parenthesis.
  3. [Figure 3] The caption states that thicker and darker colors correspond to stronger values, but the grayscale rendering makes it difficult to distinguish positive and negative edges; consider adding a legend or using distinct line styles.
  4. [Tables 1 and 2] The large-scale pruning results are reported as single runs without variance or multiple seeds; given the stochasticity of calibration sampling and fine-tuning, error bars or repeated runs would strengthen the empirical claims.
  5. [Appendix C.4] The description of connectivity input sampling is inconsistent with the main text: Section 4.2 says inputs are 'uniformly sampled between 0 and 1', while Appendix C.4 says 'uniformly sampled between 0 and the vocabulary size' for the LLM experiment. Please clarify which value is used.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the L0-approximation is a genuine theorem about the defined objective, and the Section 4.2 experiments use a stated variant whose mismatch with the theory is an experimental-span issue, not a circular reduction.

full rationale

I examined the derivation chain for circular reductions. The core claim is Theorem 3.1: minimizing -phi_tot(W), with the layer-wise normalized theta(W) of Eq. (2), yields solutions with at most |V1|+|VK|+K-3 nonzero weights. This is a nontrivial mathematical consequence of the normalization: maximizing the sum of normalized path products concentrates mass onto a single path, yielding sparsity. The proof in Appendix A.1 explicitly derives the bound, so the L0-approximation is not assumed or fitted; it is a theorem. The 'maximally connected' phrasing is definitional in that phi_tot is defined as the connectivity measure and R=-phi_tot is its negative, but the paper does not present this as an independent prediction; it is the construction of the regularizer. The stable-stationary-point claim (Theorem 3.3) is asserted with an omitted proof, but an omitted proof is an evidence gap, not circularity. There are no self-citations: no load-bearing premise is sourced to the authors' own prior work. The Section 4.2 experiments replace theta(W) with |W|, which invalidates the theoretical guarantees for that variant; however, the empirical gains are measured directly and do not reduce to the normalized-objective theorems by construction. This is a theory-experiment mismatch, not a circular step where the prediction is equivalent to its input. The regularizer coefficients are tuned, but the reported accuracies are genuine out-of-sample results against DepGraph and LLM-Pruner baselines. Overall, I find no step where a claimed result is identical to a fitted parameter or to a definition, so the paper is not circular.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central claim rests on the regularizer's definition and on several unproven or empirical assumptions. No free parameters are fundamentally new, but the regularizer coefficients are tuned per experiment, and the unnormalized variant is a significant unmotivated change.

free parameters (3)
  • lambda2 (CoNNect coefficient) = 0.1 (MLP); selected from {10, 1, 0.1, 0.01} (GNN); 1e5 (LLM); unspecified (ResNet/VGG)
    Balances connectivity against loss; main results depend on it and no sensitivity analysis is reported for CNN/LLM.
  • lambda1 (L1 coefficient) = 1e-3 (MLP); selected from {1e-2, ..., 1e-6} (GNN)
    Baseline regularizer strength.
  • lambda3 (weight decay) = 5e-4 (MLP); 1e-3 or 1e-4 (GNN)
    Standard L2 penalty included in objective (7).
assumptions (6)
  • domain assumption Neural networks can be treated as directed weighted graphs with nonnegative weights (using absolute values).
    Section 3.1 states the graph representation; absolute values are used in Eq. (2).
  • domain assumption phi_tot, the sum of normalized path products, is a meaningful connectivity measure whose maximization preserves signal flow.
    Section 3.3.1 defines the regularizer as -log(phi_tot); this is the core design hypothesis.
  • ad hoc to paper Adding -lambda log(phi_tot) to the training loss improves post-pruning accuracy.
    There is no theoretical connection between the regularizer and downstream pruning, only empirical evidence.
  • ad hoc to paper All stable stationary points of -phi_tot are global minimizers (Theorem 3.3).
    The proof is omitted in the text; used to argue no bad local minima.
  • ad hoc to paper The unnormalized theta=|W| variant used in Section 4.2 inherits the theoretical guarantees.
    Unproven; the theorems rely on layer normalization in Eq. (2).
  • domain assumption Activation functions, batch norm, and pooling can be ignored or replaced in the connectivity computation.
    Appendix B describes these simplifications; they assume connectivity is structurally unchanged.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CoNNect: Connectivity-Based Regularization for Structural Pruning." pith.science (2026). https://pith.science/paper/6WIUY6G5

@misc{pith2026250200744,
  author       = {Pith},
  title        = {Pith review of: CoNNect: Connectivity-Based Regularization for Structural Pruning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6WIUY6G5}},
  note         = {Machine review of arXiv:2502.00744}
}
abstract

Pruning encompasses a range of techniques aimed at increasing the sparsity of neural networks (NNs). These techniques can generally be framed as minimizing a loss function subject to an $L_0$ norm constraint. This paper introduces CoNNect, a novel differentiable regularizer for sparse NN training that ensures connectivity between input and output layers. We prove that CoNNect approximates $L_0$ regularization, guaranteeing maximally connected network structures while avoiding issues like layer collapse. Moreover, CoNNect is easily integrated with established structural pruning strategies. Numerical experiments demonstrate that CoNNect can improve classical pruning strategies and enhance state-of-the-art one-shot pruners, such as DepGraph and LLM-pruner.

Figures

Figures reproduced from arXiv: 2502.00744 by the authors.

Figure 1
Figure 1. Magnitude-based pruning of NN (left) leads to layer collapse (right). [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of CNN with the scaling factor. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Trained (top) and fine-tuned (bottom) models. Thicker and darker colors correspond to stronger [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Accuracies of GNNs for given pruning ratios. [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: (a)-(b) Learning curves for solving Equation (4). SynFlow pruning happens at iteration 200. [PITH_FULL_IMAGE:figures/full_fig_p021_5.png]
Figure 6
Figure 6. Figure 6: Fine-tuned accuracy after magnitude and SynFlow pruning for different regularization settings [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]
Figure 7
Figure 7. Figure 7: Accuracies of GNNs for given compression ratios under [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 8
Figure 8. Figure 8: Accuracies of GNNs for given compression ratios under [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]
Figure 9
Figure 9. Figure 9: Accuracies of GNNs for given compression ratios under CoNNect regularization, for [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]
Figure 10
Figure 10. Figure 10: Accuracies of GNNs for given compression ratios under CoNNect regularization, for [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 19 canonical work pages

  1. [1]

    Structured pruning of deep convolutional neural networks

    Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruning of deep convolutional neural networks. ACM Journal on Emerging Technologies in Computing Systems (JETC), 13 0 (3): 0 1--18, 2017

    work page 2017

  2. [2]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp.\ 7432--7439, 2020

    2020

  3. [3]

    A survey on deep neural network pruning: Taxonomy, comparison, analysis, and recommendations

    Hongrong Cheng, Miao Zhang, and Javen Qinfeng Shi. A survey on deep neural network pruning: Taxonomy, comparison, analysis, and recommendations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  4. [4]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers),...

    work page 2019

  5. [5]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    arXiv 2018

  6. [6]

    Neural network training using l_1 -regularization and bi-fidelity data

    Subhayan De and Alireza Doostan. Neural network training using l_1 -regularization and bi-fidelity data. Journal of Computational Physics, 458: 0 111010, 2022

    work page 2022

  7. [7]

    Depgraph: Towards any structural pruning

    Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. Depgraph: Towards any structural pruning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 16091--16101, 2023

    work page 2023

  8. [8]

    Maskllm: Learnable semi-structured sparsity for large language models

    Gongfan Fang, Hongxu Yin, Saurav Muralidharan, Greg Heinrich, Jeff Pool, Jan Kautz, Pavlo Molchanov, and Xinchao Wang. Maskllm: Learnable semi-structured sparsity for large language models. arXiv preprint arXiv:2409.17481, 2024

    arXiv 2024

Show all 60 references
  1. [9]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks

    Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635, 2018

    2018 arXiv

  2. [10]

    Sparsegpt: Massive language models can be accurately pruned in one-shot, 2023

    Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot, 2023. URL https://arxiv.org/abs/2301.00774

    2023 arXiv

  3. [11]

    The state of sparsity in deep neural networks

    Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.09574, 2019

    1902 arXiv

  4. [12]

    A framework for few-shot language model evaluation

    Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. Version v0. 0.1. Sept, 10: 0 8--9, 2021

    2021

  5. [13]

    Removal of hidden units and weights for back propagation networks

    Masafumi Hagiwara. Removal of hidden units and weights for back propagation networks. In Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan), volume 1, pp.\ 351--354. IEEE, 1993

    1993

  6. [14]

    Learning both weights and connections for efficient neural network

    Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015

    2015

  7. [15]

    Optimal brain surgeon and general network pruning

    Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In IEEE international conference on neural networks, pp.\ 293--299. IEEE, 1993

    1993

  8. [16]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

    2016

  9. [17]

    Structured pruning for deep convolutional neural networks: A survey

    Yang He and Lingao Xiao. Structured pruning for deep convolutional neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 2023

    2023

  10. [18]

    Channel pruning for accelerating very deep neural networks

    Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, pp.\ 1389--1397, 2017

    2017

  11. [19]

    A practical guide to training restricted boltzmann machines

    Geoffrey E Hinton. A practical guide to training restricted boltzmann machines. In Neural Networks: Tricks of the Trade: Second Edition, pp.\ 599--619. Springer, 2012

    2012

  12. [20]

    Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks

    Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22 0 (241): 0 1--124, 2021

    2021

  13. [21]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  14. [22]

    Data-driven sparse structure selection for deep neural networks

    Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks. arXiv, 2017. doi:10.48550/arxiv.1707.01213

    2017 doi arXiv

  15. [23]

    Top-kast: Top-k always sparse training

    Siddhant Jayakumar, Razvan Pascanu, Jack Rae, Simon Osindero, and Erich Elsen. Top-kast: Top-k always sparse training. Advances in Neural Information Processing Systems, 33: 0 20744--20754, 2020

    2020

  16. [24]

    A new status index derived from sociometric analysis

    Leo Katz. A new status index derived from sociometric analysis. Psychometrika, 18 0 (1): 0 39--43, 1953

    1953

  17. [25]

    Semi-supervised classification with graph convolutional networks

    Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016

    2016 arXiv

  18. [26]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009

    2009

  19. [27]

    Optimal brain damage

    Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989

    1989

  20. [28]

    Snip: Single-shot network pruning based on connection sensitivity

    Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810.02340, 2018

    2018 arXiv

  21. [29]

    A pruning feedforward small-world neural network based on katz centrality for nonlinear system modeling

    Wenjing Li, Minghui Chu, and Junfei Qiao. A pruning feedforward small-world neural network based on katz centrality for nonlinear system modeling. Neural Networks, 130: 0 269--285, 2020

    2020

  22. [30]

    Fixing weight decay regularization in adam

    Ilya Loshchilov, Frank Hutter, et al. Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.05101, 5, 2017

    2017 arXiv

  23. [31]

    Llm-pruner: On the structural pruning of large language models

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36: 0 21702--21720, 2023

    2023

  24. [32]

    Building a large annotated corpus of english: The penn treebank

    Mitch Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19 0 (2): 0 313--330, 1993

    1993

  25. [33]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2022

    2022

  26. [34]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp.\ 2381--2391, 2018

    2018

  27. [35]

    Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science

    Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9 0 (1): 0 2383, 2018

    2018

  28. [36]

    Pruning convolutional neural networks for resource efficient inference

    Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440, 2016

    2016 arXiv

  29. [37]

    Path-sgd: Path-normalized optimization in deep neural networks

    Behnam Neyshabur, Russ R Salakhutdinov, and Nati Srebro. Path-sgd: Path-normalized optimization in deep neural networks. Advances in neural information processing systems, 28, 2015

    2015

  30. [38]

    Carbon emissions and large neural network training

    David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint arXiv:2104.10350, 2021

    2021 arXiv

  31. [39]

    An analysis of the regularization between l2 and dropout in single hidden layer neural network

    Ekachai Phaisangittisagul. An analysis of the regularization between l2 and dropout in single hidden layer neural network. In 2016 7th International Conference on Intelligent Systems, Modelling and Simulation (ISMS), pp.\ 174--179. IEEE, 2016

    2016

  32. [40]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64 0 (9): 0 99--106, 2021

    2021

  33. [41]

    Movement pruning: Adaptive sparsity by fine-tuning

    Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine-tuning. Advances in neural information processing systems, 33: 0 20378--20389, 2020

    2020

  34. [42]

    Collective classification in network data

    Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Collective classification in network data. AI Magazine, 29 0 (3): 0 93--93, 2008

    2008

  35. [43]

    Very deep convolutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR), 2015

    2015

  36. [44]

    A simple and effective pruning approach for large language models

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695, 2023

    2023 arXiv

  37. [45]

    Pruning neural networks without any data by iteratively conserving synaptic flow

    Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in neural information processing systems, 33: 0 6377--6389, 2020

    2020

  38. [46]

    Stanford alpaca: An instruction-following llama model, 2023

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-following llama model, 2023

    2023

  39. [47]

    Evaluating pruning methods

    Georg Thimm and Emile Fiesler. Evaluating pruning methods. In Proceedings of the International Symposium on Artificial neural networks, pp.\ 20--25, 1995

    1995

  40. [48]

    Regression shrinkage and selection via the lasso

    Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58 0 (1): 0 267--288, 1996

    1996

  41. [49]

    Open and efficient foundation language models

    H Touvron, T Lavril, G Izacard, X Martinet, MA Lachaux, T Lacroix, B Rozi \`e re, N Goyal, E Hambro, F Azhar, et al. Open and efficient foundation language models. Preprint at arXiv. https://doi. org/10.48550/arXiv, 2302, 2023

    doi

  42. [50]

    Connectivity matters: Neural network pruning through the lens of effective sparsity

    Artem Vysogorets and Julia Kempe. Connectivity matters: Neural network pruning through the lens of effective sparsity. Journal of Machine Learning Research, 24 0 (99): 0 1--23, 2023

    2023

  43. [51]

    Picking winning tickets before training by preserving gradient flow

    Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.07376, 2020

    2002 arXiv

  44. [52]

    Learning structured sparsity in deep neural networks

    Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. Advances in neural information processing systems, 29, 2016

    2016

  45. [53]

    Structured pruning of convolutional neural networks via l1 regularization

    Chen Yang, Zhenghong Yang, Abdul Mateen Khattak, Liu Yang, Wenxin Zhang, Wanlin Gao, and Minjuan Wang. Structured pruning of convolutional neural networks via l1 regularization. IEEE Access, 7: 0 106385--106394, 2019

    2019

  46. [54]

    Model selection and estimation in regression with grouped variables

    Ming Yuan and Yi Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society Series B: Statistical Methodology, 68 0 (1): 0 49--67, 2006

    2006

  47. [55]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  48. [56]

    Subset-based training and pruning of sigmoid neural networks

    Guian Zhou and Jennie Si. Subset-based training and pruning of sigmoid neural networks. Neural networks, 12 0 (1): 0 79--89, 1999

    1999

  49. [57]

    Aligning books and movies: Towards story-like visual explanations by watching movies and reading books

    Yukun Zhu. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. arXiv preprint arXiv:1506.06724, 2015

    2015 arXiv

  50. [58]

    Neuron-level structured pruning using polarization regularizer

    Tao Zhuang, Zhixuan Zhang, Yuheng Huang, Xiaoyi Zeng, Kai Shuang, and Xiang Li. Neuron-level structured pruning using polarization regularizer. Advances in neural information processing systems, 33: 0 9865--9877, 2020

    2020

  51. [59]

    spred: Solving l1 penalty with sgd

    Liu Ziyin and Zihao Wang. spred: Solving l1 penalty with sgd. In International Conference on Machine Learning, pp.\ 43407--43422. PMLR, 2023

    2023

  52. [60]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.