Pith. sign in

REVIEW 12 cited by

Picking Winning Tickets Before Training by Preserving Gradient Flow

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2002.07376 v2 pith:3MHGN2QP submitted 2020-02-18 cs.LG cs.CVstat.ML

Chaoqi Wang , Guodong Zhang , Roger Grosse This is my paper
classification cs.LGcs.CVstat.ML
keywords trainingnetworksgradientmethodnetworkflowimagenetinitialization
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Overparameterization has been shown to benefit both the optimization and generalization of neural networks, but large networks are resource hungry at both training and test time. Network pruning can reduce test-time resource requirements, but is typically applied to trained networks and therefore cannot avoid the expensive training process. We aim to prune networks at initialization, thereby saving resources at training time as well. Specifically, we argue that efficient training requires preserving the gradient flow through the network. This leads to a simple but effective pruning criterion we term Gradient Signal Preservation (GraSP). We empirically investigate the effectiveness of the proposed method with extensive experiments on CIFAR-10, CIFAR-100, Tiny-ImageNet and ImageNet, using VGGNet and ResNet architectures. Our method can prune 80% of the weights of a VGG-16 network on ImageNet at initialization, with only a 1.6% drop in top-1 accuracy. Moreover, our method achieves significantly better performance than the baseline at extreme sparsity levels.

Discussion (0). Sign in to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Efficient Column-Wise N:M Pruning on RISC-V CPU

    cs.DC 2025-07 conditional novelty 6.0 of 10

    Column-wise N:M pruning plus fused im2col and data packing accelerates ResNet inference on RISC-V vector CPUs by up to 4x while keeping ImageNet top-1 accuracy within 2.1% of the dense model.

  2. Dynamic Sparse Training of Diagonally Sparse Networks

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A dynamic sparse training method that restricts weights to a learnable set of diagonals, preserving sparsity in both forward and backward passes to obtain GPU speedups at accuracy close to unstructured sparsity.

  3. Hyperpruning: Efficient Search through Pruned Variants of Recurrent Neural Networks Leveraging Lyapunov Spectrum

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A Lyapunov-spectrum-based distance to the dense network lets hyperparameter search for pruned RNNs stop early and select models that beat both loss-based baselines and the dense originals.

  4. NeuroTrails: Training with Dynamic Sparse Heads as the Key to Effective Ensembling

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Dynamic sparse training of multiple heads on a shared backbone outperforms full dense ensembles on ImageNet and C4 while using less compute.

  5. Examining the Efficacy of Graph Neural Network Message-Passing in Regression Contexts

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Across four NAS/DNN-predictor regression benchmarks, GEN (deep graph convolution) achieves the best average rank over 11 GNN message-passing layers, though attention GATv2 wins on the largest graphs.

  6. Double-Scoring: Reliable Extraction of Strong Lottery Tickets

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Double-scoring replaces layerwise sparsity selection with fixed half-density masking over an augmented score tensor, yielding higher untrained-subnetwork accuracy but at uncontrolled effective sparsity.

  7. Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Experts

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Sparse adapters trained with max connection sensitivity outperform LoRA and full fine-tuning both alone and after merging 20 task experts, but still lag multitask training on unseen tasks.

  8. Searching Efficient Deep Architectures for Radar Target Detection using Monte-Carlo Tree Search

    eess.SP 2025-06 conditional novelty 5.0 of 10

    Monte-Carlo tree search found a radar detection network with 60% fewer parameters than a baseline U-Net at comparable detection performance.

  9. EGGS-PTP: An Expander-Graph Guided Structured Post-training Pruning Method for Large Language Models

    cs.LG 2025-08 conditional novelty 4.0 of 10

    EGGS-PTP adds a connectivity-preserving diagonal selection to RIA-style importance pruning, achieving slightly better perplexity under N:M sparsity.

  10. PacTrain: Pruning and Adaptive Sparse Gradient Compression for Efficient Collective Communication in Distributed Deep Learning

    cs.DC 2025-05 conditional novelty 4.0 of 10

    PacTrain combines model pruning, gradient sparsity enforcement, and ternary quantization to make gradient synchronization all-reduce compatible and communication-efficient.

  11. Efficient Privacy-Preserving Cross-Silo Federated Learning with Multi-Key Homomorphic Encryption

    cs.CR 2025-05 conditional novelty 4.0 of 10

    MASER combines majority-vote weight pruning with multi-key homomorphic encryption to reduce privacy-preserving federated learning overhead by 3 to 8 times while keeping accuracy within about 1 percent of vanilla FL.

  12. SHeRL-FL: When Representation Learning Meets Split Learning in Hierarchical Federated Learning

    cs.LG 2025-08 unverdicted novelty 2.0 of 10

    The submitted body is an unrelated survey, not the SHeRL-FL method claimed in the metadata.

Pith tools