Pith. sign in

REVIEW 4 major objections 6 minor 49 references

PacTrain: Pruning and Adaptive Sparse Gradient Compression for Efficient Collective Communication in Distributed Deep Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PacTrain combines pruning with sparse gradient compression so that distributed training stays all-reduce-compatible and non-lossy, reporting 1.25–7.05x faster time-to-accuracy than FP16 and TopK baselines and up to 8.72x faster than…

desk verdict PacTrain is a reasonable engineering combination with a useful mask-tracker trick and plausible testbed results, but the evaluation omits the relevant baselines and the mask/invariant story needs to be pinned down before the central claims can be trusted. read the letter →

arxiv 2505.18563 v1 pith:PED4HALK submitted 2025-05-24 cs.DC cs.AI

Yisu Wang , Ruilong Wu , Xinjiao Li , Dirk Kutscher This is my paper
classification cs.DCcs.AI
keywords gradientcompressionnetworkpruningdistributeddeeplearningall-reducesparsecommunicationtime-to-accuracymasktrackerternaryquantization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Distributed deep learning is often bottlenecked by the network step that aggregates gradients across workers, especially on slow links. PacTrain's claim is that pruning a pretrained model to a fixed mask changes the gradient distribution enough that only the non-zero gradients need to be sent, and because every worker knows the same mask, the sparse values can be packed into a dense tensor and synchronized with the standard all-reduce operation exactly as ordinary gradients are. The paper reports time-to-accuracy improvements of 1.25–7.05x over FP16 and TopK compression baselines and up to 8.72x over uncompressed all-reduce on 100 Mbps to 1 Gbps testbed links, with less than 2% accuracy loss for pruning ratios up to 80%. This matters because existing compression methods that work with all-reduce, like TopK and quantization, are lossy, while sparse collective schemes that preserve accuracy generally do not fit all-reduce.

What carries the argument

The load-bearing object is the masking identity of Gradient Sparsity Enforcement: $G = (W \neq 0) \odot G$, applied after each backward pass so that gradients at pruned weights are exactly zero. This identity turns a locally chosen pruning mask into globally known gradient sparsity, which is what lets the non-zero gradient entries be packed into a dense tensor and exchanged through the ordinary all-reduce collective without per-worker decompression. A Mask Tracker runs on the flattened, communication-ready gradient tensor to detect whether the mask is stable; while stable, workers exchange only the dense non-zero block, and whenever the mask shifts they fall back to full-gradients all-reduce. The pruning mask itself is chosen with a gradient-flow preservation score so that the surviving parameters keep the optimization directions the model needs.

What would settle it

Give two workers deliberately non-overlapping data shards, log each worker's pruning mask after every Mask Tracker synchronization, and count the iterations in which the masks differ after a sync; if persistent mask disagreement appears, or if the full-synchronization fallback fires often enough to dominate the communication cost, the compression benefit and the non-lossy property would be disproved.

Watch

Extended reading notes

Core claim

The central claim is that gradient sparsity can be enforced rather than assumed: after the model weights are pruned by a fixed mask, the GSE operation zeros out every gradient at a pruned position, so the non-zero gradient coordinates are identical for all workers. That global knowledge is what allows PacTrain to reformat the sparse gradient into a low-dimensional dense tensor and run a standard all-reduce on it, exchanging every non-zero gradient without loss and without falling back to all-gather. The Mask Tracker maintains the correspondence between the flattened gradient tensor and the pruned weights, and the system performs a full synchronization whenever the mask changes between iterations. The paper therefore calls the compression non-lossy and treats the achievable compression as near-optimal for an all-reduce-compatible scheme: communication scales with the pruning ratio, and optional ternary quantization can reduce it further without sacrificing final accuracy.

Load-bearing premise

All workers must stay on the same pruning mask, because the compressed exchange only works when every worker agrees on which gradient coordinates are non-zero; if the masks drift apart on different data, the non-lossy claim breaks.

Editorial extensions

If this is right

  • If PacTrain is correct, distributed fine-tuning on bandwidth-limited links can make communication cost scale with the number of surviving weights, not with the full model, so all-reduce traffic shrinks by roughly the pruning ratio.
  • Because the compressed exchange carries every non-zero gradient, the scheme does not introduce the information loss that TopK- or quantization-style sparsification does, so convergence is not traded for speed.
  • The all-reduce-compatible dense reformatting means existing optimized collectives and communication backends can be reused unchanged, which keeps implementation overhead low.
  • With pruning ratios up to 80%, final accuracy drops by less than 2% on the evaluated vision tasks, giving the method a wide practical operating window.
  • Ternary quantization can be layered on top of pruning to cut traffic further, and the claimed speedups are largest when network bandwidth is the limiting resource.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implied consequence the paper does not develop is that the method's practical ceiling is set by the accuracy–pruning tradeoff shown in its own experiments: below roughly 80% sparsity the accuracy cost stays small, but above it the compression gain would come with visible accuracy loss.
  • The mask-synchronization fallback deserves a dedicated stress test: if worker data shards are heterogeneous enough that masks keep changing, full synchronization might fire often and erase the communication savings, and the paper does not analyze how quickly masks become consistent again after a sync.
  • The same fixed-mask sparsity trick is naturally extensible to dynamic sparse training or federated learning, but those settings would also need to communicate mask metadata or tolerate temporary mask disagreement, which is a new constraint this paper does not address.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes PacTrain, a distributed training framework that combines unstructured weight pruning with a sparse gradient-compression scheme designed to remain compatible with the all-reduce collective. The core mechanisms are gradient sparsity enforcement (GSE), which zeros gradients at pruned weight positions, and a mask tracker that detects when the sparsity pattern is stable and then exchanges only the non-zero coordinates of the flattened gradient tensor; ternary quantization is optionally applied on top. Experiments on an eight-GPU testbed with CIFAR-10/CIFAR-100 and VGG19, ResNet18, ResNet152, and ViT-Base-16 report time-to-accuracy (TTA) improvements of 1.25-8.72x relative to FP16 and TopK baselines under 100 Mbps-1 Gbps bandwidth bottlenecks.

Significance. PacTrain addresses a real and timely problem: improving TTA for distributed training on bandwidth-constrained links. The idea of coupling pruning to gradient sparsity, while keeping the compressed exchange compatible with all-reduce, is sensible, and the reported speedups are plausible for the tested settings. The implementation is a genuine systems contribution, in particular the use of the DDP communication hook and a mask tracker to overcome the loss of gradient-name information in the reformatted tensor. However, the central correctness claim is not fully established: the dynamic-mask path in Algorithm 1 lacks a proof or protocol for mask agreement across workers, the convergence analysis is only an unbiasedness statement for quantization, and the evaluation omits the most relevant sparse-collective baselines. The paper's headline 'state-of-the-art' and 'non-lossy' claims are therefore stronger than the current evidence.

major comments (4)
  1. [Section III.C; Algorithm 1 (lines 7-12)] The compressed all-reduce path in Algorithm 1 is defined only if every worker sends non-zero gradient values for exactly the same coordinate set, but the manuscript does not establish this invariant for the dynamic-mask variant. The statement that local weight magnitudes 'remain the same across the distributed workers' holds only if no worker updates its weights before the gradient exchange; Algorithm 1 nevertheless maintains a per-worker sparse mask and a local stability check, and the fallback in line 12 performs all-reduce on the gradient update, not on mask metadata. If one worker's mask becomes unstable for a data-dependent reason, nothing in the protocol forces the masks to re-converge, and the compressed all-reduce would sum different coordinates. The paper either needs to prove mask agreement for its synchronous DDP loop, add an explicit mask-synchronization and aggregation step and verify it, or re-scope the adaptive claims to a mask that is fixed at pruning time; with a fixed mask, the mask tracker and unstable-mask fallback become unnecessary.
  2. [Section III.D, Eq. (3)] The convergence analysis provides only the equality E[g̃_t] = s E[sign(g_t)] and a remark that quantization increases variance; it does not establish convergence to the same solution as the uncompressed optimizer, and for a fixed scalar s the stated equality is not generally an unbiased estimator of g_t unless the quantization rule is magnitude-dependent. The paper gives no error-feedback mechanism or bounded-variance argument, and the experiments do not isolate quantization error from pruning error. Consequently, the abstract's claims that the scheme is 'non-lossy' and preserves accuracy are not supported by analysis; GSE and ternary quantization both change the update relative to the dense gradient. I would ask the authors to either provide a proper convergence statement or explicitly define 'non-lossy' as 'lossless reconstruction of the pruned gradient before quantization' and soften the accuracy guarantee.
  3. [Section IV.C, Fig. 3] The evaluation compares PacTrain only with native all-reduce, FP16, and TopK at two rates, while the related-work discussion and Table 1 identify DGC, TernGrad, THC, OmniReduce, and Zen as the relevant gradient-compression and sparse-collective methods. OmniReduce and Zen are precisely the systems that exploit sparse communication, so the abstract's claim of a 1.25-8.72x improvement over 'compression-enabled systems' and the phrase 'state-of-the-art' are stronger than the experimental evidence. Adding these baselines, or limiting the headline claim to FP16/TopK-style all-reduce-compatible compression, is necessary.
  4. [Section III.B; Section IV.C] The proposed workflow starts from a pre-trained model and performs task-specific fine-tuning, and the experiments follow that workflow, but the paper also describes PacTrain as a general distributed-training framework. No experiment trains from scratch, and the TTA curves are all for the fine-tuning setting. The paper should state this scope explicitly and, if from-scratch training is intended to be supported, provide at least one from-scratch experiment or an argument that the pruning-based sparsity enforcement remains beneficial in that regime.
minor comments (6)
  1. [Section I] In the introduction, 'neutral language processing' should read 'natural language processing'.
  2. [Section III.D vs Section IV.C] Section III.D states that GraSP (Eq. 4) is used to compute pruning scores, but Section IV.C says the mask is chosen by weight ranking; please clarify which criterion is actually used and report the pruning criterion for the experiments.
  3. [Abstract; Section IV.C.1] The abstract and the body report different speedup scopes: the abstract says 1.25-8.72x versus compression-enabled systems, while the body reports 8.72x versus native all-reduce and 1.25-7.05x versus FP16/TopK; these numbers should be reconciled and quoted with their exact baseline.
  4. [Section IV.A] No code or configuration files are provided, so the pruning ratio schedule, the quantization scaling factor s, the mask-stability threshold, and the per-model target accuracy thresholds are not reproducible.
  5. [Table 1] Table 1 marks PacTrain as improving convergence speed, but Section IV.C.2 measures wall-clock TTA; no epochs-to-target comparison is reported, so the convergence-speed check mark is not directly evidenced.
  6. [Section IV.C] All TTA numbers appear to be single runs; no standard deviations or repeat counts are reported, which is important for convergence-time measurements on shared testbeds.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PacTrain is an empirical systems paper whose claims are measured, not derived from fitted parameters or self-citations.

full rationale

The paper's central claims are the speedup and accuracy retention of the PacTrain compression scheme, supported by testbed experiments. The compression ratio (0.5) and quantization scaling factor are design choices, not parameters fitted to reproduce the target result. The GSE formula (Eq. 2) is a definition of the masking operation, not a derivation that assumes the conclusion. The mask tracker is a mechanism for adapting to sparsity patterns; its correctness depends on the assumption that masks are synchronized across workers, which is a stated design invariant ('local weight magnitudes, which remain the same across the distributed workers') rather than a circular argument. The paper does not invoke a uniqueness theorem from the authors' own prior work, nor does it rename an existing empirical pattern as a new result. All references are external and non-self-citational. Therefore, the derivation chain is self-contained and no step reduces to its own inputs.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central assumptions are the preservation of accuracy under pruning, the global consistency of pruning masks across workers, the stability of masks over time, and the unbiasedness of ternary quantization. The first three are empirical or architectural assumptions that the paper does not fully validate; the last is a standard math property that alone does not guarantee convergence.

free parameters (2)
  • pruning_ratio = 0.5 (default)
    Chosen by hand; Fig. 6 shows accuracy impact across ratios. The default 0.5 is used in most experiments and directly controls compression ratio.
  • ternary quantization scaling factor s = not specified
    Eq. 3 defines E(\tilde g_t) = s E(sign(g_t)); s is assumed to make quantization unbiased in expectation, but its value or selection rule is not given.
assumptions (4)
  • domain assumption Pruning a pre-trained model and applying GSE preserves final accuracy sufficiently (accuracy drop < 2% for pruning ratio <= 0.8)
    Fig. 6 provides empirical support, but this is an assumption that the mask is stable and does not harm convergence.
  • domain assumption All workers maintain identical pruning masks, giving global knowledge of gradient sparsity
    This is the core assumption enabling compression without communication; the paper does not prove mask consistency across workers during training.
  • domain assumption The mask becomes stable after the early phase of training so that compressed communication can be used most of the time
    The mask tracker switches to full sync when unstable; the paper provides no evidence of the frequency of instability.
  • standard math Gradient quantization to {-1,0,+1} with scaling s is unbiased in expectation (Eq. 3)
    This is a standard property of stochastic quantization, but it does not imply convergence for non-convex deep networks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PacTrain: Pruning and Adaptive Sparse Gradient Compression for Efficient Collective Communication in Distributed Deep Learning." pith.science (2026). https://pith.science/paper/PED4HALK

@misc{pith2026250518563,
  author       = {Pith},
  title        = {Pith review of: PacTrain: Pruning and Adaptive Sparse Gradient Compression for Efficient Collective Communication in Distributed Deep Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PED4HALK}},
  note         = {Machine review of arXiv:2505.18563}
}
read the original abstract

Large-scale deep neural networks (DNN) exhibit excellent performance for various tasks. As DNNs and datasets grow, distributed training becomes extremely time-consuming and demands larger clusters. A main bottleneck is the resulting gradient aggregation overhead. While gradient compression and sparse collective communication techniques are commonly employed to alleviate network load, many gradient compression schemes do not achieve acceleration of the training process while also preserving accuracy. This paper introduces PacTrain, a novel framework that accelerates distributed training by combining pruning with sparse gradient compression. Active pruning of the neural network makes the model weights and gradients sparse. By ensuring the global knowledge of the gradient sparsity among all distributed training workers, we can perform lightweight compression communication without harming accuracy. We show that the PacTrain compression scheme achieves a near-optimal compression strategy while remaining compatible with the all-reduce primitive. Experimental evaluations show that PacTrain improves training throughput by 1.25 to 8.72 times compared to state-of-the-art compression-enabled systems for representative vision and language models training tasks under bandwidth-constrained conditions.

Figures

Figures reproduced from arXiv: 2505.18563 by the authors.

Figure 1
Figure 1. Pruning and Fine-tune on specific data. LTH shows that the existence of sparse sub-networks at initialization can achieve almost the same performance as the dense model when trained alone. By identifying these optimal sparse sub-networks after several training epochs, we can prune most of the model weights and reduce both the computation and communication costs during training. The effectiveness of these strategies … view at source ↗
Figure 3
Figure 3. End-to-End TTA speedup with different WAN bandwidths (relative to native all-reduce, log scale). [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Evaluation topology. implemented and evaluated PacTrain using a virtual testbed hosted in our laboratory. A. Setup and workloads We built our test bed using the ESXI virtualization platform with a server equipped with eight A40 GPUs and an Intel Xeon Platinum 8358P processor with 64 CPU cores. To simulate the network environment shown in [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Time-to-accuracy Comparison of performance for the [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

49 extracted references · 36 canonical work pages

  1. [1]

    Lamda: Language models for dialog applications,

    R. Thoppilan, D. D. Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, and et al., “Lamda: Language models for dialog applications,”

    work page

  2. [2]

    Software- defined network assimilation: bridging the last mile towards centralized network configuration management with nassim,

    H. Chen, Y . Miao, L. Chen, H. Sun, H. Xu, L. Liu, and et al., “Software- defined network assimilation: bridging the last mile towards centralized network configuration management with nassim,” ser. SIGCOMM ’22, 2022, p. 281–297

    work page 2022

  3. [3]

    Netllm: Adapting large language models for networking,

    D. Wu, X. Wang, Y . Qiao, Z. Wang, J. Jiang, S. Cui, and F. Wang, “Netllm: Adapting large language models for networking,” ser. ACM SIGCOMM ’24, 2024, p. 661–678

    work page 2024

  4. [4]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, and et al., “Llama: Open and efficient foundation language models,” 2023

    work page 2023

  5. [5]

    Language models are few-shot learners,

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, and et al., “Language models are few-shot learners,” ser. NIPS ’20. Curran Associates Inc., 2020

    work page 2020

  6. [6]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, and et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” ICLR, 2021

    work page 2021

  7. [7]

    You only look once: Unified, real-time object detection,

    J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in CVPR, 2016, pp. 779–788

    work page 2016

  8. [8]

    Accelerating model training in multi-cluster environments with consumer-grade gpus,

    H. Lim, J. Ye, S. Abdu Jyothi, and D. Han, “Accelerating model training in multi-cluster environments with consumer-grade gpus,” ser. ACM SIGCOMM ’24, 2024, p. 707–720

    work page 2024

Show all 49 references
  1. [9]

    Crux: Gpu-efficient communication scheduling for deep learning training,

    J. Cao, Y . Guan, K. Qian, J. Gao, W. Xiao, J. Dong, and et al., “Crux: Gpu-efficient communication scheduling for deep learning training,” ser. ACM SIGCOMM ’24. Association for Computing Machinery, 2024

    2024

  2. [10]

    MegaScale: Scaling large language model training to more than 10,000 GPUs,

    Z. Jiang, H. Lin, Y . Zhong, Q. Huang, Y . Chen, Z. Zhang, and et al., “MegaScale: Scaling large language model training to more than 10,000 GPUs,” in NSDI 24. USENIX Association, Apr. 2024, pp. 745–760

    2024

  3. [11]

    Beyond throughput and compression ratios: Towards high end-to-end utility of gradient compression,

    W. Han, S. Vargaftik, M. Mitzenmacher, B. Karp, and R. B. Basat, “Beyond throughput and compression ratios: Towards high end-to-end utility of gradient compression,” 2024. [Online]. Available: https://arxiv.org/abs/2407.01378

    2024 arXiv

  4. [12]

    Optimal and near-optimal adaptive vector quantization,

    R. Ben-Basat, Y . Ben-Itzhak, M. Mitzenmacher, and S. Vargaftik, “Optimal and near-optimal adaptive vector quantization,” 2024. [Online]. Available: https://arxiv.org/abs/2402.03158

    2024 arXiv

  5. [13]

    Terngrad: Ternary gradients to reduce communication in distributed deep learning,

    W. Wen, C. Xu, F. Yan, C. Wu, Y . Wang, Y . Chen, and H. Li, “Terngrad: Ternary gradients to reduce communication in distributed deep learning,”

  6. [14]

    Thc: Accelerating distributed deep learning using tensor homomorphic compression,

    M. Li, R. B. Basat, S. Vargaftik, C. Lao, K. Xu, M. Mitzenmacher, and M. Yu, “Thc: Accelerating distributed deep learning using tensor homomorphic compression,” 2024. [Online]. Available: https: //arxiv.org/abs/2302.08545

    2024 arXiv

  7. [15]

    Sparse communication for distributed gradient descent,

    A. F. Aji and K. Heafield, “Sparse communication for distributed gradient descent,” in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , M. Palmer, R. Hwa, and S. Riedel, Eds. Association for Computational Linguistics, Sep. 2017, pp. 440–445

    2017

  8. [16]

    Deep gradient compression: Reducing the communication bandwidth for distributed training,

    Y . Lin, S. Han, H. Mao, Y . Wang, and W. J. Dally, “Deep gradient compression: Reducing the communication bandwidth for distributed training,” 2018, the full paper is available at https://openreview.net/ forum?id=rkxoQg-CZ

    2018

  9. [17]

    Powersgd: Practical low-rank gradient compression for distributed optimization,

    T. V ogels, S. P. Karimireddy, and M. Jaggi, “Powersgd: Practical low-rank gradient compression for distributed optimization,” 2020. [Online]. Available: https://arxiv.org/abs/1905.13727

    2020 arXiv

  10. [18]

    Grace: A compressed communication framework for distributed machine learning,

    H. Xu, C.-Y . Ho, A. M. Abdelmoniem, A. Dutta, E. H. Bergou, K. Karatsenidis, and et al., “Grace: A compressed communication framework for distributed machine learning,” in ICDCS, 2021

    2021

  11. [19]

    Efficient sparse collective communication and its application to accelerate distributed deep learning,

    J. Fei, C.-Y . Ho, A. N. Sahu, M. Canini, and A. Sapio, “Efficient sparse collective communication and its application to accelerate distributed deep learning,” ser. SIGCOMM ’21. Association for Computing Machinery, 2021, p. 676–691

    2021

  12. [20]

    Zen: Near-optimal sparse tensor synchronization for distributed dnn training,

    Z. Wang, Z. Xu, A. Shrivastava, and T. S. E. Ng, “Zen: Near-optimal sparse tensor synchronization for distributed dnn training,” 2023. [Online]. Available: https://arxiv.org/abs/2309.13254

    2023 arXiv

  13. [21]

    Embrace: Accelerating sparse communication for distributed training of deep neural networks,

    S. Li, Z. Lai, D. Li, Y . Zhang, X. Ye, and Y . Duan, “Embrace: Accelerating sparse communication for distributed training of deep neural networks,” ser. ICPP ’22. Association for Computing Machinery, 2023

    2023

  14. [22]

    Hi-speed dnn training with espresso: Unleashing the full potential of gradient compression with near-optimal usage strategies,

    Z. Wang, H. Lin, Y . Zhu, and T. S. E. Ng, “Hi-speed dnn training with espresso: Unleashing the full potential of gradient compression with near-optimal usage strategies,” ser. EuroSys ’23. Association for Computing Machinery, 2023, p. 867–882

    2023

  15. [23]

    Mccs: A service-based approach to collective communication for multi- tenant cloud,

    Y . Wu, Y . Xu, J. Chen, Z. Wang, Y . Zhang, M. Lentz, and D. Zhuo, “Mccs: A service-based approach to collective communication for multi- tenant cloud,” ser. ACM SIGCOMM ’24. Association for Computing Machinery, 2024, p. 679–690

    2024

  16. [24]

    Swing: Short- cutting rings for higher bandwidth allreduce,

    D. D. Sensi, T. Bonato, D. Saam, and T. Hoefler, “Swing: Short- cutting rings for higher bandwidth allreduce,” in NSDI 24 . USENIX Association, Apr. 2024, pp. 1445–1462

    2024

  17. [25]

    Scaling distributed machine learning with the parameter server,

    M. Li, D. G. Andersen, J. W. Park, A. J. Smola, A. Ahmed, V . Josifovski, and et al., “Scaling distributed machine learning with the parameter server,” in OSDI 14. USENIX Association, Oct. 2014, pp. 583–598

    2014

  18. [26]

    Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation,

    J. Ansel, E. Yang, H. He, N. Gimelshein, A. Jain, M. V oznesensky, and et al., “Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation,” ser. ASPLOS ’24. Association for Computing Machinery, 2024, p. 929–947

    2024

  19. [27]

    Learning multiple layers of features from tiny images,

    A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” University of Toronto, Tech. Rep. 0, 2009. [Online]. Available: https://www.cs.toronto.edu/~kriz/learning-features-2009-TR. pdf

    2009

  20. [28]

    Very deep convolutional networks for large-scale image recognition,

    K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” 2015. [Online]. Available: https: //arxiv.org/abs/1409.1556

    2015 arXiv

  21. [29]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” 2015. [Online]. Available: https://arxiv.org/abs/1512.03385

    2015 arXiv

  22. [30]

    Learning both weights and connections for efficient neural networks,

    S. Han, J. Pool, J. Tran, and W. J. Dally, “Learning both weights and connections for efficient neural networks,” 2015

    2015

  23. [31]

    Pruning filters for efficient convnets,

    H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, “Pruning filters for efficient convnets,” 2017

    2017

  24. [32]

    Pruning convolutional neural networks for resource efficient inference,

    P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz, “Pruning convolutional neural networks for resource efficient inference,” 2017

    2017

  25. [33]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks,

    J. Frankle and M. Carbin, “The lottery ticket hypothesis: Finding sparse, trainable neural networks,” 2019

    2019

  26. [34]

    Earlybert: Efficient bert training via early-bird lottery tickets,

    X. Chen, Y . Cheng, S. Wang, Z. Gan, Z. Wang, and J. Liu, “Earlybert: Efficient bert training via early-bird lottery tickets,” 2021

    2021

  27. [35]

    When to prune? a policy towards early structural pruning,

    M. Shen, P. Molchanov, H. Yin, and J. M. Alvarez, “When to prune? a policy towards early structural pruning,” in CVPR, 2022

    2022

  28. [36]

    Similarity of neural network representations revisited,

    S. Kornblith, M. Norouzi, H. Lee, and G. Hinton, “Similarity of neural network representations revisited,” 2019. [Online]. Available: https://arxiv.org/abs/1905.00414

    2019 arXiv

  29. [37]

    Model sparsity can simplify machine unlearning,

    J. Jia, J. Liu, P. Ram, Y . Yao, G. Liu, Y . Liu, P. Sharma, and S. Liu, “Model sparsity can simplify machine unlearning,” 2024. [Online]. Available: https://arxiv.org/abs/2304.04934

    2024 arXiv

  30. [38]

    Deepzero: Scaling up zeroth-order optimization for deep model training,

    A. Chen, Y . Zhang, J. Jia, J. Diffenderfer, J. Liu, K. Parasyris, Y . Zhang, Z. Zhang, B. Kailkhura, and S. Liu, “Deepzero: Scaling up zeroth-order optimization for deep model training,” 2024. [Online]. Available: https://arxiv.org/abs/2310.02025

    2024 arXiv

  31. [39]

    Fedmef: Towards memory-efficient federated dynamic pruning,

    H. Huang, W. Zhuang, C. Chen, and L. Lyu, “Fedmef: Towards memory-efficient federated dynamic pruning,” 2024. [Online]. Available: https://arxiv.org/abs/2403.14737

    2024 arXiv

  32. [40]

    Zerofl: Efficient on-device training for federated learning with local sparsity,

    X. Qiu, J. Fernandez-Marques, P. P. Gusmao, Y . Gao, T. Parcollet, and N. D. Lane, “Zerofl: Efficient on-device training for federated learning with local sparsity,” 2022. [Online]. Available: https: //arxiv.org/abs/2208.02507

    2022 arXiv

  33. [41]

    Federated dynamic sparse training: Computing less, communicating less, yet learning better,

    S. Bibikar, H. Vikalo, Z. Wang, and X. Chen, “Federated dynamic sparse training: Computing less, communicating less, yet learning better,” 2021. [Online]. Available: https://arxiv.org/abs/2112.09824

    2021 arXiv

  34. [42]

    Distributed pruning towards tiny neural networks in federated learning,

    H. Huang, L. Zhang, C. Sun, R. Fang, X. Yuan, and D. Wu, “Distributed pruning towards tiny neural networks in federated learning,” 2023. [Online]. Available: https://arxiv.org/abs/2212.01977

    2023 arXiv

  35. [43]

    Imagenet large scale visual recognition challenge,

    O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “Imagenet large scale visual recognition challenge,” 2015. [Online]. Available: https://arxiv.org/abs/1409.0575

    2015 arXiv

  36. [44]

    Tiny imagenet visual recognition challenge,

    Y . Le and X. S. Yang, “Tiny imagenet visual recognition challenge,”

  37. [45]

    Picking winning tickets before training by preserving gradient flow,

    C. Wang, G. Zhang, and R. Grosse, “Picking winning tickets before training by preserving gradient flow,” 2020. [Online]. Available: https://arxiv.org/abs/2002.07376

    2020 arXiv

  38. [46]

    Nvidia collective communications library (nccl),

    NVIDIA, “Nvidia collective communications library (nccl),” https:// developer.nvidia.com/nccl., 2024

    2024

  39. [2015]

    Available: https://api.semanticscholar.org/CorpusID: 16664790

    [Online]. Available: https://api.semanticscholar.org/CorpusID: 16664790

  40. [2017]

    Available: https://arxiv.org/abs/1705.07878

    [Online]. Available: https://arxiv.org/abs/1705.07878

    arXiv

  41. [2022]

    Available: https://arxiv.org/abs/2201.08239

    [Online]. Available: https://arxiv.org/abs/2201.08239

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.