Pith. sign in

REVIEW 3 major objections 4 minor 46 references

You Don't Need All Attentions: Distributed Dynamic Fine-Tuning for Foundation Models

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read D2FT's central claim: most attention subnets can be skipped or run forward-only during fine-tuning, cutting compute 40% and communication 50% with only 1–2% accuracy loss.

desk verdict Interesting scheduling framework with a solid core idea, but the headline compute savings are inflated because the Fisher pre-pass is excluded from the accounting. read the letter →

arxiv 2504.12471 v1 pith:HGAOEBEN submitted 2025-04-16 cs.LG cs.DCcs.PF

Shiwei Ding , Lan Zhang , Zhenlin Wang , Giuseppe Ateniese , Xiaoyong Yuan This is my paper · ORCID
classification cs.LGcs.DCcs.PF
keywords distributedfine-tuningattentionsparsitymultipleknapsackforward-onlypropagationLoRAworkloadbalancingViTtrainingefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes D2FT (Distributed Dynamic Fine-Tuning), which argues that fine-tuning a pretrained transformer does not need every attention module for every training sample. Instead, each attention-head subnet is assigned one of three operations per sample—full forward and backward, forward-only, or skip—and a distributed scheduler chooses these operations to cut training computation by 40% and communication by 50% while losing only 1–2% top-1 accuracy on CIFAR-10, CIFAR-100, and Stanford Cars. The method scores each subnet using pretrained weight magnitudes and one-shot Fisher information, then casts operation selection as a multiple-knapsack problem solved per device by dynamic programming. If the claim holds, memory-constrained commercial devices could fine-tune foundation models at a fraction of the usual training traffic, with device workloads balanced exactly. The same scheduling shell extends to LoRA, where the 40%/50% savings cost 4–6% accuracy on Stanford Cars.

What carries the argument

The load-bearing object is the operation-selection table $T_{opt}$, produced by two nested knapsack solves: an outer solve picks which samples get full forward and backward updates by maximizing summed weight-magnitude scores under each device's full-operation cost, and an inner solve picks which samples get forward-only passes by maximizing summed Fisher-information scores under the forward-only cost. Residual shortcut routes carry activations and gradients around skipped subnets so the rest of the model still trains. This decoupling converts the NP-hard multiple-knapsack orchestration into per-device dynamic programs that jointly balance workload and maximize contribution.

What would settle it

A direct test: on CIFAR-100, replace D2FT's contribution scores with random values but keep the same knapsack schedule structure; if accuracy stays near 89%, the score proxy is not doing the work. A stronger test: compute leave-one-subnet-out loss changes on a validation batch and compare their ranking with the paper's weight-magnitude and Fisher-information ranking; if the rank correlation is near zero, the central claim fails.

Watch

Extended reading notes

Core claim

D2FT partitions a transformer into subnets, each one attention head plus a slice of the feed-forward network, and gives each subnet a per-sample instruction from {full, forward-only, skip}. The instructions come from a bi-level heuristic that first fills each device's full-operation budget using weight-magnitude backward contribution scores, then fills the forward-only budget using empirical Fisher information scores, merging the two into a single schedule table. The paper's central empirical discovery is that this scheduled sparsity costs almost nothing: relative to standard fine-tuning, D2FT reports 40% lower training computation and 50% lower communication with 1–2% top-1 accuracy drops on CIFAR-10, CIFAR-100, and Stanford Cars; with LoRA, the same savings cost 4–6% on Stanford Cars, while beating small-rank LoRA at equal compute. The schedule also drives device workload variance to zero, so no subnet acts as a straggler.

Load-bearing premise

The schedule's value over random selection rests on the claim that a subnet's weight size and a one-time gradient-based importance estimate predict how much that subnet will actually help on each training sample; if those proxies do not track real loss reduction, the knapsack optimizer is just maximizing a number that does not matter.

Editorial extensions

If this is right

  • A device with a tight compute budget can fine-tune ViT-small at 60% of full compute and keep about 89.4% top-1 accuracy on CIFAR-100, roughly 35 points above random scheduling and well above both dynamic pruning baselines.
  • Because skipped and forward-only subnets keep residual routes, the schedule can change per micro-batch without breaking gradient flow, so the framework works across different batch-to-micro-batch splits.
  • D2FT reduces workload variance to zero across 74 subnets, which corresponds to shorter per-batch execution time than random scheduling and dynamic pruning.
  • Applied to LoRA, the same scheduling saves 40% compute or 50% communication on Stanford Cars while losing 4–6% top-1 accuracy relative to full-rank standard LoRA, and it outperforms simply shrinking LoRA rank at equal cost.
  • Heterogeneous device pools, with mixed memory sizes or compute speeds, keep the same accuracy as homogeneous pools because the knapsack capacities are set per device.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the schedule is computed once from static pretrained scores, so refreshing the contribution scores during training could improve accuracy if subnet importance shifts; the paper does not test this.
  • Beyond the paper: since the method's edge over random scheduling rests entirely on the contribution-score proxy, the knapsack shell could be reused with other cheap saliency signals or a learned predictor of per-sample loss reduction as a direct test of where the gain comes from.
  • Beyond the paper: if attention redundancy in language models is as high as in vision transformers, the same 40%/50% savings might transfer to LLM fine-tuning; that is an extrapolation, not a claim in the paper.
  • Beyond the paper: because inference still uses all parameters, D2FT is purely a training-time distribution strategy, and combining it with post-training pruning could compound savings, but that combination is unexplored here.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes D2FT, a distributed fine-tuning framework for transformer-based foundation models. During fine-tuning, each attention-head subnet is assigned one of three operations per micro-batch: full forward and backward, forward-only, or skip, with residual routes preserving signal flow. The assignment is formulated as a multiple-knapsack problem using proxy contribution scores (weight magnitude for backward selection and Fisher information for forward selection), decoupled into per-device bi-level knapsacks solved by dynamic programming. Experiments on CIFAR-10, CIFAR-100, and Stanford Cars with ViT-small compare D2FT against random scheduling, dynamic pruning baselines, an MoE-style baseline, and standard full fine-tuning, and additionally extend the framework to LoRA. The paper reports 40% computational-cost savings and 50% communication-cost savings with small accuracy drops.

Significance. If the headline results hold, D2FT would be a useful contribution to efficient fine-tuning and distributed workload balancing: it converts a training-time sparsity observation into a concrete scheduling algorithm that exactly satisfies per-device budget constraints. The workload-variance result in Table I and the execution-time comparison in Table II are concrete strengths, as is the extension to LoRA with rank-matched and reduced-rank baselines. The algorithmic description, including the dynamic-programming scheduling tables, is detailed enough to be reimplemented, although no code is provided. The main risks are the omission of the Fisher scoring pre-pass from the computational-cost accounting, the post hoc selection of scoring metrics, and the lack of direct validation of the proxy objective; these issues affect the magnitude and generality of the central savings claims.

major comments (3)
  1. [§II-A3, §IV-A; abstract] The reported computational-cost savings omit the cost of the Fisher-scoring pre-pass. Section II-A3 states that computing the scores requires feeding all samples for forward and backward propagation without updating weights, which costs the same as one epoch of standard fine-tuning. The cost metric in Section IV-A counts only the pf and po operations during fine-tuning. If D2FT fine-tuning runs E epochs, its true relative cost at the headline 60% scheduling budget is 0.6 + 1/E, so the true saving is 0.4 − 1/E, not 0.4. For E = 5 the saving is 20%, and for E = 2 D2FT is 10% more expensive than standard fine-tuning. The paper never reports E for any experiment, and the LoRA results inherit the same issue because Fisher scores are also computed over all samples for the LoRA matrices. The abstract and Figures 1–3 should be revised to include this pre-pass in the cost accounting, or the paper should report E and restate the savings for realistic epoch counts.
  2. [§III-B3, §III-B, Table III] The scoring metrics are selected on the same dataset used for the headline results. Table III reports that the choice of Weight Magnitude as the backward score and Fisher Information as the forward score was made by testing eight combinations on Stanford Cars, and Stanford Cars is one of the three datasets in Figures 1 and 3 and in the abstract's 1–2% accuracy-drop claim. The reported Stanford Cars results are therefore in-sample with respect to metric selection. The paper should select the scores on a held-out validation set or on a different dataset, or should report all eight combinations on all three datasets, so that the headline numbers are not biased by the post hoc choice.
  3. [§II-B, Eqs. (4)–(8), Table II] The optimization objective assumes that per-subnet and per-sample contribution scores are additive and that maximizing their sum maximizes fine-tuning accuracy, but this proxy is not validated. The large accuracy gap between D2FT and random scheduling in Table II (89.4% versus 44.4%) indicates that the scheduling mechanism carries signal, but it does not establish that the Fisher/weight-magnitude proxy is the source of the signal rather than the knapsack structure itself. Section II-B also concedes that the scores are hard to determine in practice. The paper should add a controlled experiment in which the same DP scheduler is run with random contribution scores, and/or a correlation analysis between the proxy scores and per-sample loss reduction, to support the use of this objective as a surrogate for fine-tuning performance.
minor comments (4)
  1. [Algorithm 2, line 27] The backtracking loop updates the sample index with "i = 1 − 1"; this should be "i = i − 1", otherwise the loop will not terminate as intended.
  2. [§II-A1] The sentence "where l and h denotes the index of attention heads and index of blocks" reverses the definitions: earlier in the same paragraph k = (l, h) with l the block index and h the attention-head index. This should be corrected.
  3. [§III-B2] The first computational-cost setting for LoRA states that 3 micro-batches performing pf and 2 performing po reduce cost to 95% of standard LoRA. Given the paper's own cost model in Section IV-A that po costs 40% of pf, the resulting cost is 3 + 2×0.4 = 3.8 out of 5, i.e., 76%, not 95%. Please clarify whether the cost model differs for LoRA or correct the reported percentage and the corresponding point in Figure 3.
  4. [§III-A] The experimental settings list datasets, model, hardware, and partition details but omit the number of fine-tuning epochs and optimizer hyperparameters other than the optimizer name. Reporting these is necessary both for reproducibility and for interpreting the cost-savings claims.

Circularity Check

1 steps flagged · score 4.0 of 10

Partial circularity: the scoring-rule combination is fitted on Stanford Cars, and the same dataset is then used for headline and LoRA accuracy claims; other results remain independently measured.

  1. fitted input called prediction [Section III-B3, Table III; abstract headline accuracy claims]
    "We tested eight combinations of backward and forward scores to identify the scoring method that yields the best model performance."

    The paper selects Weight Magnitude as the backward score and Fisher Information as the forward score by maximizing top-1 accuracy on Stanford Cars (Table III). The same Stanford Cars dataset is then used in the abstract's headline claim ('with only 1% to 2% accuracy drops on ... Stanford Cars') and in the LoRA evaluation (Section III-B2, Figure 3). The reported Stanford Cars performance is therefore not an independent out-of-sample test of the framework: it is the accuracy of the configuration chosen on that exact dataset, so the Stanford Cars component of the headline result is partly self-confirming rather than a predicted outcome.

full rationale

Aside from the Stanford Cars model-selection issue, the paper's derivation is largely self-contained and does not reduce to its inputs by construction. The contribution scores are explicitly proxies, and the paper acknowledges they are hard to determine in practice; the knapsack formulation maximizes these proxies, but the final accuracy comparisons are measured, not derived from the proxies. There is no load-bearing self-citation chain: the cited Fisher-information and pruning works are external, and no 'uniqueness theorem' from the authors is invoked. The CIFAR-10 and CIFAR-100 results were not used to choose the scoring rule and provide independent evidence that the scheduling method beats random and pruning baselines. The main caveat outside circularity is that the 40% computational-cost claim appears to omit the one-epoch Fisher-scoring pre-pass described in Section II-A3; that is an accounting omission rather than a circular derivation.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The efficiency claims rest on constants the paper itself supplies: po costs 40% of pf compute and 50% of pf communication, ps costs zero, and the contribution-score rule was chosen by testing eight combinations on the evaluation dataset. The only externally grounded input is a timm ViT-small checkpoint. Because the headline percentages follow from these self-supplied constants, the ledger is short but the independence of the numbers is limited. No new entities (particles, forces, dimensions, mediators) are postulated.

free parameters (5)
  • po compute-cost fraction = 0.40*pf (measured 0.43-0.47 in Table IV)
    Used as the universal cost of Forward-Only versus Full in Eq. 4 and in every reported savings percentage; it is measured on one GPU model, then treated as exact.
  • po communication-cost fraction = 0.50*pf
    Assumed from the claim that forward and backward communication tensor sizes are equal (Section IV-A); no network measurement supports it.
  • device capacity budgets C_k = set to realize 60%, 50%, 64%, 75%, 95%, 70%, 90% cost levels
    The claimed savings percentages are budgets handed to the scheduler, not outcomes of a tradeoff search; accuracy at those budgets is then measured.
  • contribution-score metric combination = Weight Magnitude (backward) + Fisher Information (forward)
    Selected as the best of 8 combinations tested on Stanford Cars (Table III), the same dataset used for the headline full-tuning results; no held-out validation split.
  • LoRA rank for standard baseline = R = 240
    Chosen as the 'Standard LoRA' comparison; unusually high for a 384-dim ViT-small and not justified in the text.
assumptions (5)
  • domain assumption Transformer residual structure lets skipped subnets be bypassed by a shortcut route without degrading gradient flow (Section II-A2).
    The ps and po operations assume activations and gradients can be routed around a skipped attention head at zero extra cost; this is the mechanism enabling all reported savings.
  • ad hoc to paper Empirical Fisher information (sum of squared gradient norms) and weight magnitude of a subnet predict its fine-tuning contribution per sample, and contributions are additive across subnets and samples (Eqs. 2-6).
    The scheduler optimizes this proxy rather than the true fine-tuning loss; no correlation analysis is given, and the metric combination is chosen post hoc on the evaluation dataset.
  • ad hoc to paper Solving K independent per-device knapsacks and merging pf/po selections by priority approximately solves the joint orchestration problem (Eqs. 6-8).
    The decoupling ignores cross-subnet interactions in the training objective; the paper validates it only empirically (Table X) against a scaler baseline, with no approximation guarantee.
  • domain assumption Computational and communication costs are linear in the number of selected operations, with ps costing zero and po costing 40% compute / 50% communication (Section IV-A).
    Underlies every percentage in the efficiency claims; Table IV loosely supports the compute linearity on one GPU, but nothing supports the communication model.
  • standard math Multi-knapsack is NP-hard and dynamic programming is an appropriate solver (Section II-B, citing Martello and Toth; Kan et al.).
    Standard background for the heuristic; not a point of contention.

how reviews work

0 comments
Cite this review

Pith. "Pith review of You Don't Need All Attentions: Distributed Dynamic Fine-Tuning for Foundation Models." pith.science (2026). https://pith.science/paper/HGAOEBEN

@misc{pith2026250412471,
  author       = {Pith},
  title        = {Pith review of: You Don't Need All Attentions: Distributed Dynamic Fine-Tuning for Foundation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HGAOEBEN}},
  note         = {Machine review of arXiv:2504.12471}
}
read the original abstract

Fine-tuning plays a crucial role in adapting models to downstream tasks with minimal training efforts. However, the rapidly increasing size of foundation models poses a daunting challenge for accommodating foundation model fine-tuning in most commercial devices, which often have limited memory bandwidth. Techniques like model sharding and tensor parallelism address this issue by distributing computation across multiple devices to meet memory requirements. Nevertheless, these methods do not fully leverage their foundation nature in facilitating the fine-tuning process, resulting in high computational costs and imbalanced workloads. We introduce a novel Distributed Dynamic Fine-Tuning (D2FT) framework that strategically orchestrates operations across attention modules based on our observation that not all attention modules are necessary for forward and backward propagation in fine-tuning foundation models. Through three innovative selection strategies, D2FT significantly reduces the computational workload required for fine-tuning foundation models. Furthermore, D2FT addresses workload imbalances in distributed computing environments by optimizing these selection strategies via multiple knapsack optimization. Our experimental results demonstrate that the proposed D2FT framework reduces the training computational costs by 40% and training communication costs by 50% with only 1% to 2% accuracy drops on the CIFAR-10, CIFAR-100, and Stanford Cars datasets. Moreover, the results show that D2FT can be effectively extended to recent LoRA, a state-of-the-art parameter-efficient fine-tuning technique. By reducing 40% computational cost or 50% communication cost, D2FT LoRA top-1 accuracy only drops 4% to 6% on Stanford Cars dataset.

Figures

Figures reproduced from arXiv: 2504.12471 by the authors.

Figure 1
Figure 1. (Full parameter) Fine-tuning performance comparison. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The top-1 accuracy comparison under the same or [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparison of LoRA fine-tuning. We present the top-1 [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 18 canonical work pages

  1. [1]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018

    arXiv 2018

  2. [2]

    Roberta: A robustly optimized bert pretraining approach,

    Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692 , 2019

    arXiv 1907

  3. [3]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of machine learning research, vol. 21, no. 140, pp. 1–67, 2020

    2020

  4. [4]

    Xlnet: Generalized autoregressive pretraining for language understanding,

    Z. Yang, Z. Dai, Y . Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V . Le, “Xlnet: Generalized autoregressive pretraining for language understanding,” Advances in neural information processing systems , vol. 32, 2019

    2019

  5. [5]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,

    V . Sanh, L. Debut, J. Chaumond, and T. Wolf, “Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,” arXiv preprint arXiv:1910.01108, 2019

    arXiv 1910

  6. [6]

    Sparks of artificial general intelligence: Early experiments with gpt-4,

    S. Bubeck, V . Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Ka- mar, P. Lee, Y . T. Lee, Y . Li, S. Lundberg et al. , “Sparks of artificial general intelligence: Early experiments with gpt-4,” arXiv preprint arXiv:2303.12712, 2023

    arXiv 2023

  7. [7]

    Tokens-to-token vit: Training vision transformers from scratch on imagenet,

    L. Yuan, Y . Chen, T. Wang, W. Yu, Y . Shi, Z.-H. Jiang, F. E. Tay, J. Feng, and S. Yan, “Tokens-to-token vit: Training vision transformers from scratch on imagenet,” in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 558–567

    2021

  8. [8]

    Train big, then compress: Rethinking model size for efficient training and inference of transformers,

    Z. Li, E. Wallace, S. Shen, K. Lin, K. Keutzer, D. Klein, and J. Gonzalez, “Train big, then compress: Rethinking model size for efficient training and inference of transformers,” in International Conference on machine learning. PMLR, 2020, pp. 5958–5968

    work page 2020

Show all 46 references
  1. [9]

    Decoupled greedy learning of cnns,

    E. Belilovsky, M. Eickenberg, and E. Oyallon, “Decoupled greedy learning of cnns,” in International Conference on Machine Learning . PMLR, 2020, pp. 736–745

    2020

  2. [10]

    Distributed learning of fully connected neural networks using indepen- dent subnet training,

    B. Yuan, C. R. Wolfe, C. Dun, Y . Tang, A. Kyrillidis, and C. Jermaine, “Distributed learning of fully connected neural networks using indepen- dent subnet training,” Proceedings of the VLDB Endowment , vol. 15, no. 8, pp. 1581–1590, 2022

    2022

  3. [11]

    Decentralized training of foundation models in heterogeneous environments,

    B. Yuan, Y . He, J. Davis, T. Zhang, T. Dao, B. Chen, P. S. Liang, C. Re, and C. Zhang, “Decentralized training of foundation models in heterogeneous environments,” Advances in Neural Information Process- ing Systems , vol. 35, pp. 25 464–25 477, 2022

    2022

  4. [12]

    Megatron-lm: Training multi-billion parameter language models using model parallelism,

    M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catan- zaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” arXiv preprint arXiv:1909.08053 , 2019

    1909 arXiv

  5. [13]

    Gspmd: general and scalable parallelization for ml computation graphs,

    Y . Xu, H. Lee, D. Chen, B. Hechtman, Y . Huang, R. Joshi, M. Krikun, D. Lepikhin, A. Ly, M. Maggioni et al. , “Gspmd: general and scalable parallelization for ml computation graphs,” arXiv preprint arXiv:2105.04663, 2021

    2021 arXiv

  6. [14]

    An efficient 2d method for training super-large deep learning models,

    Q. Xu and Y . You, “An efficient 2d method for training super-large deep learning models,” in 2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS) . IEEE, 2023, pp. 222–232

    2023

  7. [15]

    Maximizing parallelism in distributed training for huge neural networks,

    Z. Bian, Q. Xu, B. Wang, and Y . You, “Maximizing parallelism in distributed training for huge neural networks,” arXiv preprint arXiv:2105.14450, 2021

    2021 arXiv

  8. [16]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    2017

  9. [17]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al. , “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929 , 2020

    2010 arXiv

  10. [18]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685 , 2021

    2021 arXiv

  11. [19]

    Parameter-efficient transfer learning for nlp,

    N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for nlp,” in International Conference on Machine Learning . PMLR, 2019, pp. 2790–2799

    2019

  12. [20]

    Transformer in transformer,

    K. Han, A. Xiao, E. Wu, J. Guo, C. Xu, and Y . Wang, “Transformer in transformer,” Advances in neural information processing systems , vol. 34, pp. 15 908–15 919, 2021

    2021

  13. [21]

    Dynamic model pruning with feedback,

    T. Lin, S. U. Stich, L. Barba, D. Dmitriev, and M. Jaggi, “Dynamic model pruning with feedback,” arXiv preprint arXiv:2006.07253 , 2020

    2006 arXiv

  14. [22]

    Single-shot pruning for pre-trained models: Rethinking the importance of magnitude pruning,

    H. Kohama, H. Minoura, T. Hirakawa, T. Yamashita, and H. Fujiyoshi, “Single-shot pruning for pre-trained models: Rethinking the importance of magnitude pruning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 1433–1442

    2023

  15. [23]

    Resource- efficient transformer pruning for finetuning of large models,

    F. Ilhan, G. Su, S. F. Tekin, T. Huang, S. Hu, and L. Liu, “Resource- efficient transformer pruning for finetuning of large models,” in Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 16 206–16 215

    2024

  16. [24]

    Martello and P

    S. Martello and P. Toth, Knapsack problems: algorithms and computer implementations. John Wiley & Sons, Inc., 1990

    1990

  17. [25]

    A class of generalized greedy algorithms for the multi-knapsack problem,

    A. R. Kan, L. Stougie, and C. Vercellis, “A class of generalized greedy algorithms for the multi-knapsack problem,” Discrete applied mathematics, vol. 42, no. 2-3, pp. 279–290, 1993

    1993

  18. [26]

    Dynamic programming revisited: Improving knapsack algorithms,

    U. Pferschy, “Dynamic programming revisited: Improving knapsack algorithms,” Computing, vol. 63, no. 4, pp. 419–430, 1999

    1999

  19. [27]

    An approximate dynamic programming ap- proach to multidimensional knapsack problems,

    D. Bertsimas and R. Demir, “An approximate dynamic programming ap- proach to multidimensional knapsack problems,” Management Science , vol. 48, no. 4, pp. 550–565, 2002

    2002

  20. [28]

    Pytorch image models,

    R. Wightman, “Pytorch image models,” https://github.com/rwightman/ pytorch-image-models, 2019

    2019

  21. [29]

    Pytorch: An imperative style, high-performance deep learning library,

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al. , “Pytorch: An imperative style, high-performance deep learning library,” in Advances in Neural Information Processing Systems (NeurIPS) , 2019, pp. 8024– 8035

    2019

  22. [30]

    Gshard: Scaling giant models with conditional computation and automatic sharding,

    D. Lepikhin, H. Lee, Y . Xu, D. Chen, O. Firat, Y . Huang, M. Krikun, N. Shazeer, and Z. Chen, “Gshard: Scaling giant models with conditional computation and automatic sharding,” arXiv preprint arXiv:2006.16668 , 2020

    2006 arXiv

  23. [31]

    Where to pay attention in sparse training for feature selection?

    G. Sokar, Z. Atashgahi, M. Pechenizkiy, and D. C. Mocanu, “Where to pay attention in sparse training for feature selection?” Advances in Neural Information Processing Systems , vol. 35, pp. 1627–1642, 2022

    2022

  24. [32]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,

    N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” arXiv preprint arXiv:1701.06538 , 2017

    2017 arXiv

  25. [33]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,

    W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,”The Journal of Machine Learning Research , vol. 23, no. 1, pp. 5232–5270, 2022

    2022

  26. [34]

    Flexmoe: Scaling large-scale sparse pre-trained model training via dynamic device placement,

    X. Nie, X. Miao, Z. Wang, Z. Yang, J. Xue, L. Ma, G. Cao, and B. Cui, “Flexmoe: Scaling large-scale sparse pre-trained model training via dynamic device placement,” Proceedings of the ACM on Management of Data , vol. 1, no. 1, pp. 1–19, 2023

    2023

  27. [35]

    M6-t: Exploring sparse expert models and beyond,

    A. Yang, J. Lin, R. Men, C. Zhou, L. Jiang, X. Jia, A. Wang, J. Zhang, J. Wang, Y . Li et al. , “M6-t: Exploring sparse expert models and beyond,” arXiv preprint arXiv:2105.15082 , 2021

    2021 arXiv

  28. [36]

    Taming sparsely activated transformer with stochastic experts,

    S. Zuo, X. Liu, J. Jiao, Y . J. Kim, H. Hassan, R. Zhang, T. Zhao, and J. Gao, “Taming sparsely activated transformer with stochastic experts,” arXiv preprint arXiv:2110.04260 , 2021

    2021 arXiv

  29. [37]

    Hash layers for large sparse models,

    S. Roller, S. Sukhbaatar, J. Weston et al. , “Hash layers for large sparse models,” Advances in Neural Information Processing Systems , vol. 34, pp. 17 555–17 566, 2021

    2021

  30. [38]

    Snip: Single-shot network pruning based on connection sensitivity,

    N. Lee, T. Ajanthan, and P. H. Torr, “Snip: Single-shot network pruning based on connection sensitivity,”arXiv preprint arXiv:1810.02340, 2018

    2018 arXiv

  31. [39]

    Picking winning tickets before training by preserving gradient flow,

    C. Wang, G. Zhang, and R. Grosse, “Picking winning tickets before training by preserving gradient flow,” arXiv preprint arXiv:2002.07376 , 2020

    2002 arXiv

  32. [40]

    Learning both weights and con- nections for efficient neural network,

    S. Han, J. Pool, J. Tran, and W. Dally, “Learning both weights and con- nections for efficient neural network,” Advances in neural information processing systems, vol. 28, 2015

    2015

  33. [41]

    Dynamic network surgery for efficient dnns,

    Y . Guo, A. Yao, and Y . Chen, “Dynamic network surgery for efficient dnns,” Advances in neural information processing systems , vol. 29, 2016

    2016

  34. [42]

    Compression-aware training of deep networks,

    J. M. Alvarez and M. Salzmann, “Compression-aware training of deep networks,” Advances in neural information processing systems , vol. 30, 2017

    2017

  35. [43]

    “learning-compression

    M. A. Carreira-Perpin ´an and Y . Idelbayev, ““learning-compression” al- gorithms for neural net pruning,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 8532–8541

    2018

  36. [44]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks,

    J. Frankle and M. Carbin, “The lottery ticket hypothesis: Finding sparse, trainable neural networks,” arXiv preprint arXiv:1803.03635 , 2018

    2018 arXiv

  37. [45]

    Efficient lottery ticket finding: Less data is more,

    Z. Zhang, X. Chen, T. Chen, and Z. Wang, “Efficient lottery ticket finding: Less data is more,” in International Conference on Machine Learning. PMLR, 2021, pp. 12 380–12 390

    2021

  38. [46]

    The lottery ticket hypothesis for object recognition,

    S. Girish, S. R. Maiya, K. Gupta, H. Chen, L. S. Davis, and A. Shri- vastava, “The lottery ticket hypothesis for object recognition,” in Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 762–771

    2021

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.