Pith. sign in

REVIEW 4 major objections 5 minor 41 references

ICE-Pruning: An Iterative Cost-Efficient Pruning Pipeline for Deep Neural Networks

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ICE-Pruning speeds deep-network pruning up to 9.61x by skipping unnecessary fine-tuning.

desk verdict A useful pruning pipeline whose headline speedup likely ignores auto-tuning cost. read the letter →

arxiv 2505.07411 v2 pith:GTO7D5LZ submitted 2025-05-12 cs.LG cs.CVstat.ML

Wenhao Hu , Paul Henderson , José Cano This is my paper · ORCID
classification cs.LGcs.CVstat.ML
keywords networkpruningiterativefine-tuninglayerfreezinglearningrateschedulinghyperparameterauto-tuningmodelcompressionstructured
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that fine-tuning after every pruning step is wasteful, because over-parameterized networks often lose little accuracy at intermediate steps. ICE-Pruning therefore fine-tunes only when the measured accuracy drop crosses a threshold, freezes the layers that change least during fine-tuning, and lowers the maximum learning rate as the network gets narrower. These three mechanisms cut the number and cost of fine-tuning rounds. The paper reports that this speeds pruning by up to 9.61x while keeping final accuracy close to, and sometimes above, existing pipelines.

What carries the argument

The mechanism that carries the argument is a threshold-gated fine-tuning loop. Each pruning step is followed by an accuracy test, and fine-tuning is triggered only when the drop is at least $\theta$. The second component is layer freezing: after one fine-tuning, per-layer weight changes are normalized by the layer weights, and the $\eta$ fraction of layers with the smallest changes stay frozen thereafter. The third is a pruning-aware learning-rate schedule, $LR_{\max} = LR_{\mathrm{base}} - \Delta / (1 + (\alpha / (2(1-p) - \alpha))^\beta)$, where $\alpha$ is the fraction of unpruned parameters, so narrower models receive lower maximum learning rates. An auto-tuning objective $(\mathit{PT} + \Delta A)/\max(\mathit{PT}, \Delta A)$ balances pruning time $\mathit{PT}$ and accuracy loss $\Delta A$. The same pruning criteria as in existing pipelines can be dropped in, making the efficiency gains independent of the choice of criterion.

What would settle it

Run the Stage 1 auto-tuning on several disjoint random subsets of the same dataset for the same model, then apply each resulting hyperparameter set to the full dataset. If the chosen threshold and freezing percentage vary so much across subsets that the reported speedups or accuracies do not reproduce, the transfer assumption that underpins the method is false.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that an iterative pruning pipeline can become much cheaper without introducing a new pruning criterion. After each pruning step the model is tested; fine-tuning runs only if the accuracy drop reaches a threshold. Layers whose normalized weight changes are small after the first fine-tuning are frozen in later steps, and a pruning-aware learning-rate scheduler caps the maximum learning rate based on how much of the network remains. An auto-tuning stage picks the few pipeline hyperparameters on a sampled subset and then applies them to the full dataset. The evaluation reports up to 9.61x faster pruning than a baseline iterative pipeline and a state-of-the-art automatic pipeline, with comparable or better final accuracy in most cases.

Load-bearing premise

The load-bearing premise is that tuning on a small random sample of the data gives settings (the accuracy-drop threshold, freezing percentage, and learning-rate schedule) that still work when the pipeline runs on the full dataset.

Editorial extensions

If this is right

  • For over-parameterized models, many intermediate pruning steps can skip fine-tuning entirely, so a target compression ratio is reached with substantially fewer training epochs.
  • The pipeline works with existing structured-pruning criteria, so teams can keep their preferred criterion and still get the time savings.
  • The speedup grows with pruning ratio, reaching 9.61x at 80% pruning, because that is where the largest number of fine-tuning rounds are avoided.
  • Final accuracy stays competitive with the baseline and automatic pipelines, and in some reported settings it is higher, so the time savings do not come from accepting a lower-quality model.
  • The auto-tuning stage removes the need for manual tuning of the introduced hyperparameters, making the pipeline plug-and-play.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same test-then-spend-compute gating could be applied to other expensive training loop operations, such as validation or checkpointing, whenever a cheap metric indicates that extra compute would not change the outcome.
  • The reported results are on convolutional vision models; the natural stress test is a model family with less stable layer-sensitivity rankings, where the one-step freezing rule could misfire.
  • Varying the sample size in Stage 1 across several random subsets would show how sensitive the chosen settings are, turning the representative-sample assumption into a measurable quantity.
  • An ablation that runs the full pipeline with always-fine-tune and never-fine-tune extremes would isolate how much of the 9.61x comes from skipping fine-tuning versus from freezing and learning-rate scheduling.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper proposes ICE-Pruning, an iterative structured-pruning pipeline that wraps existing pruning criteria rather than introducing a new one. The pipeline has two stages: Stage 1 auto-tunes six hyperparameters (accuracy-drop threshold, frozen-layer percentage, base learning rate, maximum LR change, the pruning-level parameter p, and shape parameter beta) by running the pruning-fine-tuning loop on a randomly subsampled dataset; Stage 2 reruns the loop on the full dataset with the selected hyperparameters. Three mechanisms reduce fine-tuning cost: skipping fine-tuning when the post-pruning accuracy drop is below a threshold, freezing layers whose normalized weight changes are small, and a pruning-aware LR scheduler that lowers the maximum LR as the average model width shrinks. The evaluation compares ICE-Pruning with a naive iterative baseline and with AAP on ResNet-152/CIFAR-10, DenseNet-121/TinyImageNet, and WRN-101-2/ImageNet, and reports speedups up to 9.61x at 80% pruning with similar or better accuracy in most cells.

Significance. If the headline speedup survives a complete accounting of all pipeline stages, this is a useful practical contribution: it is a model-agnostic wrapper, it supports multiple pruning criteria, it includes an ablation study of each component, and the code is released. The paper is explicit that it introduces no new pruning criterion, and the experimental claims are falsifiable through the reported times and accuracies. The main open risk is whether the 9.61x figure is an end-to-end wall-clock speedup or only a Stage-2 fine-tuning speedup; this is the difference between a strong systems result and a more modest one. The accuracy comparisons would also be more persuasive with repeated runs and variance estimates.

major comments (4)
  1. [Algorithm 1, Table III] The headline 9.61x speedup is not yet supported as an end-to-end wall-clock claim. In Algorithm 1, Stage 1 runs the complete PFT loop for every candidate hyperparameter on sampled data, and Stage 2 reruns PFT on the full dataset; the Time values in Table III appear to report only the Stage-2 PFT time. The paper never reports the number of hyperparameter candidates evaluated, the data subsampling ratio, or the duration of Stage 1. With, for example, 20 candidates on 10% of the data, Stage 1 alone would add roughly two full-pipeline equivalents of compute, which could materially shrink the reported speedup. In addition, the freezing strategy of Section III-B requires a warm-up pruning-plus-fine-tuning step to measure layer weight changes before Algorithm 2 calls Freeze(M, eta); the cost of that warm-up is not visible in the time figures. Please report a complete time budget that separates Stage 1 search cost, the freezing warm-up, and Stage 2, and recompute the speedups on the total.
  2. [Section IV-A, Algorithm 1] The experimental protocol is inconsistent with the algorithm description. Section IV-A says that for TinyImageNet and ImageNet, fine-tuning is performed on subsampled datasets, whereas Algorithm 1's Stage 2 calls PFT with the full dataset D. The sampling fractions are not given, and no distinction is made between the subset used for Stage 1 and any subset used in Stage 2. If the Table III numbers for DenseNet-121 and WRN-101-2 were obtained with subsampled fine-tuning data, then the comparison is not against full-data baseline and AAP runs on equal terms, and the 9.61x speedup applies to a different protocol than the one in the algorithm. Please clarify precisely which data each stage uses and report the subsampling ratios.
  3. [Section IV-E, Table III] The 'maintaining similar accuracy' claim is supported only loosely. The experiments appear to be single runs with no error bars, so differences in the table cannot be distinguished from noise. In several cells ICE-Pruning is a few points below a baseline: DenseNet-121 at 70% pruning (54.50 vs 57.00 for the baseline), WRN-101-2 at 70% (54.97 vs 55.65), and ResNet-152 at 80% (91.10 vs 91.20 for AAP). Please add repeated-seed experiments with means and standard deviations for at least the central comparisons, or explicitly qualify the accuracy claim as comparable in single-run evaluation.
  4. [Section III-D] The transferability of the Stage-1 hyperparameter search to the full dataset is asserted rather than demonstrated. The paper states that the subsampled data has a similar distribution to the whole dataset, but it reports no sampling fraction, no sensitivity analysis over that fraction, and no comparison of the objective in Eq. (2) on sampled versus full data. Since the six searched hyperparameters determine the speed-accuracy trade-off, an unrepresentative subset would directly invalidate the reported results. Please report the size of the search space and at least one experiment varying the subsampling ratio (for example 5%, 10%, and 25%) to validate the transfer.
minor comments (5)
  1. [Title] The title in the PDF reads 'An I terative Cost-Efficient...' with a stray space in 'Iterative'; please fix the typo.
  2. [Equation (1)] Equation (1) contains unmatched parentheses and is ambiguous as typeset; please rewrite it with explicit parentheses and define the domain of each variable in the same display.
  3. [Section III-D] In the paragraph after Eq. (2), 'since accuracyPTis always larger than Delta-A' should read 'since PT is always larger than Delta-A'; the current wording is ungrammatical.
  4. [Section III-D, Algorithm 2] Please state explicitly that the hyperparameter set lambda in Eq. (2) is the tuple (theta, eta, LR_base, Delta, p, beta) from Algorithm 2; the link is currently implicit.
  5. [Figure 1(b)] The four fine-tuning cases described in the text are not labeled in the figure; add labels or a legend so the reader can map cases 1-4 to the plotted curves.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: ICE-Pruning's speedup and accuracy claims are measured against external baselines; the auto-tuning stage is an optimization procedure, not a fitted prediction, and the single self-citation to ICE-Pick is not load-bearing.

full rationale

I examined the derivation chain in Algorithms 1 and 2, Equations (1)-(3), the ablation study, and the comparison with external baselines. The headline speedup (up to 9.61x) is a ratio of measured wall-clock times reported in Table III (e.g., 432s vs 1209s for DenseNet-121/TinyImageNet at 80% pruning) and is not the output of any equation whose inputs include the claimed result. Stage 1 auto-tuning minimizes Equation (2), a combined time/accuracy-loss objective, on a randomly sampled subset; the final reported times and accuracies in Stage 2 are measured outcomes of applying the selected hyperparameters, not the fitted objective values themselves renamed as predictions. This is parameter optimization, not a fitted-input-called-prediction pattern. The layer-freezing justification in Section IV-B rests on an empirical observation that the ranks of layer weight changes are maintained across pruning steps, not on a self-citation or an imported uniqueness theorem. The only self-citation, ICE-Pick [16], appears in Section II-B as a related-work remark ('To avoid the high cost of fine-tuning, ICE-Pick [16] tries to reduce the overall time consumption') and carries no load in the derivation, evaluation, or choice of pruning strategy. The pruning-aware learning rate scheduler (Equation 1) is explicitly inspired by the external S-Cyc work [28] and is ablated in Section IV-D, so it is not an ansatz smuggled in via self-citation. The auto-tuning objective and the subsequent evaluation do not reduce to one another by construction: the reported accuracy and time are empirical results of running the pipeline, not the value of the minimized objective. One non-circular caveat is that the paper does not report the wall-clock duration of Stage 1 auto-tuning, so the end-to-end speedup may be overstated; this is an evaluation-accounting gap, not a definitional or self-referential circularity.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

ICE-Pruning introduces no new physical entities. Its central claim rests on six auto-tuned hyperparameters and three empirical assumptions: subset representativeness, rank persistence of layer sensitivity, and the narrow-loss-landscape view of pruned models. These assumptions are reasonable for the tested CNNs but are not proven generally.

free parameters (6)
  • Accuracy drop threshold theta = auto-tuned, value not reported
    Controls whether a pruning step is followed by fine-tuning; chosen by grid search on a subsampled dataset in Stage 1.
  • Frozen layers percentage eta = auto-tuned, value not reported
    Determines the fraction of layers frozen during fine-tuning; chosen by grid search.
  • Base learning rate LR_base = auto-tuned for searched experiments; default 0.001 in Table I
    Initial maximum learning rate for the unpruned model; the paper says some experiments need to search for it.
  • Maximum LR change Delta = auto-tuned, value not reported
    Maximum decrease of the learning rate as pruning progresses; searched during auto-tuning.
  • Pruning level p = auto-tuned, value not reported
    Pruning level at which half of the LR decrease is reached; searched during auto-tuning.
  • Shape control beta = auto-tuned, value not reported
    Shape parameter of the LR decrease curve in Equation (1); searched during auto-tuning.
assumptions (3)
  • domain assumption Hyperparameters optimized on a random subset of the dataset also optimize the full dataset.
    Section III-D relies on the subset having a similar distribution to the full dataset and transfers Stage 1 hyperparameters to Stage 2 without re-validation. This is load-bearing and not formally justified.
  • domain assumption The relative sensitivity of layers to fine-tuning is maintained across pruning steps.
    Section III-B uses one pruning step to measure weight-change ranks and freezes the least sensitive layers for all later steps. Figure 2 shows this for two models, but no quantitative metric is reported.
  • domain assumption Narrower pruned models have narrower loss landscapes, motivating a lower maximum learning rate.
    Section III-C cites Li et al. [27] for this and uses it to justify the pruning-aware LR schedule. It is a borrowed empirical assumption, not derived.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ICE-Pruning: An Iterative Cost-Efficient Pruning Pipeline for Deep Neural Networks." pith.science (2026). https://pith.science/paper/GTO7D5LZ

@misc{pith2026250507411,
  author       = {Pith},
  title        = {Pith review of: ICE-Pruning: An Iterative Cost-Efficient Pruning Pipeline for Deep Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GTO7D5LZ}},
  note         = {Machine review of arXiv:2505.07411}
}
read the original abstract

Pruning is a widely used method for compressing Deep Neural Networks (DNNs), where less relevant parameters are removed from a DNN model to reduce its size. However, removing parameters reduces model accuracy, so pruning is typically combined with fine-tuning, and sometimes other operations such as rewinding weights, to recover accuracy. A common approach is to repeatedly prune and then fine-tune, with increasing amounts of model parameters being removed in each step. While straightforward to implement, pruning pipelines that follow this approach are computationally expensive due to the need for repeated fine-tuning. In this paper we propose ICE-Pruning, an iterative pruning pipeline for DNNs that significantly decreases the time required for pruning by reducing the overall cost of fine-tuning, while maintaining a similar accuracy to existing pruning pipelines. ICE-Pruning is based on three main components: i) an automatic mechanism to determine after which pruning steps fine-tuning should be performed; ii) a freezing strategy for faster fine-tuning in each pruning step; and iii) a custom pruning-aware learning rate scheduler to further improve the accuracy of each pruning step and reduce the overall time consumption. We also propose an efficient auto-tuning stage for the hyperparameters (e.g., freezing percentage) introduced by the three components. We evaluate ICE-Pruning on several DNN models and datasets, showing that it can accelerate pruning by up to 9.61x. Code is available at https://github.com/gicLAB/ICE-Pruning

Figures

Figures reproduced from arXiv: 2505.07411 by the authors.

Figure 1
Figure 1. ICE-Pruning pipeline: (a) Using subsampled datasets, we automatically search for the best hyperparameters by running [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Pruning time vs. accuracy of ICE-Pruning with four [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 2
Figure 2. Normalized weight changes (∆W / ∥W0∥2) per layer for each pruning step in DNN iterative pruning; ∆W is the weight changes, ∥W0∥2 is the L2 norm of the initial weights. The learning rate is 0.001 and the pruning ratio is 30%. that those layers are insensitive to fine-tuning, or at least it is not worth spending time on them for fine-tuning because of the small changes. We also observe that the ranks of the weight cha… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 27 canonical work pages

  1. [2]

    DLAS: A Conceptual Model for Across-Stack Deep Learning Acceleration,

    P. Gibson, J. Cano, E. Crowley, A. Storkey, and M. O’boyle, “DLAS: A Conceptual Model for Across-Stack Deep Learning Acceleration,”ACM Transactions on Architecture and Code Optimization, vol. 22, no. 1, Mar. 2025

    work page 2025

  2. [3]

    Linearly Replaceable Filters for Deep Network Channel Pruning,

    D. Joo, E. Yi, S. Baek, and J. Kim, “Linearly Replaceable Filters for Deep Network Channel Pruning,”Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 9, pp. 8021–8029, May 2021

    work page 2021

  3. [4]

    An entropy-based pruning method for cnn compression,

    J.-H. Luo and J. Wu, “An entropy-based pruning method for cnn compression,”ArXiv, vol. abs/1706.05791, 2017

    arXiv 2017

  4. [5]

    Thinet: A filter level pruning method for deep neural network compression,

    J.-H. Luo, J. Wu, and W. Lin, “Thinet: A filter level pruning method for deep neural network compression,” inProceedings of the IEEE international conference on computer vision, 2017, pp. 5058–5066

    work page 2017

  5. [6]

    Pruning filters for efficient convnets,

    H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, “Pruning filters for efficient convnets,” inInternational Conference on Learning Representations, 2017

    work page 2017

  6. [7]

    Automatic attention pruning: Improv- ing and automating model pruning using attentions,

    K. Zhao, A. Jain, and M. Zhao, “Automatic attention pruning: Improv- ing and automating model pruning using attentions,” inInternational Conference on Artificial Intelligence and Statistics, 2023, pp. 10 470– 10 486

    work page 2023

  7. [8]

    Pruning convolutional neural networks for resource efficient inference,

    P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz, “Pruning convolutional neural networks for resource efficient inference,” inIn- ternational Conference on Learning Representations, 2017

    2017

  8. [9]

    What is the State of Neural Network Pruning?

    D. Blalock, J. J. Gonzalez Ortiz, J. Frankle, and J. Guttag, “What is the State of Neural Network Pruning?” inProceedings of Machine Learning and Systems, vol. 2, 2020, pp. 129–146

    work page 2020

Show all 41 references
  1. [10]

    Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks,

    T. Hoefler, D. Alistarh, T. Ben-Nun, N. Dryden, and A. Peste, “Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks,”The Journal of Machine Learning Research, vol. 22, no. 1, pp. 10 882–11 005, 2021

    2021

  2. [11]

    Model compression and hardware acceleration for neural networks: A comprehensive survey,

    L. Deng, G. Li, S. Han, L. Shi, and Y . Xie, “Model compression and hardware acceleration for neural networks: A comprehensive survey,” Proceedings of the IEEE, vol. 108, no. 4, pp. 485–532, 2020

    2020

  3. [12]

    Soft filter pruning for accelerating deep convolutional neural networks,

    Y . He, G. Kang, X. Dong, Y . Fu, and Y . Yang, “Soft filter pruning for accelerating deep convolutional neural networks,” inProceedings of the 27th International Joint Conference on Artificial Intelligence, 2018, p. 2234–2240

    2018

  4. [13]

    Filter Pruning via Similarity Clustering for Deep Convolutional Neural Networks,

    K. Song, W. Yao, and X. Zhu, “Filter Pruning via Similarity Clustering for Deep Convolutional Neural Networks,” inInternational Conference on Neural Information Processing, 2022, pp. 88–99

    2022

  5. [14]

    Are Sixteen Heads Really Better than One?

    P. Michel, O. Levy, and G. Neubig, “Are Sixteen Heads Really Better than One?” inAdvances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch ´e-Buc, E. Fox, and R. Garnett, Eds., 2019

    2019

  6. [15]

    Pruning Convolutional Neural Networks for Resource Efficient Inference,

    P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz, “Pruning Convolutional Neural Networks for Resource Efficient Inference,” in International Conference on Learning Representations, 2017

    2017

  7. [16]

    ICE-pick: Iterative cost-efficient pruning for DNNs,

    W. Hu, P. Gibson, and J. Cano, “ICE-pick: Iterative cost-efficient pruning for DNNs,” inICML 2023 Workshop Neural Compression: From Information Theory to Applications, 2023

    2023

  8. [17]

    Nfp: A no fine- tuning pruning approach for convolutional neural network compression,

    R. Liu, J. Cao, P. Li, W. Sun, Y . Zhang, and Y . Wang, “Nfp: A no fine- tuning pruning approach for convolutional neural network compression,” in2020 3rd International Conference on Artificial Intelligence and Big Data (ICAIBD), 2020, pp. 74–77

    2020

  9. [18]

    Pruning filters while training for efficiently optimizing deep learning networks,

    S. Roy, P. Panda, G. Srinivasan, and A. Raghunathan, “Pruning filters while training for efficiently optimizing deep learning networks,” in International Joint Conference on Neural Networks, 2020, pp. 1–7

    2020

  10. [19]

    FreezeOut: Accelerate Training by Progressively Freezing Layers,

    A. Brock, T. Lim, J. M. Ritchie, and N. Weston, “FreezeOut: Accelerate Training by Progressively Freezing Layers,” in10th NIPS Workshop on Optimization for Machine Learning, 2017

    2017

  11. [20]

    AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning,

    Y . Liu, S. Agarwal, and S. Venkataraman, “AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning,”arXiv preprint arXiv:2102.01386, 2021

    2021 arXiv

  12. [21]

    What would elsa do? freezing layers during transformer fine-tuning,

    J. Lee, R. Tang, and J. Lin, “What would elsa do? freezing layers during transformer fine-tuning,”arXiv preprint arXiv:1911.03090, 2019

    1911 arXiv

  13. [22]

    Local Masking Meets Progressive Freezing: Crafting Efficient Vision Transformers for Self-Supervised Learning,

    U. M. Topc ¸uo˘glu and E. Akag ¨und¨uz, “Local Masking Meets Progressive Freezing: Crafting Efficient Vision Transformers for Self-Supervised Learning,” inSeventeenth International Conference on Machine Vision (ICMV), 2024

    2024

  14. [23]

    The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks,

    J. Frankle and M. Carbin, “The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks,” inInternational Conference on Learning Representations, 2019

    2019

  15. [24]

    Do we actually need dense over-parameterization? in-time over-parameterization in sparse training,

    S. Liu, L. Yin, D. C. Mocanu, and M. Pechenizkiy, “Do we actually need dense over-parameterization? in-time over-parameterization in sparse training,” in38th International Conference on Machine Learning, vol. 139, 18–24 Jul 2021, pp. 6989–7000

    2021

  16. [25]

    Towards explaining the regularization effect of initial large learning rate in training neural networks,

    Y . Li, C. Wei, and T. Ma, “Towards explaining the regularization effect of initial large learning rate in training neural networks,”Advances in Neural Information Processing Systems, vol. 32, 2019

    2019

  17. [26]

    On the Variance of the Adaptive Learning Rate and Beyond,

    L. Liu, H. Jiang, P. He, W. Chen, X. Liu, J. Gao, and J. Han, “On the Variance of the Adaptive Learning Rate and Beyond,” inInternational Conference on Learning Representations, 2020

    2020

  18. [27]

    Visualizing the loss landscape of neural nets,

    H. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein, “Visualizing the loss landscape of neural nets,”Advances in neural information processing systems, vol. 31, 2018

    2018

  19. [28]

    S-Cyc: A Learning Rate Schedule for Iterative Pruning of ReLU-based Networks,

    S. Liu, C. M. J. Tan, and M. Motani, “S-Cyc: A Learning Rate Schedule for Iterative Pruning of ReLU-based Networks,”arXiv preprint arXiv:2110.08764, 2021

    2021 arXiv

  20. [29]

    Algorithms for hyper- parameter optimization,

    J. Bergstra, R. Bardenet, Y . Bengio, and B. K ´egl, “Algorithms for hyper- parameter optimization,” inAdvances in Neural Information Processing Systems, vol. 24, 2011

    2011

  21. [30]

    Optuna: A Next-generation Hyperparameter Optimization Framework,

    T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, “Optuna: A Next-generation Hyperparameter Optimization Framework,” in25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2019

    2019

  22. [31]

    Learning Multiple Layers of Features from Tiny Im- ages,

    A. Krizhevsky, “Learning Multiple Layers of Features from Tiny Im- ages,” University of Toronto, Tech. Rep., 2009

    2009

  23. [32]

    Tiny ImageNet Visual Recognition Challenge,

    Y . Le and X. Yang, “Tiny ImageNet Visual Recognition Challenge,” Stanford University, Tech. Rep., 2015

    2015

  24. [33]

    Imagenet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 248–255

    2009

  25. [34]

    Distilling the Knowledge in a Neural Network,

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the Knowledge in a Neural Network,”arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  26. [35]

    Distilling with Performance Enhanced Students,

    J. Turner, E. J. Crowley, V . Radu, J. Cano, A. Storkey, and M. O’Boyle, “Distilling with Performance Enhanced Students,”arXiv preprint arXiv:1810.10460, 2019

    2019 arXiv

  27. [36]

    Deep Residual Learning for Image Recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,”IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778, 2016

    2016

  28. [37]

    Densely connected convolutional networks,

    G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” inIEEE conference on computer vision and pattern recognition, 2017, pp. 4700–4708

    2017

  29. [38]

    Wide Residual Networks,

    S. Zagoruyko and N. Komodakis, “Wide Residual Networks,”arXiv preprint arXiv:1605.07146, 2017

    2017 arXiv

  30. [39]

    An Overview of Gradient Descent Optimization Algorithms,

    S. Ruder, “An Overview of Gradient Descent Optimization Algorithms,” arXiv preprint arXiv:1609.04747, 2016

    2016 arXiv

  31. [40]

    Pytorch: An imperative style, high-performance deep learning library,

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antigaet al., “Pytorch: An imperative style, high-performance deep learning library,”Advances in neural information processing systems, vol. 32, 2019

    2019

  32. [41]

    MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications,

    A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications,”arXiv:1704.04861, 2017

    2017 arXiv

  33. [42]

    A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models,

    J. Ye, X. Chen, N. Xu, C. Zu, Z. Shao, S. Liu, Y . Cui, Z. Zhou, C. Gong, Y . Shen, J. Zhou, S. Chen, T. Gui, Q. Zhang, and X. Huang, “A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models,”arXiv preprint arXiv:2303.10420, 2023

    2023 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.