Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Demystifying Learning Rate Policies for High Accuracy Training of Deep Neural Networks

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a structured benchmark of 13 learning-rate schedules can select policies that beat deep-learning framework defaults in accuracy and cost.

desk verdict A practically useful LR-benchmarking paper with the right code and the wrong evaluation protocol: the headline gains are selected on the same test sets used for reporting, so trust the tool, not the margins. read the letter →

arxiv 1908.06477 v2 pith:TNB2FDPW submitted 2019-08-18 cs.LG stat.ML

Yanzhao Wu , Ling Liu , Juhyun Bae , Ka-Ho Chow , Arun Iyengar , Calton Pu , Wenqi Wei , Lei Yu
show 1 more author
Qi Zhang
This is my paper · ORCID
classification cs.LGstat.ML
keywords learningratepoliciesdeepneuralnetworktraininghyperparametertuningcyclicratesbenchmarkingclassificationconfidencemetricsLRBenchCIFAR-10
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that learning-rate selection for deep neural networks can be turned from a manual trial-and-error chore into a structured benchmarking problem. It unifies 13 learning-rate functions—fixed, decaying, and cyclic—under one parameterized formula and defines metrics for utility, cost, and robustness of a learning-rate policy. These metrics drive LRBench, a system that estimates a good learning-rate value range from a small set of runs, transfers that range to cyclic schedules, and ranks policies. On MNIST with LeNet and on CIFAR-10 with a three-layer CNN and ResNet-32, LRBench-chosen schedules achieve higher top-1 accuracy than the framework defaults while often using substantially fewer iterations. The paper's payoff, if the transferability assumption holds, is that developers can choose good learning-rate policies cheaply and avoid bad ones without exhaustive hyperparameter search.

What carries the argument

The unifying mechanism is the parameterized learning-rate update formula $$\eta(t) = |k_0-k_1|\,g(t)+\min(k_0,k_1),$$ with $g(t) \in [0,1]$ a schedule function; it expresses all 13 fixed, decaying, and cyclic policies as choices of the value-range endpoints $k_0,k_1$ and the schedule function $g(t)$. LRBench operationalizes this by sweeping a small set of candidate values on a model-dataset pair, ranking them with four metric families—average classification confidence (AC), confidence deviation (CD and CDAC), loss difference (LD), and iteration cost—and then transferring the best range and step size to related cyclic schedules. The transfer heuristic is the load-bearing step: it cuts the search space from the full $[0,1]$ learning-rate domain to a narrow interval (a 99.41% reduction for CIFAR-10) and lets a developer set $k_1$ near the upper bound of the constant-LR range and $k_0$ a decade below the lower bound.

What would settle it

On a held-out dataset such as SVHN or CIFAR-100, run LRBench's procedure: find the good constant-LR range, set $k_1$ near its upper bound and $k_0$ one decade below its lower bound for a cyclic schedule such as TRI2, and compare the resulting top-1 accuracy with a default or random policy. If the transferred range yields no improvement, or if the best range on the new dataset lies outside the transferred one, the transferability premise at the core of LRBench is wrong.

Watch

Extended reading notes

Core claim

The paper's central claim is that every learning-rate schedule can be understood as a combination of a value range, a step or cycle length, and an update function, and that benchmarking a few candidate policies in that space is enough to find good ones. The evidence is a set of head-to-head comparisons: on CIFAR-10 with a three-layer CNN, the LRBench-selected SINEXP schedule ($k_0=0.00005$, $k_1=0.006$, $\gamma=0.99994$, $l=2000$) reaches 82.16% top-1 accuracy versus 81.61% for the default NSTEP policy, and it does so at iteration 52,000 rather than the full 70,000; on ResNet-32, SINEXP reaches 92.81% versus 92.38% for the original paper's NSTEP setting. On MNIST with LeNet, SIN2 reaches 99.33% at 4,000 iterations versus 99.12% at 10,000 for the default. The authors attribute these gains to the discovery of good learning-rate value ranges, such as [0.0005, 0.006] for CNN3 on CIFAR-10, and to step sizes that match multiples of an epoch.

Load-bearing premise

The paper's recommendations all depend on the idea that a good learning-rate range discovered with one schedule and one network stays good when carried to other schedules and other networks; the paper never tests that transfer on a truly new dataset or architecture.

Editorial extensions

If this is right

  • A developer who runs one constant-learning-rate sweep over a few epochs can read off a good value range and use it to configure cyclic schedules without a second exhaustive search.
  • Cyclic schedules with a low lower bound and a high upper bound beat fixed-step decay in the tested cases, so the ranking can upgrade default policies by roughly 0.4 to 0.6 accuracy points on CIFAR-10.
  • The cost metric enables early stopping: on MNIST with LeNet, the best policy reaches its peak accuracy at 4,000 iterations, less than half the default 10,000, so training can be truncated without loss.
  • The metrics separate accuracy from confidence stability; on CNN3 the top-accuracy policies have slightly higher confidence deviation than the default NSTEP, so a user who values stable class predictions may deliberately choose a lower-accuracy policy.
  • Because the LR module and metrics are implemented as independent components, the benchmarking procedure can be lifted into other deep-learning frameworks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the transfer heuristic holds beyond the two reported cases, LRBench's stored database becomes a recommendation engine: a new dataset's good range could be estimated from a single constant-LR sweep over a few epochs, and cyclic schedules tuned from that range without exhaustive search. The paper demonstrates the transfer only from one CNN to another on CIFAR-10, so a natural test is to apply the
  • The reported good range [0.0005, 0.006] for CIFAR-10 brackets the framework default of 0.001 by about a factor of ten on each side; a plausible pattern not stated in the paper is that the optimal range scales with a gradient-noise or batch-size factor, which would let the heuristic be replaced by a formula.
  • The confidence metrics are used only for post-hoc evaluation, but they could serve as online training signals: if confidence deviation rises during training, that may flag an overly large learning rate and trigger an early schedule switch. The paper does not test this, so it remains an extension.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents LRBench, a benchmarking system for learning-rate (LR) policies. It categorizes 13 LR functions (fixed, decaying, and cyclic), decomposes them into range, step, and update parameters, and proposes utility, cost, and robustness metrics (Top-1/Top-5 accuracy, average confidence, confidence deviation, loss difference, iteration cost). The central claim is that LRBench can assist end-users and DNN developers to select good LR policies and avoid bad ones, based on experiments on MNIST and CIFAR-10 with LeNet, CNN3, and ResNet-32. The paper reports accuracy improvements such as SINEXP reaching 82.16% versus NSTEP's 81.61% on CNN3/CIFAR-10 (Table VIII) and 92.81% versus 92.38% on ResNet-32 (Table IX).

Significance. If the central claim is established, the paper would provide a useful systematic study of LR-schedule selection, a practical open-source benchmarking tool, and a taxonomy that could aid practitioners. The strengths include a comprehensive characterization of 13 LR functions, a clearly described set of metrics, and repeated-run measurements for Top-1 accuracy in the main tables. However, the empirical evidence is currently weakened by the absence of a held-out validation protocol: the same test sets are used both for selecting the recommended policies and for reporting their final accuracy, which risks selection bias. The transferability of the learned LR ranges and schedules across LR functions and models is also asserted rather than validated. These issues affect the load-bearing empirical claim, though they are addressable in a major revision.

major comments (3)
  1. [Section IV, Section V-C, Tables VII-IX] The LR policy ranking algorithm in Section IV selects top-N policies 'based on the empirical measurement results,' and Section V-C states that 'we primarily show the top LR policies recommended by LRBench.' The same 10,000-example test sets are used for both selection and final reporting, with no validation split. Because 13 LR functions and multiple grid values are evaluated, the number of candidate policies is large, and the best-of-many test-set accuracies can be inflated. For example, the 0.55-point Top-1 gap between SINEXP (82.16%) and NSTEP (81.61%) in Table VIII could plausibly arise from selection noise among many configurations. The paper should either use a validation set for selection and a held-out test set for final reporting, or report the total number of configurations evaluated and apply a correction for multiple comparisons (e.g., a significance test on the selected policy versus the baseline).
  2. [Section V-A, Section V-B] The load-bearing premise that good LR ranges and schedules transfer across LR functions and models is not validated. Section V-A uses the FIX range [0.0005, 0.006] learned for CNN3 on CIFAR-10 to set the k1 bound for TRI2 and other cyclic LR functions, and Section V-B carries step-size schedules l = 1000, 5000, 10000 from CNN3 to ResNet-32. The paper never tests whether these ranges or schedules remain near-optimal on a held-out dataset or architecture. Without such evidence, the recommendation mechanism has no demonstrable foundation beyond the specific configurations already searched. Please either validate the transfer empirically (e.g., sweep a smaller grid on a new model/dataset) or explicitly frame these as heuristics whose validity is untested.
  3. [Section V, Tables II-IX] The empirical scope is limited to two small datasets (MNIST, CIFAR-10) and three small models (LeNet, CNN3, ResNet-32). No larger-scale dataset (e.g., CIFAR-100, ImageNet) or modern architecture (e.g., deeper ResNets, WideResNet) is considered. Given the paper's abstract and conclusion claim that LRBench can 'demystify' LR policy tuning generally, the current breadth is insufficient to support that general claim. A single additional held-out workload would substantially strengthen the paper.
minor comments (6)
  1. [Section V-C, Tables VII-IX] Only Top-1 accuracy is reported as mean ± stddev over five runs; Top-5, AC, CD, CDAC, and LD are reported as single values. Because several LRBench recommendations rely on these metrics (e.g., the claim that CLRs have lower LD), the lack of repeated measurements or error bars makes it impossible to judge whether differences in these metrics are meaningful.
  2. [Section V-C, Table VII] In the paragraph summarizing MNIST results, 'NSETP' is a typo for 'NSTEP.'
  3. [Section VI] The related-work section includes a typo: 'CIRFAR-10' should be 'CIFAR-10.'
  4. [Section II, References] In the introduction, 'ResNet [20]' is cited as a popular network, but reference [20] is the channel-pruning paper, not the original ResNet paper (which is [5]); please correct the citation.
  5. [Section V-A] The 99.41% search-space reduction claim is computed only over the one-dimensional LR value range [0,1] and does not account for the other parameters (k0, k1, l, gamma, etc.) in the actual tuning process, so the reduction is misleading as stated.
  6. [Figure 1] The three-panel visualization of optimization paths with FIX, NSTEP, and TRIEXP is not clearly tied to the quantitative claims in the text; please add a description of how the paths were obtained (e.g., which dataset, loss surface, or projection) and how the reader should interpret the 'global optimum' label.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports measured LR-policy search results and does not present a derivation that reduces to its own inputs.

full rationale

LRBench's central outputs are empirical selections: Section IV states that the ranking algorithm selects top-N policies "based on the empirical measurement results", and Tables VI-IX report the measured accuracies of those selected policies, with external anchors such as Caffe defaults, Smith's [22] CLR settings, and He et al.'s [5] ResNet default. The good LR ranges and step sizes in Section V are summaries of sweeps (e.g., Figures 3-5, Tables IV-VI), not quantities derived from the target claim by construction. The transfer of a FIX-derived range to TRI2 is explicitly a search heuristic ("LRBench will first determine the search space for good value ranges of TRI2 by leveraging the good value range learned from FIX"), and the resulting TRI2/TRIEXP/SINEXP accuracies are then measured rather than inferred, so no equation or defined quantity is equivalent to its own input. Self-citations [16]-[18] are contextual benchmarking references and are not load-bearing for the empirical conclusions. The absence of a held-out evaluation protocol is a legitimate correctness and validity concern about selection bias, but selection on the evaluation set is a methodological risk, not a circular reduction under the criteria used here.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

The central claim is not derived from first principles; it is an empirical claim supported by test accuracy measurements. The free parameters listed above are all values chosen or optimized on the same datasets used for evaluation. The axioms are domain assumptions about what accuracy on these benchmarks tells us about LR quality and about transfer of ranges across schedules and models. No invented physical entities are introduced; LRBench is a software artifact and the proposed metrics are definitions.

free parameters (6)
  • k0 (initial or minimum LR) = e.g., 0.1, 0.01, 0.001, 0.00005, 0.0001
    Chosen by hand or by LRBench sweeps to maximize measured test accuracy; recommended ranges such as [0.0005, 0.006] for CNN3 on CIFAR-10 are fitted to test data in Section V-A.
  • k1 (maximum LR for CLRs) = e.g., 0.006, 0.06, 0.3, 0.6, 0.9
    Upper bound of cyclic ranges, selected by scanning values scaled from the FIX good range; see Section V-A and Tables IV and V.
  • gamma (decay factor) = e.g., 0.85, 0.94, 0.99994, 0.32, 0.1
    Decay multipliers for STEP, EXP, TRIEXP, SINEXP, and NSTEP; fitted to maximize accuracy in Tables IV, VI, VII, VIII, and IX.
  • l (step or cycle length) = e.g., 1000, 2000, 5000, 7000, 10000, 2000
    Update schedule interval, chosen as multiples of epochs and tuned in Section V-B; affects both decaying and cyclic policies.
  • NSTEP schedule l_i = e.g., [30000, 50000, 60000, 65000]
    Variable step sizes selected to align with observed loss or accuracy plateaus; see Section V-B and Table VI.
  • p (polynomial or inverse-time exponent) = 1.2, 0.75
    Used for POLY and INV in Table VII; fixed by hand for the experiments.
assumptions (3)
  • domain assumption Accuracy on a held-out test set is the appropriate proxy for LR policy quality.
    The paper uses Top-1 and Top-5 test accuracy and confidence metrics as the utility measure; if test accuracy is not the target, the recommendations may not transfer.
  • ad hoc to paper The good LR value range found for the FIX policy transfers to other LR functions.
    Section V-A uses the FIX range [0.0005, 0.006] for CNN3 on CIFAR-10 to bound the TRI2 search, but this transfer is not independently validated.
  • domain assumption Default hyperparameters and training lengths from Caffe are sufficient for comparing LR policies.
    Experiments keep all non-LR hyperparameters fixed and run 10,000 to 70,000 iterations; conclusions are conditioned on these defaults.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Demystifying Learning Rate Policies for High Accuracy Training of Deep Neural Networks." pith.science (2026). https://pith.science/paper/TNB2FDPW

@misc{pith2026190806477,
  author       = {Pith},
  title        = {Pith review of: Demystifying Learning Rate Policies for High Accuracy Training of Deep Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TNB2FDPW}},
  note         = {Machine review of arXiv:1908.06477}
}
read the original abstract

Learning Rate (LR) is an important hyper-parameter to tune for effective training of deep neural networks (DNNs). Even for the baseline of a constant learning rate, it is non-trivial to choose a good constant value for training a DNN. Dynamic learning rates involve multi-step tuning of LR values at various stages of the training process and offer high accuracy and fast convergence. However, they are much harder to tune. In this paper, we present a comprehensive study of 13 learning rate functions and their associated LR policies by examining their range parameters, step parameters, and value update parameters. We propose a set of metrics for evaluating and selecting LR policies, including the classification confidence, variance, cost, and robustness, and implement them in LRBench, an LR benchmarking system. LRBench can assist end-users and DNN developers to select good LR policies and avoid bad LR policies for training their DNNs. We tested LRBench on Caffe, an open source deep learning framework, to showcase the tuning optimization of LR policies. Evaluated through extensive experiments, we attempt to demystify the tuning of LR policies by identifying good LR policies with effective LR value ranges and step sizes for LR update schedules.

Figures

Figures reproduced from arXiv: 1908.06477 by the authors.

Figure 1
Figure 1. Visualization of the Training Process with Different LRs proposed triangle-based CLRs with periodic schedule, cut￾in-half cyclic decay schedule (TRI2) and exponential decay schedule (TRIEXP). [23] proposed a variant of triangle based CLRs using the cosine function. We also implement three sin￾based CLR functions: sin, sin2 and sin exp, corresponding to the TRI, TRI2 and TRIEXP. For each cyclic LR function, one needs… view at source ↗
Figure 2
Figure 2. Architecture of LRBench IV. LRBENCH LRBench is an LR benchmarking system that provides auto￾mated or semi-automated tuning and optimization for finding and selecting a good LR policy when DNN developers or end￾users have chosen the dataset and the DNN model for training. LRBench consists of four main functional components as shown in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Acc/Error with Varying k0 (FIX, CIFAR-10, CNN3) V. EXPERIMENTAL RESULTS AND ANALYSIS We conduct experiments using LRBench to show how the LR tuning is performed and the impact of datasets and DNN models on the utility, cost and robustness metrics of various LR policies. For each pair of the dataset and neural network, i.e., CIFAR-10 with CNN3, we only vary the default setting of LR value and keep the default setting… view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: NSTEP (CIFAR-10, CNN3) NSTEP. NSTEP is a multi-step decaying LR function with variable step sizes, aiming to update the LR value when the test accuracy or loss fails to improve. Table VI shows the the highest Top-1 accuracy and the corresponding #Iterations for trainin…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Performance Optimization of Ratings-Based Reinforcement Learning

    cs.LG 2025-01 reject novelty 4.0 of 10

    An empirical study proposes hyperparameter settings that improve rating-based RL on three control tasks, but the gains are measured on the same runs used to select the settings.

Reference graph

Works this paper leans on

32 extracted references · 18 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gradient-based learning applied to document recognition,

    Y . Lecun, L. Bottou, Y . Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE , vol. 86, no. 11, pp. 2278–2324, Nov 1998

    work page 1998

  2. [2]

    Imagenet classification with deep convolutional neural networks,

    A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems 25 , F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, Eds. Curran Associates, Inc., 2012, pp. 1097–1105. [Online]. Available: http://papers.nips.cc/paper/4824- imagenet-classifica...

    work page 2012

  3. [3]

    No more pesky learning rates,

    T. Schaul, S. Zhang, and Y . LeCun, “No more pesky learning rates,” in Proceedings of the 30th International Conference on International Conference on Machine Learning - Volume 28 , ser. ICML’13. JMLR.org, 2013, pp. III–343–III–351. [Online]. Available: http://dl.acm.org/citation.cfm?id=3042817.3042975

    arXiv 2013

  4. [4]

    An overview of gradient descent optimization algorithms,

    S. Ruder, “An overview of gradient descent optimization algorithms,” arXiv e-prints, p. arXiv:1609.04747, Sep 2016

    arXiv 2016

  5. [5]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016

    2016

  6. [6]

    Stochastic gradient descent tricks,

    L. Bottou, “Stochastic gradient descent tricks,” in Neural networks: Tricks of the trade . Springer, 2012, pp. 421–436

    work page 2012

  7. [7]

    On the momentum term in gradient descent learning algorithms,

    N. Qian, “On the momentum term in gradient descent learning algorithms,” Neural Networks , vol. 12, no. 1, pp. 145 – 151,

    work page

  8. [8]

    On the importance of initialization and momentum in deep learning,

    I. Sutskever, J. Martens, G. Dahl, and G. Hinton, “On the importance of initialization and momentum in deep learning,” in Proceedings of the 30th International Conference on Machine Learning , ser. Proceedings of Machine Learning Research, vol. 28. Atlanta, Georgia, USA: PMLR, 17–19 Jun 2013, pp. 1139–1147. [Online]. Available: http://proceedings.mlr.pres...

    work page 2013

Show all 32 references
  1. [9]

    Adaptive subgradient methods for online learning and stochastic optimization,

    J. Duchi, E. Hazan, and Y . Singer, “Adaptive subgradient methods for online learning and stochastic optimization,” J. Mach. Learn. Res., vol. 12, pp. 2121–2159, Jul. 2011. [Online]. Available: http://dl.acm.org/citation.cfm?id=1953048.2021068

    2011

  2. [10]

    ADADELTA: An Adaptive Learning Rate Method,

    M. D. Zeiler, “ADADELTA: An Adaptive Learning Rate Method,” arXiv e-prints, p. arXiv:1212.5701, Dec 2012

    2012 arXiv

  3. [11]

    Adam: A method for stochastic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” CoRR, vol. abs/1412.6980, 2014. [Online]. Available: http://arxiv.org/abs/1412.6980

    2014 arXiv

  4. [12]

    Bengio, Practical Recommendations for Gradient-Based Training of Deep Architectures

    Y . Bengio, Practical Recommendations for Gradient-Based Training of Deep Architectures . Berlin, Heidelberg: Springer Berlin Heidelberg, 2012, pp. 437–478. [Online]. Available: https://doi.org/10.1007/978-3- 642-35289-8 26

    2012 doi

  5. [13]

    Hot Swapping for On- line Adaptation of Optimization Hyperparameters,

    K. Bache, D. DeCoste, and P. Smyth, “Hot Swapping for On- line Adaptation of Optimization Hyperparameters,” arXiv e-prints , p. arXiv:1412.6599, Dec 2014

    2014 arXiv

  6. [14]

    The Effects of Hyperparameters on SGD Training of Neural Networks,

    T. M. Breuel, “The Effects of Hyperparameters on SGD Training of Neural Networks,” arXiv e-prints, p. arXiv:1508.02788, Aug 2015

    2015 arXiv

  7. [15]

    Dawnbench: An end- to-end deep learning benchmark and competition,

    C. Coleman, D. Narayanan, D. Kang, T. Zhao, J. Zhang, L. Nardi, P. Bailis, K. Olukotun, C. R ´e, and M. Zaharia, “Dawnbench: An end- to-end deep learning benchmark and competition,” in NIPS ML Systems Workshop, 2017

    2017

  8. [16]

    Benchmark- ing Deep Learning Frameworks: Design Considerations, Metrics and Beyond,

    L. Liu, Y . Wu, W. Wei, W. Cao, S. Sahin, and Q. Zhang, “Benchmark- ing Deep Learning Frameworks: Design Considerations, Metrics and Beyond,” in 2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS) , July 2018, pp. 1258–1269

    2018

  9. [17]

    Experimental Characterizations and Analysis of Deep Learning Frameworks,

    Y . Wu, , W. Cao, S. Sahin, and L. Liu, “Experimental Characterizations and Analysis of Deep Learning Frameworks,” in 2018 IEEE 38th International Conference on Big Data , December 2018

    2018

  10. [18]

    A comparative measurement study of deep learning as a service framework,

    Y . Wu, L. Liu, C. Pu, W. Cao, S. Sahin, W. Wei, and Q. Zhang, “A comparative measurement study of deep learning as a service framework,” IEEE Transactions on Services Computing , pp. 1–1, 2019

    2019

  11. [19]

    Caffe: Convolutional architecture for fast feature embedding,

    Y . Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell, “Caffe: Convolutional architecture for fast feature embedding,” in Proceedings of the 22Nd ACM International Conference on Multimedia , ser. MM ’14. New York, NY , USA: ACM, 201...

    2014

  12. [20]

    Channel pruning for accelerating very deep neural networks,

    Y . He, X. Zhang, and J. Sun, “Channel pruning for accelerating very deep neural networks,” in The IEEE International Conference on Computer Vision (ICCV) , Oct 2017

    2017

  13. [21]

    Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates,

    L. N. Smith and N. Topin, “Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates,” arXiv e-prints , p. arXiv:1708.07120, Aug 2017

    2017 arXiv

  14. [22]

    Cyclical Learning Rates for Training Neural Networks,

    L. N. Smith, “Cyclical Learning Rates for Training Neural Networks,” arXiv e-prints, p. arXiv:1506.01186, Jun. 2015

    2015 arXiv

  15. [23]

    SGDR: Stochastic Gradient Descent with Warm Restarts,

    I. Loshchilov and F. Hutter, “SGDR: Stochastic Gradient Descent with Warm Restarts,” arXiv e-prints, p. arXiv:1608.03983, Aug. 2016

    2016 arXiv

  16. [24]

    Cost-sensitive learning of deep feature representations from imbalanced data,

    S. H. Khan, M. Hayat, M. Bennamoun, F. A. Sohel, and R. Togneri, “Cost-sensitive learning of deep feature representations from imbalanced data,” IEEE Transactions on Neural Networks and Learning Systems , vol. 29, no. 8, pp. 3573–3587, Aug 2018

    2018

  17. [25]

    ”caffe: a fast open framework for deep learning

    Caffe, “”caffe: a fast open framework for deep learning.”,” https://github. com/BVLC/caffe, 2018, [Online; accessed 23-Sep-2018]

    2018

  18. [26]

    Understanding the difficulty of training deep feedforward neural networks,

    X. Glorot and Y . Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , ser. Proceedings of Machine Learning Research, Y . W. Teh and M. Tittering...

    2010

  19. [27]

    Accurate, large minibatch SGD: training imagenet in 1 hour,

    P. Goyal, P. Doll ´ar, R. B. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y . Jia, and K. He, “Accurate, large minibatch SGD: training imagenet in 1 hour,” CoRR, vol. abs/1706.02677, 2017. [Online]. Available: http://arxiv.org/abs/1706.02677

    2017 arXiv

  20. [28]

    ”understanding learning rates and how it improves performance in deep learning

    H. Zulkifli, “”understanding learning rates and how it improves performance in deep learning”,” https://towardsdatascience.com/ understanding-learning-rates-and-how-it-improves-performance-in- deep-learning-d0d4059c1c10, 2018, [Online; accessed 23-Sep-2018]

    2018

  21. [29]

    ”hyperopt — distributed asynchronous hyper- parameter optimization in python

    Hyperopt Developers, “”hyperopt — distributed asynchronous hyper- parameter optimization in python”,” http://hyperopt.github.io/hyperopt/, 2019, [Online; accessed 13-Aug-2019]

    2019

  22. [30]

    Sequential model- based optimization for general algorithm configuration,

    F. Hutter, H. H. Hoos, and K. Leyton-Brown, “Sequential model- based optimization for general algorithm configuration,” in Learning and Intelligent Optimization , C. A. C. Coello, Ed. Berlin, Heidelberg: Springer Berlin Heidelberg, 2011, pp. 507–523

    2011

  23. [31]

    Optuna: A next- generation hyperparameter optimization framework,

    T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, “Optuna: A next- generation hyperparameter optimization framework,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , ser. KDD ’19. New York, NY , USA: ACM, 2019, pp. 2623...

    2019

  24. [1999]

    Available: http://www.sciencedirect.com/science/article/ pii/S0893608098001166

    [Online]. Available: http://www.sciencedirect.com/science/article/ pii/S0893608098001166

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.