Pith. sign in

REVIEW 6 major objections 6 minor 39 references

ExpTest: Automating Learning Rate Searching and Tuning with Insights from Linearized Neural Networks

T0 review · 6 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ExpTest treats the training loss curve as a signal and uses an exponential-versus-linear hypothesis test to set and anneal the learning rate without manual selection.

desk verdict A genuinely new automatic LR search and annealing method built on linearized-network bounds, but the F-test at its core is uncalibrated and the empirical claims are overstated; worth refereeing with major revision. read the letter →

arxiv 2411.16975 v1 pith:MV3W4QCB submitted 2024-11-25 cs.LG cs.AI

Zan Chaudhry , Naoko Mizuno This is my paper · ORCID
classification cs.LGcs.AI MSC 68T07
keywords learningratesearchschedulinglosscurveanalysisexponentialdecaylinearizedneuralnetworkstangentkernelhypothesistestingstochasticgradientdescent
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ExpTest claims that the learning rate for a neural network can be found and annealed automatically by watching the training loss curve. Building on the linearized (neural tangent kernel) view of training, the paper argues that a converging loss curve looks like a single exponential decay, so a statistical test can decide whether the current learning rate is workable. The proposed algorithm starts at a theoretically derived upper bound, fits linear and exponential models to the loss in a window, applies an F-test, and lowers the learning rate and restarts whenever the exponential fit loses. The authors report that this removes the need for initial learning rate selection and scheduling, adds only light overhead, and matches or beats well-tuned SGD, Adam, RMSprop, and Adadelta on MNIST, California Housing, and CIFAR-10.

What carries the argument

The machinery is an exponential-decay hypothesis test operating on the live loss curve. After computing an upper bound $\eta_{\max}$ from the largest eigenvalue of the input covariance (with a special bound for cross-entropy loss), the algorithm sets a window $w = \lfloor 2\sqrt{2L_0}/(\eta e) + 1/2 \rfloor$ from the curvature of a decaying exponential, collects losses over that window, and fits both a linear and an exponential model by least squares. An F-test on the residual sums of squares decides whether the exponential model is significantly better; a one-tailed t-test on the linear slope later detects plateaus. These tests are the decision rule that replaces manual learning-rate selection and decay scheduling.

What would settle it

Train a linear network on synthetic data with a known input-covariance spectrum chosen so the true loss is a sum of two exponentials whose decay rates differ by orders of magnitude, using a learning rate that is known to converge; if ExpTest rejects that rate at $\alpha = 0.05$ and restarts with a smaller one, ending at a worse final loss than a fixed-rate run, the single-exponential model is not capturing the signal.

Watch

Extended reading notes

Core claim

The central claim is that exponential decay of the loss curve is a reliable, real-time signature of convergence, and that this signature can automate both learning-rate search and annealing. For a linear layer trained by gradient descent, the paper derives that each output is a linear combination of decaying exponentials plus a constant, so the MSE loss is approximately a single exponential decay; a second-order neural tangent kernel argument extends the form to MSE and cross-entropy losses for nonlinear networks. ExpTest operationalizes this by computing an initial upper bound $\eta_{\max}$ from the spectrum of the sample covariance, sizing a fitting window from the maximum-curvature time of a single exponential, and then accepting the current rate only when an exponential regression explains the windowed loss significantly better than a linear regression (F-test). When the test fails, the model is reinitialized at a reduced rate $\beta\eta$, and after the first acceptance a t-test on the linear slope handles plateau detection. The paper's stated conclusion is that the method works without any user-selected initial learning rate or schedule and performs comparably to or better than tuned baselines.

Load-bearing premise

The load-bearing premise is that the loss curve inside the fitting window is well described by a single exponential decay; if the true loss falls in multiple stages or the linearized network picture fails, the hypothesis test can reject a workable learning rate and force an unnecessary restart.

Editorial extensions

If this is right

  • Users no longer need to choose an initial global learning rate or a decay schedule; ExpTest sets and anneals the rate automatically, and its two hyperparameters (significance level $\alpha$ and decay factor $\beta$) have standard values (0.05 and 0.33) with moderate sensitivity in the reported experiments.
  • On the three tested tasks, ExpTest matches the best tuned baseline: approximately 92.3 percent accuracy on MNIST logistic regression, approximately 81.5 percent on CIFAR-10 with a VGG-16 and momentum, and test MSE 0.2846 on California Housing.
  • Because ExpTest is SGD with learning-rate decay, it inherits SGD's convergence guarantees and costs only two small regressions plus a statistical test at each window boundary.
  • The method composes with momentum and can serve as an automatic front-end for other optimizers such as Adam, removing their initial-rate requirement.
  • The theoretically derived upper bound is generally respected in practice, though deep networks can converge above it because of implicit regularization, a limitation the paper flags for future work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to replace the single-exponential window with piecewise-exponential segmentation, which would let ExpTest handle loss curves with multiple distinct decay regimes instead of restarting.
  • The curvature-based window formula suggests the optimal observation horizon is tied to the NTK spectrum; one could directly measure whether the loss curve's maximum-curvature time matches the derived expression for different architectures and batch sizes.
  • ExpTest could be used as a warm-start probe: run it for a few windows to find a good rate, then switch to a fixed schedule or adaptive optimizer, since the paper already demonstrates compatibility with momentum.
  • A stress test worth running is at very small batch sizes, where gradient noise may flatten or roughen the loss curve; the paper's window correction factor $c_w$ is a heuristic and may need recalibration for extreme mini-batch settings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. The paper proposes ExpTest, an SGD variant that aims to remove the need to preselect an initial learning rate or a learning-rate schedule. It uses a linearized-network/NTK argument to motivate that a convergent training loss decays approximately exponentially, computes an upper-bound learning rate from the input covariance, and then runs an F-test comparing an exponential fit with a linear fit on a window of the early loss curve. If the exponential fit is not significantly better, the model is reinitialized at a reduced learning rate; once exponential decay is detected, later plateaus trigger further reductions based on a t-test of the linear slope. Experiments cover MNIST logistic regression, California Housing fully connected regression, and CIFAR-10 with a modified VGG-16, comparing against SGD, momentum SGD, Adam, RMSprop, and Adadelta, with ablations for alpha, beta, and batch size. The paper claims minimal overhead, robustness to hyperparameter choice, and state-of-the-art performance without initial learning rate selection or scheduling.

Significance. If the core claim holds, ExpTest would be a practically useful contribution: it turns a manual learning-rate search into an automated early-loss-curve test while remaining a simple wrapper around SGD. The manuscript's strengths include publicly released code and data, explicit robustness checks for alpha, beta, and mini-batch size, and a clear mathematical motivation from linearized training dynamics. However, the central statistical decision mechanism is not currently validated: the F-test is used outside its assumptions, and the test can accept exponentially growing losses. The empirical results are suggestive but do not yet establish the advertised 'automatic initial learning rate search' claim.

major comments (6)
  1. [Algorithm 1, Step 'Fit and Evaluate' (Section III)] The F-test does not constrain the sign of B in the exponential fit f1 = A e^{-Bt} + C. When the learning rate is too high and the loss diverges, an exponential-growth model (B < 0) will often fit the window better than a line; the test then returns p < alpha, disables exponential testing, and training continues at a diverging learning rate instead of restarting. The subsequent t-test on the linear slope may eventually detect the rise, but only after wasted iterations. Please constrain B >= 0, or explicitly test whether the fitted exponential is decaying, and show on synthetic divergent loss curves that the first-window decision rejects them.
  2. [Algorithm 1, Step 'Fit and Evaluate' (Section III)] The F-test is statistically uncalibrated. The linear model f2 = Dt + E is not a special case of the exponential model (the B -> 0 limit is constant, not linear), so the models are non-nested and the F-distribution used for the test does not apply. In addition, successive mini-batch losses are serially correlated and heteroscedastic, so the nominal alpha = 0.05 is not the actual false-positive rate. Because this test is the sole decision rule for accepting the initial learning rate, the paper needs a proper null model, a calibration study against synthetic exponential, linear, and divergent loss curves, and a precise statement of the F-statistic and its degrees of freedom.
  3. [Section III, Eq. (52); Section II.A, Eqs. (23)-(31); Appendix D] The derivation of the window size assumes that the early loss is well approximated by a single exponential with amplitude bounded by L0. The approximation of a sum of exponentials by one exponential is uncontrolled and can be poor when decay rates are disparate, and the window formula t_max = sqrt(2 C_exp)/e is derived for an idealized single-exponential curve. For finite-width networks with multiple decay modes, or when NTK linearization is inaccurate, the maximum-curvature time may fall outside the computed window, so the first F-test may miss a convergent learning rate and trigger a wasteful restart. The paper should provide a bound relating the true early loss to the single-exponential model, or empirically validate the window choice on the three tasks.
  4. [Algorithm 1, Step 5 (t-test branch)] The t-test logic in the pseudocode is inconsistent with the prose. The text says the one-tailed t-test checks whether the slope is significantly less than zero and decreases the learning rate if not, but the pseudocode 'One-tailed t-test if A > 0: t(A, 0); if p < alpha then Update start time; else Reinitialize' would continue training when the slope is significantly positive and restart when it is not, which is the opposite of the intended plateau detection. Please fix the pseudocode and state the null and alternative hypotheses explicitly.
  5. [Appendix E and Section IV.B] The architecture for the California Housing experiment is selected on the test set: Appendix E reports that several width-depth models were evaluated on the test set and the model with (width, depth) = (2, 32) was chosen because it 'consistently displays the lowest test loss.' This is a form of test-set leakage that makes the reported test MSE for ExpTest and the baselines optimistically biased. The architecture should be selected using the validation set only, or with nested cross-validation, before reporting test performance.
  6. [Appendix C] The derivation of the CE-loss learning-rate bound uses a second-order approximation and replaces the Hessian by its maximum eigenvalue (lambda_max,H). The paper states lambda_max,H > 1 and therefore 2/(lambda_max,H lambda_max,Sigma) < 2/lambda_max,Sigma, but it does not show that this bound applies to the true CE gradient-flow dynamics, nor that the bound is usefully tight. Because eta_max is the algorithm's starting point and also enters the window formula, this derivation should be made rigorous or replaced by an empirically validated bound with a sensitivity analysis.
minor comments (6)
  1. [Introduction] The word 'comonly' should be 'commonly' in the first paragraph.
  2. [Eq. (52) and Algorithm 1] The nearest-integer rounding convention should be defined explicitly; the expression floor(2 sqrt(2 L0) / (eta e) + 1/2) is not standard notation for rounding to nearest and should be clarified.
  3. [Table 3] The ExpTest row is identical in every learning-rate column; the caption should state that ExpTest was not rerun at each eta but rather chooses its own trajectory, to avoid implying duplicated experiments.
  4. [Appendix D] The symbol lambda_max is used both for the largest eigenvalue of the covariance matrix and for the eigenvalue that maximizes the curvature time; please rename one of them to remove ambiguity.
  5. [Experimental comparison] The experiments do not compare ExpTest against standard learning-rate schedulers such as cosine annealing, step decay, or ReduceLROnPlateau; given the abstract's 'without learning rate scheduling' claim, this comparison should be added or the claim should be softened.
  6. [Eq. (53)] The window correction c_w = S_mag / ||S_vec|| is undefined when the sum of gradient vectors vanishes; please add a safeguard for this edge case.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ExpTest's exponential-decay model is derived from linearized dynamics, and its reported accuracies are external benchmarks rather than fitted outputs.

full rationale

The paper's derivation chain starts from gradient descent on a linear layer (Equations 5-11), obtains an exponential-decay loss form under convergence (Equations 17-23), extends the result to SGD in expectation (Equation 38), and then to nonlinear networks via the neural tangent kernel (Equations 45-48, Appendices A and B). These steps rely on standard external results (Jacot et al., Lee et al., and classic linear convergence analyses), not on the paper's own prior work, and they do not assume the paper's conclusion. The F-test in Algorithm 1 compares a least-squares exponential fit to a least-squares linear fit on the observed loss curve; the test outcome is used to decide whether to decay the learning rate or restart, but it is not reported as a scientific prediction and no fitted parameter is renamed as a result. The initial learning-rate upper bound (Equation 16 and Appendix C) is computed from the input covariance matrix or from a convexity bound on the loss, not fitted to final accuracies. The window-size formula (Equation 52 and Appendix D) is an analytic curvature maximization for an exponential curve; setting Cexp equal to the initial loss L0 is an explicit upper-bound heuristic rather than a parameter fitted to the method's outputs. The experimental results (Tables 1-4) are external benchmarks on MNIST, California Housing, and CIFAR-10, so the claimed state-of-the-art performance is not forced by construction. The Discussion's stated limitation about the learning-rate upper bound and batch normalization is a recognized weakness, not a circular step. Statistical concerns about the non-nested F-test and uncalibrated null distribution are correctness risks, not circularity. No self-citations appear in the paper. Accordingly, the derivation is self-contained and no circular step is found.

Assumptions & free parameters 3 free parameters · 7 assumptions · 0 invented entities

The central claim rests on a chain of modeling assumptions: linearized network dynamics, single-exponential loss decay, a curvature-derived window, and a heuristic batch correction. The only hand-chosen values are alpha, beta, and the C_exp = L0 bound used in the window size. No new physical entities or latent variables are introduced.

free parameters (3)
  • alpha = 0.05
    Significance level for the F-test and t-test. Chosen a priori as a standard statistical value; robustness across 0.01, 0.05, 0.1 is shown in Section IV.A.3.
  • beta = 0.33
    Learning rate reduction factor. Chosen from common logarithmic search values (0.1, 0.33) cited in [3]; robustness across 0.05, 0.1, 0.33 is shown in Section IV.A.3.
  • C_exp (window amplitude) = L0 (initial loss)
    Set as an upper bound on the exponential amplitude in the window size formula (Equation 52 and Appendix D). This is a heuristic modeling choice, not a quantity fitted to test performance.
assumptions (7)
  • standard math span{X} = R^n ensures a unique minimizer and invertible sample covariance XX^T
    Invoked in Section II.A to derive the closed-form T[k] and the learning rate bounds.
  • domain assumption Input data are normalized to zero mean and unit variance
    Used in Section II.A to bound the eigenvalues of the sample covariance matrix and simplify the analysis.
  • domain assumption Finite-width nonlinear networks are well approximated by their linearization (NTK) early in training
    Invoked in Section II.C to extend the exponential decay and the upper bound to deep networks; based on cited prior work [10]-[13].
  • ad hoc to paper The loss curve under convergence is well approximated by a single exponential decay
    Section II.A, Equations 23-31: the sum of exponentials is approximated by one exponential and the error can be large when decay rates differ. This is the core signal the algorithm tests.
  • ad hoc to paper For cross-entropy loss, a second-order approximation gives linear gradient-flow dynamics, and 2/lambda_max,Sigma is a usable initial learning rate bound
    Appendix C shows the true CE bound is stricter (since lambda_max,H > 1), so the chosen eta_max can exceed the guaranteed bound; the algorithm relies on the F-test to retry at lower rates.
  • ad hoc to paper The window size formula derived from maximum curvature of a single exponential with amplitude L0 is appropriate for detecting exponential decay
    Section III and Appendix D, Equation 52: the derivation assumes a specific maximum curvature timing and sets C_exp = L0 as an upper bound; this is a heuristic for choosing the regression window.
  • ad hoc to paper Batch-size correction c_w = sum||g||/||sum g|| approximates gradient noise and should multiply the window
    Equation 53 in Section III: proposed without derivation; it affects the timing of the first F-test for mini-batch training.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ExpTest: Automating Learning Rate Searching and Tuning with Insights from Linearized Neural Networks." pith.science (2026). https://pith.science/paper/MV3W4QCB

@misc{pith2026241116975,
  author       = {Pith},
  title        = {Pith review of: ExpTest: Automating Learning Rate Searching and Tuning with Insights from Linearized Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MV3W4QCB}},
  note         = {Machine review of arXiv:2411.16975}
}
read the original abstract

Hyperparameter tuning remains a significant challenge for the training of deep neural networks (DNNs), requiring manual and/or time-intensive grid searches, increasing resource costs and presenting a barrier to the democratization of machine learning. The global initial learning rate for DNN training is particularly important. Several techniques have been proposed for automated learning rate tuning during training; however, they still require manual searching for the global initial learning rate. Though methods exist that do not require this initial selection, they suffer from poor performance. Here, we present ExpTest, a sophisticated method for initial learning rate searching and subsequent learning rate tuning for the training of DNNs. ExpTest draws on insights from linearized neural networks and the form of the loss curve, which we treat as a real-time signal upon which we perform hypothesis testing. We mathematically justify ExpTest and provide empirical support. ExpTest requires minimal overhead, is robust to hyperparameter choice, and achieves state-of-the-art performance on a variety of tasks and architectures, without initial learning rate selection or learning rate scheduling.

Figures

Figures reproduced from arXiv: 2411.16975 by the authors.

Figure 2
Figure 2. Training loss curves on MNIST logistic regression for [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 1
Figure 1. Training loss curves for logistic regression on MNIST, [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 4
Figure 4. Training loss curves for modified VGG-16 on CIFAR [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Test MSE-Losses for model width-depth combinations [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Validation loss curves for regression with fully con [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 19 canonical work pages

  1. [1]

    Has artificial intelligence become alchemy?

    M. Hutson, “Has artificial intelligence become alchemy?” Science, vol. 360, no. 6388, pp. 478–478, 2018. [Online]. Available: https://www.science.org/doi/abs/10.1126/science.360.6388.478

    work page doi:10.1126/science.360.6388.478 2018

  2. [2]

    Hyper-parameter optimization: A review of algorithms and applications,

    T. Yu and H. Zhu, “Hyper-parameter optimization: A review of algorithms and applications,” 2020. [Online]. Available: https: //arxiv.org/abs/2003.05689

    arXiv 2020

  3. [3]

    Practical recommendations for gradient-based training of deep architectures,

    Y . Bengio, “Practical recommendations for gradient-based training of deep architectures,” 2012. [Online]. Available: https://arxiv.org/abs/ 1206.5533

    arXiv 2012

  4. [4]

    Some methods of speeding up the convergence of iteration methods,

    B. Polyak, “Some methods of speeding up the convergence of iteration methods,” USSR Computational Mathematics and Mathematical Physics, vol. 4, no. 5, pp. 1–17, 1964. [Online]. Available: https://www.sciencedirect.com/science/article/pii/0041555364901375

    arXiv 1964

  5. [5]

    Increased rates of convergence through learning rate adaptation,

    R. A. Jacobs, “Increased rates of convergence through learning rate adaptation,” Neural Networks , vol. 1, no. 4, pp. 295–307, 1988. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ 0893608088900032

    work page 1988

  6. [6]

    Cyclical learning rates for training neural networks,

    L. N. Smith, “Cyclical learning rates for training neural networks,”

    work page

  7. [7]

    Adam: A method for stochastic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”

  8. [8]

    A disciplined approach to neural network hyper- parameters: Part 1 – learning rate, batch size, momentum, and weight decay,

    L. N. Smith, “A disciplined approach to neural network hyper- parameters: Part 1 – learning rate, batch size, momentum, and weight decay,” 2018. [Online]. Available: https://arxiv.org/abs/1803.09820

    arXiv 2018

Show all 39 references
  1. [9]

    Available: https://arxiv.org/abs/1412.6980

    [Online]. Available: https://arxiv.org/abs/1412.6980

    arXiv

  2. [10]

    Six lectures on linearized neural networks,

    T. Misiakiewicz and A. Montanari, “Six lectures on linearized neural networks,” 2023. [Online]. Available: https://arxiv.org/abs/2308.13431

    2023 arXiv

  3. [11]

    The shape of learning curves: A review,

    T. Viering and M. Loog, “The shape of learning curves: A review,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 45, no. 6, pp. 7799–7819, 2023

    2023

  4. [12]

    What can linearized neural networks actually say about generalization?

    G. Ortiz-Jimenez, S.-M. Moosavi-Dezfooli, and P. Frossard, “What can linearized neural networks actually say about generalization?” (a) η = 0.0001ηmax (b) η = 0.001ηmax (c) η = 0.01ηmax (d) η = 0.1ηmax (e) η = ηmax Fig. 6: Validation loss curves for regression with fully con- ...

    2021

  5. [13]

    Neural tangent kernel: Convergence and generalization in neural networks,

    A. Jacot, F. Gabriel, and C. Hongler, “Neural tangent kernel: Convergence and generalization in neural networks,” in Advances in Neural Information Processing Systems , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., vol. 31. Curran Ass...

    2018

  6. [14]

    Incremental pid controller-based learning rate scheduler for stochastic gradient descent,

    Z. Wang and J. Zhang, “Incremental pid controller-based learning rate scheduler for stochastic gradient descent,” IEEE Transactions on Neural Networks and Learning Systems , vol. 35, no. 5, pp. 7060–7071, 2024

    2024

  7. [15]

    Wide neural networks of any depth evolve as linear models under gradient descent,

    J. Lee, L. Xiao, S. Schoenholz, Y . Bahri, R. Novak, J. Sohl- Dickstein, and J. Pennington, “Wide neural networks of any depth evolve as linear models under gradient descent,” in Advances in Neural Information Processing Systems , H. Wallach, H. Larochelle, A. Beygelzimer, F. ...

    2019

  8. [16]

    Distribution of the largest eigenvalue for real wishart and gaussian random matrices and a simple approximation for the tracy–widom distribution,

    M. Chiani, “Distribution of the largest eigenvalue for real wishart and gaussian random matrices and a simple approximation for the tracy–widom distribution,” Journal of Multivariate Analysis , vol. 129, pp. 69–81, 2014. [Online]. Available: https://www.sciencedirect.com/ scie...

    2014

  9. [17]

    M. T. Hagan, H. B. Demuth, and M. Beale, Neural network design . USA: PWS Publishing Co., 1997

    1997

  10. [18]

    A convergence analysis of gradient descent for deep linear neural networks,

    S. Arora, N. Cohen, N. Golowich, and W. Hu, “A convergence analysis of gradient descent for deep linear neural networks,” 2019. [Online]. Available: https://arxiv.org/abs/1810.02281

    2019 arXiv

  11. [19]

    Maximal initial learning rates in deep relu networks,

    G. Iyer, B. Hanin, and D. Rolnick, “Maximal initial learning rates in deep relu networks,” inProceedings of the 40th International Conference on Machine Learning , ser. ICML’23. JMLR.org, 2023

    2023

  12. [20]

    Adadelta: An adaptive learning rate method,

    M. D. Zeiler, “Adadelta: An adaptive learning rate method,” 2012. [Online]. Available: https://arxiv.org/abs/1212.5701

    2012 arXiv

  13. [21]

    Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude,

    T. Tieleman and G. Hinton, “Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude,” COURSERA: Neural networks for machine learning , vol. 4, no. 2, pp. 26–31, 2012

    2012

  14. [22]

    Array programming with NumPy,

    C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. F. del Río, M. Wiebe, P. Peterson, P. Gérard-Marchant, K. Sheppard, T....

    2020 doi

  15. [23]

    Van Rossum and F

    G. Van Rossum and F. L. Drake, Python 3 Reference Manual . Scotts Valley, CA: CreateSpace, 2009

    2009

  16. [24]

    Scikit-learn: Machine learning in Python,

    F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V . Dubourg, J. Vander- plas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duch- esnay, “Scikit-learn: Machine learning in Python,” Journal of Machine ...

    2011

  17. [25]

    SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python,

    P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey,˙I. Polat, Y . Feng...

    2020

  18. [26]

    Matplotlib: A 2d graphics environment,

    J. D. Hunter, “Matplotlib: A 2d graphics environment,” Computing in Science & Engineering , vol. 9, no. 3, pp. 90–95, 2007

    2007

  19. [27]

    Paszke, S

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, PyTorch: an imperative style, high- pe...

    2019

  20. [28]

    Sparse spatial autoregressions,

    R. Kelley Pace and R. Barry, “Sparse spatial autoregressions,” Statistics and Probability Letters , vol. 33, no. 3, pp. 291–297, 1997. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ S016771529600140X

    1997

  21. [29]

    The mnist database of handwritten digit images for machine learning research [best of the web],

    L. Deng, “The mnist database of handwritten digit images for machine learning research [best of the web],” IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 141–142, 2012

    2012

  22. [30]

    Learning multiple layers of features from tiny images,

    A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” University of Toronto, Toronto, Ontario, Tech. Rep. 0, 2009. [Online]. Available: https://www.cs.toronto.edu/~kriz/ learning-features-2009-TR.pdf

    2009

  23. [31]

    The marginal value of adaptive gradient methods in machine learning,

    A. C. Wilson, R. Roelofs, M. Stern, N. Srebro, and B. Recht, “The marginal value of adaptive gradient methods in machine learning,” in Proceedings of the 31st International Conference on Neural Information Processing Systems , ser. NIPS’17. Red Hook, NY , USA: Curran Associate...

    2017

  24. [32]

    Implicit regularization in deep learning,

    B. Neyshabur, “Implicit regularization in deep learning,” 2017. [Online]. Available: https://arxiv.org/abs/1709.01953

    2017 arXiv

  25. [33]

    Very deep convolutional networks for large-scale image recognition

    K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition.” in ICLR, Y . Bengio and Y . LeCun, Eds.,

  26. [34]

    Implicit gradient regularization,

    D. Barrett and B. Dherin, “Implicit gradient regularization,” in International Conference on Learning Representations , 2021. [Online]. Available: https://openreview.net/forum?id=3q5IqUrkcF

    2021

  27. [35]

    Dropout: A simple way to prevent neural networks from overfitting,

    N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” Journal of Machine Learning Research , vol. 15, no. 56, pp. 1929–1958, 2014. [Online]. Available: http://jmlr.org/papers/v15/srivas...

    1929

  28. [36]

    Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning,

    C. H. Martin and M. W. Mahoney, “Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning,” 2018. [Online]. Available: https://arxiv.org/abs/1810.01075

    2018 arXiv

  29. [39]

    Batch normalization: Accelerating deep network training by reducing internal covariate shift,

    S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” 2015. [Online]. Available: https://arxiv.org/abs/1502.03167

    2015 arXiv

  30. [2015]

    Available: http://dblp.uni-trier.de/db/conf/iclr/iclr2015

    [Online]. Available: http://dblp.uni-trier.de/db/conf/iclr/iclr2015. html#SimonyanZ14a

  31. [2017]

    Available: https://arxiv.org/abs/1506.01186

    [Online]. Available: https://arxiv.org/abs/1506.01186

    arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.