Pith. sign in

REVIEW 3 major objections 5 minor 28 references

A single backtracking line search measures the local curvature that sets Adam's stability limit, and one fixed learning-rate cap derived from it prevents divergence across architectures.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 08:42 UTC pith:KPCBUFNR

load-bearing objection The Armijo sharpness-probe result is the real, solid contribution; the 'calibration-free' rescue claim is oversold in the abstract, but the paper is worth refereeing. the 3 major comments →

arxiv 2607.03998 v3 pith:KPCBUFNR submitted 2026-07-04 cs.LG

Directional Curvature from Armijo Backtracking: A Low-Cost Sharpness Probe and a Calibration-Free Learning-Rate Safeguard for Adam

Ashmitha R , J\"org Frochte This is my paper
classification cs.LG
keywords Armijo backtrackingdirectional curvatureHessian eigenvalueEdge of Stabilitylearning-rate safeguardAdamline search probesharpness measurement
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish two things. First, that the step size accepted by a standard Armijo backtracking line search is a cheap, Hessian-free measurement of local curvature: it brackets the directional curvature q = gᵀHg/‖g‖² within a factor of two, and empirically its logarithm tracks the top Hessian eigenvalue λ₁ with Pearson correlation −0.91 to −0.95 across three architectures. Second, that this measurement, taken once at initialization, can be used as a learning-rate safeguard for Adam: capping the initial η at κ times the probed step prevents divergence across more than three orders of magnitude of misspecification, at about one percent overhead, and is a no-op when the rate is already safe. The practical payoff is that a user no longer needs an expensive range test or manual sweep to avoid the single divergent first step; a few forward passes before training give the same safety signal. The paper recommends probing along Adam's own update direction with a single fixed safety factor κ=2, which transfers across nine architectures and to AdamW.

Core claim

The central claim is that the inverse of the accepted line-search step, 1/α, inverts the directional curvature q_d = dᵀHd/(−gᵀd) along the probed direction, up to the multiplicative backtracking band set by the backtracking factor β. On a quadratic model the Armijo condition accepts α exactly when α ≤ 2(1−c)/q_d, so backtracking returns a step within [β·2(1−c)/q_d, 2(1−c)/q_d]; hence the line search is a curvature sensor, not a tuner. Because the gradient direction carries a roughly constant 0.63 fraction of the top eigenvalue, log α tracks log λ₁ strongly, making the probe an online Edge-of-Stability reading at the cost of about five forward passes. Used once at initialization along Adam's

What carries the argument

The central object is the accepted step α of an Armijo backtracking line search: starting from 1, halve the candidate step until the sufficient-decrease condition L(θ+αd) ≤ L(θ) + cαgᵀd holds. On a quadratic model this accepts α exactly when α ≤ 2(1−c)/q_d, so 1/α inverts the directional curvature q_d = dᵀHd/(−gᵀd) up to the multiplicative band [1, 1/β]. The second ingredient is the empirical alignment q ≈ 0.63λ₁, inherited from Edge-of-Stability dynamics, which converts the directional reading into a top-eigenvalue proxy. The direction-matched probe runs the same search along Adam's first-step direction −g/(|g|+ε), making the conversion factor κ transferable across architectures.

Load-bearing premise

The rescue claim rests on the assumption that the ratio between the largest stable Adam learning rate and the Armijo-accepted step at initialisation is constant enough across architectures for one fixed safety factor κ=2 to work without per-architecture calibration.

What would settle it

On ten unseen architectures spanning MLPs, Transformers, and normalisation-free CNNs, measure the largest stable Adam learning rate and the initial Armijo-accepted step ᾱ. If the ratio η_stable/ᾱ varies by more than the margin allowed by κ=2 (some architectures needing η below 2ᾱ or tolerating η far above 2ᾱ), the fixed cap either fails to engage or lands in a stable-but-unproductive band, falsifying the calibration-free claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • On any architecture where the Armijo constraint binds, the accepted step gives an online estimate of local curvature at about five forward passes per reading, with no Hessian-vector products needed.
  • One probe at initialization converts a run that would diverge in the first step into a run within a few points of tuned accuracy, across η∈[10⁻³, 3.0] on CIFAR-10, Fashion-MNIST and Imagenette, and it is a no-op when the learning rate is already safe.
  • Probing along Adam's preconditioned direction with the single fixed κ=2 avoids divergence on nine architectures and across the full learning-rate grids of four benchmarks, and transfers unchanged to AdamW.
  • Periodic in-training probing adds no robust benefit; the one-shot init probe is the component that generalises.
  • Gradient-norm clipping does not rescue Adam from a too-large initial rate, because the 1/√v amplification happens after clipping.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The AG News result suggests that on architectures whose stable learning-rate range sits far below the initial Armijo step, the fixed cap degrades from a full rescue to a pure divergence safeguard; a practical deployment might combine the fixed cap with a one-time sweep to detect such regimes.
  • The measurement claim implies the same probe could be attached to other adaptive optimisers whose first-step direction differs from Adam's; the paper leaves such optimisers untested, but the direction-matched construction is optimiser-agnostic in principle.
  • Because the probe reading saturates at α=1 on flat regions, its use as an online diagnostic is range-limited; a refined probe could use a finer backtracking factor near saturation to extend the readable curvature range.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper makes two related claims. First, a measurement claim: a single Armijo backtracking line search, run on one mini-batch, returns a step size α whose reciprocal brackets the directional curvature q_d = d^T H d / (-g^T d) along the probed direction within a multiplicative band set by the backtracking factor; empirically, log α tracks log λ_1 with Pearson correlation −0.91 to −0.95 across CIFAR-10/ResNet-18, Fashion-MNIST/CNN, and Imagenette/ResNet-18. Second, a rescue claim: using this probe once at initialization to cap Adam's learning rate at κ·ᾱ makes Adam robust to initial learning rates spanning 10^{-3} to 3.0, with a raw-gradient variant requiring per-architecture κ and a direction-matched variant, probing along Adam's preconditioned direction d ≈ −sign(g), claimed to need only a single fixed κ = 2 across nine architectures and four benchmarks, including transfer to AdamW. The paper is explicit that the fix is a safeguard, not a faster optimizer, and that the direction-matched cap on AG News is 'a divergence safeguard only'.

Significance. If the measurement claim stands, it is a genuinely useful and low-cost diagnostic: the derivation in Section 5 is elementary and correct, the empirical correlations are reported with seed statistics and p-values, and the honest treatment of limitations (including the AG News counterexample) is a strength. The practical rescue claim is more fragile. The fixed κ = 2 is chosen from a sweep over the same architectures on which it is then validated, and the paper's own Table 7 shows that on AG News the fixed cap prevents divergence but leaves training at chance level. Thus the 'calibration-free safeguard' is at best a divergence-prevention mechanism on that benchmark, not a learning-preserving rescue. The measurement contribution is publishable; the rescue claim needs either sharp qualification or additional validation before the abstract-level claim is supportable.

major comments (3)
  1. [Section 5, Table 7] Table 7 (final block) is direct counterevidence to the abstract's 'calibration-free safeguard' claim. At fixed κ=2, the direction-matched probe gives 0.250 (chance) for every η ≥ 10^{-2} on AG News, and the text concedes 'the fixed cap is a divergence safeguard only'. Thus the method does not preserve learning on one of the four benchmarks; 'robust to a too-large initial learning rate across ... all four benchmarks' is too strong. Either replace the central claim with a divergence-safety claim on this architecture or provide a mechanism that also keeps the run productive.
  2. [Section 5, Table 10] The recommended κ=2 is selected from a safety-factor sweep over the same nine architectures on which the transfer claim is then tested (Table 9, Figure 6). This is training-set model selection, not a parameter-free constant. The claim 'single fixed safety factor κ=2 ... across every architecture we test' is therefore not a prediction. Please report a held-out split (e.g., choose κ on four architectures, then evaluate on five unseen ones) or state explicitly that κ is a global hyperparameter tuned on the test architectures.
  3. [Algorithm 1, step 15] The rescue mechanism assumes η_max,productive ≈ κ·ᾱ_dm with a universal κ, but the paper never reports ᾱ_dm or the actual cap values per architecture. The raw-gradient numbers show the conversion varies by roughly 30×: CIFAR ᾱ≈0.06 with stable η≈0.024; Fashion-MNIST ᾱ=1 with stable η≈0.01; AG News ᾱ≈0.031 with stable η≈3×10^{-3}. AG News (Table 7, κ=2) shows the direction-matched cap can land in the stable-but-unproductive band, so the proportionality is not established for productive training. Report ᾱ_dm per architecture and test the proportionality directly.
minor comments (5)
  1. [Section 3, Algorithm 1] The Armijo constant c is not given a numerical value; specify it (e.g., c = 10^{-4}) alongside β = 1/2 and K = 8, since the derived band depends on c through 2(1−c).
  2. [Section 5, Table 2 and mechanism runs] The text states 73% of active probe points fall inside the predicted band in the mechanism runs, while Table 2 reports 86% in band at β=0.5. Clarify whether these summarize different subsets or experiments.
  3. [Section 5, Table 9] Table 9 headlines a single fixed κ=3, while the recommended recipe is κ=2 from Table 10. The final claim would be easier to audit if the main transfer table used the recommended κ=2 and presented κ=3 only as a sensitivity check.
  4. [Figure 1] The fitted slope on CIFAR-10 is 1.46, while Fashion-MNIST and Imagenette are near 1.0. The text calls this a 'unit-slope law with a constant offset'; explain the residual slope discrepancy or soften the wording.
  5. [Abstract and Title] The term 'calibration-free' is used for a recipe whose κ=2 is globally tuned on the test architectures. Consider replacing it with 'per-architecture calibration-free' or 'fixed safety factor' to avoid the impression that no hyperparameter selection is involved.

Circularity Check

1 steps flagged

The 'calibration-free' direction-matched safeguard uses a κ=2 selected by a sweep over exactly the nine architectures on which its zero-divergence success is then reported; the measurement claim (Armijo band, α–λ1 correlation) is self-contained.

specific steps
  1. fitted input called prediction [Section 5 (Tables 9–10 and surrounding text) and Abstract]
    "Both failures occur on a plain MLP at the100×-too-large rate, and a sweep of the safety factor (Table 10) removes them: κ = 2 and even κ = 1 give zero divergences in all54runs at no accuracy cost, so we recommend κ = 2 as the less intrusive of the two divergence-free settings. ... Abstract: 'a single fixed safety factor κ = 2 avoids divergence on all nine architectures we test and across the full learning-rate grids of all four benchmarks'"

    The 'calibration-free' headline is the output of the same sweep that fixes the parameter: Table 10 evaluates κ ∈ {1,2,3} over the same nine architectures × two rates, and κ=2 is chosen because it gives zero divergences in all 54 runs; the Abstract then reports that same zero-divergence property as an empirical finding for κ=2 on those nine architectures. The prediction is statistically forced by the fit — validation is performed on the tuning set. The only κ with a nominal holdout description (κ=3, 'chosen on four architectures and held out on five unseen ones') actually failed on the held-out MLP and is abandoned; κ=2 receives no independent held-out validation. The measurement claim (Section 5 band derivation and the α–λ1 correlation) is independent and unaffected.

full rationale

The band derivation (Section 5: an Armijo-accepted backtracking step satisfies βα* < α ≤ α*, hence 1/α brackets q_d within the 1/β band) is a self-contained mathematical consequence of the sufficient-decrease condition; testing it against q measured on the same batch is a sanity check, not circularity. The α–λ1 correlation is an external empirical measurement against Lanczos eigenvalues, parameter-free and falsifiable, so it does not depend on κ. The circularity is confined to the rescue claim's calibration-free variant: the recommended fixed κ=2 is selected from Table 10's sweep over exactly the nine architectures on which the paper then reports zero divergences, so the headline 'κ=2 avoids divergence on all nine architectures' is the fit objective restated as a result. The paper is transparent about the raw-gradient probe needing calibration, and it discloses the AG News plateau, so there is independent content in the full-grid and AdamW checks. Nevertheless, the central 'no calibration' claim reduces, in its nine-architecture evidence, to parameter selection on the evaluation set, giving partial circularity (score 6 rather than 8; the measurement claim keeps the paper from collapsing entirely).

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central measurement claim rests on standard Armijo theory plus an empirical alignment observation; the rescue claim rests on two fitted safety factors (κ per architecture for the raw-gradient probe, and a κ=2 constant selected on the nine evaluation architectures for the direction-matched probe). No new physical or mathematical entities are introduced.

free parameters (3)
  • raw-gradient safety factor κ = 0.25 (CIFAR-10), 0.005 (Fashion-MNIST), 0.05 (AG News)
    Chosen as roughly half the empirical divergence boundary of vanilla Adam per architecture (Appendix C, Table 13); varies by over an order of magnitude across testbeds.
  • direction-matched safety factor κ = 2
    Selected from a sweep over {1,2,3} on the same nine architectures (Table 10); advertised as calibration-free but the constant is fitted to the evaluation set.
  • q/λ1 alignment ratio = ≈0.63
    Median directional-curvature-to-top-eigenvalue ratio measured on CIFAR-10 and Fashion-MNIST; used in Section 5 to explain why 1/α tracks λ1 with a constant offset.
axioms (4)
  • domain assumption Loss is locally quadratic around the probe point so the second-order expansion is exact
    Used in Section 5 to derive the bracketing band; on mini-batch losses the paper reports 73% of probe points fall inside the predicted band, so the assumption is approximate.
  • domain assumption The largest stable Adam learning rate is proportional to the Armijo-accepted step at initialization
    Algorithm 1 step 15 caps η at κᾱ; the measured ratio is ≈0.6 on CIFAR-10 but differs by orders of magnitude on Fashion-MNIST and AG News, requiring per-architecture recalibration of κ.
  • domain assumption Gradient alignment with the top Hessian eigenvector is roughly constant (q ≈ 0.63 λ1)
    Empirical observation on two architectures used to link the probe reading to λ1; the paper notes the link is weaker on a plain MLP (r = −0.775) and not universal.
  • domain assumption Adam's first update direction is well approximated by −sign(g)
    Used for the direction-matched probe: with bias correction at t=1, m=v=g and d ≈ −g/(|g|+ε) ≈ −sign(g).

pith-pipeline@v1.3.0-alltime-deepseek · 23604 in / 17911 out tokens · 187044 ms · 2026-08-02T08:42:46.878414+00:00 · methodology

0 comments
read the original abstract

The local sharpness of the loss, the top Hessian eigenvalue $\lambda_1$, determines the largest stable gradient step, but measuring it normally requires Lanczos or Hessian-vector iterations. We observe that a single Armijo backtracking line search already carries this information at the cost of a few forward passes: the accepted step $\alpha$ brackets the \emph{directional} curvature $q = g^\top H g/\|g\|^2$ within the multiplicative band set by the backtracking factor. Across CIFAR-10, Fashion-MNIST and Imagenette, $\log\alpha$ tracks $\log\lambda_1$ at Pearson $-0.91$ to $-0.95$, giving a low-cost online Edge-of-Stability reading. Used once at initialisation, this measurement yields a learning-rate cap (a safeguard, not a faster optimiser) that makes Adam robust to a too-large initial learning rate across more than three orders of magnitude ($10^{-3}$ to $3.0$), at about one percent overhead, and it is a no-op when the chosen rate is already safe. One probe is enough: periodic in-training probing adds no robust benefit. The raw-gradient probe exposes the mechanism but needs a safety factor calibrated to the architecture by a one-minute divergence sweep. Probing along Adam's own update direction removes this calibration: a single fixed safety factor $\kappa = 2$ avoids divergence on all nine architectures we test and across the full learning-rate grids of all four benchmarks, and the recipe transfers to AdamW unchanged.

Figures

Figures reproduced from arXiv: 2607.03998 by Ashmitha R, J\"org Frochte.

Figure 1
Figure 1. Figure 1: Within each architecture, 1/α rises with λ1 on log–log scales with fitted slopes 1.46 (CIFAR-10), 1.06 (Fashion-MNIST) and 1.02 (Imagenette) and strong correlation (r = −0.91, −0.97 and −0.91 between log α and log λ1; the fits pool the three seeds of each architecture, per-seed statistics in [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The low-cost Armijo probe tracks local sharpness over training. Per architecture (one [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 2
Figure 2. Figure 2: The low-cost Armijo probe tracks local sharpness over training. Per architecture (one [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Test accuracy vs. initial learning rate, mean [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figure 3
Figure 3. Figure 3: Test accuracy vs. initial learning rate, mean [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Training loss at the misspecified ηinit = 0.1 (mean over 5 seeds, shaded min–max). The grey dashed reference is vanilla Adam at its well-tuned η = 10−3 , which trains normally: Adam is not failing at CIFAR-10, only at this 100×-too-large rate. At η = 0.1 vanilla Adam instead diverges at step 2 (loss → ∞, off chart), whereas Adam-InitOnly and the two periodic controllers train fine and approach the tuned re… view at source ↗
Figure 5
Figure 5. Figure 5: Controller-adjusted η at ηinit = 0.1 (mean over 5 seeds, shaded min–max). The init probe caps the requested η = 0.1 to ≈ 0.016 before training (dotted); the watchdog then relaxes η back to the user’s 0.1 ceiling, while the tracker follows the probed step size up to ≈ 0.25 [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The transfer result at a glance: per-architecture test accuracy at the [PITH_FULL_IMAGE:figures/full_fig_p019_6.png] view at source ↗
Figure 6
Figure 6. Figure 6: The transfer result at a glance: per-architecture test accuracy at the [PITH_FULL_IMAGE:figures/full_fig_p020_6.png] view at source ↗

Review history (3 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

28 extracted references · 4 linked inside Pith

  1. [1]

    Why Warmup the Learning Rate? Underlying Mechanisms and Improvements

    Dayal Singh Kalra and Maissam Barkeshli. Why Warmup the Learning Rate? Underlying Mechanisms and Improvements. InAdvances in Neural Information Processing Systems (NeurIPS), 2024.https://doi.org/10.48550/arXiv.2406.09405

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.09405 2024

  2. [2]

    Miller, and Michael Shvartsman

    Dayal Singh Kalra, Jean-Christophe Gagnon-Audet, Andrey Gromov, Ishita Mediratta, Kelvin Niu, Alexander H. Miller, and Michael Shvartsman. A Scalable Measure of Loss Landscape Curvature for Analyzing the Training Dynamics of LLMs.arXiv preprint arXiv:2601.16979, 2026

    arXiv 2026

  3. [3]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations (ICLR), 2015.https://doi.org/10. 48550/arXiv.1412.6980

    Pith/arXiv arXiv doi:10.48550/arxiv.1412.6980 2015

  4. [4]

    Decoupled Weight Decay Regularization

    Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. InInternational Conference on Learning Representations (ICLR), 2019.https://openreview.net/forum?id= Bkg6RiCqY7

    2019

  5. [5]

    Incorporating Nesterov Momentum into Adam

    Timothy Dozat. Incorporating Nesterov Momentum into Adam. InICLR Workshop Track, 2016.https://openreview.net/forum?id=OM0jvwB8jIp57ZJjtNEZ

    2016

  6. [6]

    Reddi, Satyen Kale, and Sanjiv Kumar

    Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the Convergence of Adam and Beyond. InInternational Conference on Learning Representations (ICLR), 2018. https: //openreview.net/forum?id=ryQu7f-RZ

    2018

  7. [7]

    Lecture 6.5—RMSProp: Divide the Gradient by a Running Average of Its Recent Magnitude.Neural Networks for Machine Learning, Coursera, 2012

    Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5—RMSProp: Divide the Gradient by a Running Average of Its Recent Magnitude.Neural Networks for Machine Learning, Coursera, 2012

    2012

  8. [8]

    Leslie N. Smith. Cyclical Learning Rates for Training Neural Networks. InIEEE Winter Conference on Applications of Computer Vision (WACV), pages 464–472, 2017.https://doi. org/10.1109/WACV.2017.58

    doi:10.1109/wacv.2017.58 2017

  9. [9]

    Smith and Nicholay Topin

    Leslie N. Smith and Nicholay Topin. Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates. InArtificial Intelligence and Machine Learning for Multi-Domain Operations Applications (Proceedings of SPIE, volume 11006), article 1100612, 2019.https://doi.org/10.1117/12.2520589

    doi:10.1117/12.2520589 2019

  10. [10]

    SGDR: Stochastic Gradient Descent with Warm Restarts

    Ilya Loshchilov and Frank Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts. InInternational Conference on Learning Representations (ICLR), 2017.https://openreview. net/forum?id=Skq89Scxx

    2017

  11. [11]

    Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour.arXiv preprint arXiv:1706.02677, 2017.https://doi.org/10.48550/ arXiv.1706.02677

    Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Łukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour.arXiv preprint arXiv:1706.02677, 2017.https://doi.org/10.48550/ arXiv.1706.02677

    Pith/arXiv arXiv doi:10.48550/arxiv.1706.02677 2017

  12. [12]

    Minimization of Functions Having Lipschitz Continuous First Partial Derivatives

    Larry Armijo. Minimization of Functions Having Lipschitz Continuous First Partial Derivatives. Pacific Journal of Mathematics, 16(1):1–3, 1966.https://doi.org/10.2140/pjm.1966.16.1 23

    doi:10.2140/pjm.1966.16.1 1966

  13. [13]

    Laradji, Mark Schmidt, Gauthier Gidel, and Simon Lacoste-Julien

    Sharan Vaswani, Aaron Mishkin, Issam H. Laradji, Mark Schmidt, Gauthier Gidel, and Simon Lacoste-Julien. Painless Stochastic Gradient: Interpolation, Line-Search, and Con- vergence Rates. InAdvances in Neural Information Processing Systems (NeurIPS), volume 32, pages 3727–3740, 2019. https://proceedings.neurips.cc/paper_files/paper/2019/ hash/2557911c1bf7...

    arXiv 2019

  14. [14]

    Laradji, and Simon Lacoste-Julien

    Nicolas Loizou, Sharan Vaswani, Issam H. Laradji, and Simon Lacoste-Julien. Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergence. InInternational Conference on Artificial Intelligence and Statistics (AISTATS), volume 130 ofProceedings of Machine Learning Research, pages 1306–1314, 2021.https://proceedings.mlr.press/v130/ loi...

    2021

  15. [15]

    Parabolic Approximation Line Search for DNNs

    Maximus Mutschler and Andreas Zell. Parabolic Approximation Line Search for DNNs. InAdvances in Neural Information Processing Systems (NeurIPS), volume 33, pages 5405–5416, 2020. https://proceedings.neurips.cc/paper_files/paper/2020/ hash/3a30be93eb45566a90f4e95ee72a089a-Abstract.html

    2020

  16. [16]

    Cohen, Simran Kaur, Yuanzhi Li, J

    Jeremy M. Cohen, Simran Kaur, Yuanzhi Li, J. Zico Kolter, and Ameet Talwalkar. Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability. InInternational Conference on Learning Representations (ICLR), 2021.https://openreview.net/forum?id= jh-rTtvkGeM

    2021

  17. [17]

    The Large Learning Rate Phase of Deep Learning: The Catapult Mechanism.arXiv preprint arXiv:2003.02218, 2020.https://doi.org/10.48550/arXiv.2003.02218

    Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The Large Learning Rate Phase of Deep Learning: The Catapult Mechanism.arXiv preprint arXiv:2003.02218, 2020.https://doi.org/10.48550/arXiv.2003.02218

    Pith/arXiv arXiv doi:10.48550/arxiv.2003.02218 2003

  18. [18]

    Alex Damian, Eshaan Nichani, and Jason D. Lee. Self-Stabilization: The Implicit Bias of Gradi- ent Descent at the Edge of Stability. InInternational Conference on Learning Representations (ICLR), 2023.https://openreview.net/forum?id=nhKHA59gXz

    2023

  19. [19]

    Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W. Mahoney. PyHessian: Neural Networks Through the Lens of the Hessian. InIEEE International Conference on Big Data, pages 581–590, 2020.https://doi.org/10.1109/BigData50022.2020.9378171

    arXiv 2020

  20. [20]

    On the Difficulty of Training Recurrent Neural Networks

    Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the Difficulty of Training Recurrent Neural Networks. InInternational Conference on Machine Learning (ICML), volume 28 of Proceedings of Machine Learning Research, pages 1310–1318, 2013.https://proceedings. mlr.press/v28/pascanu13.html

    2013

  21. [21]

    Learning-Rate-Free Learning by D-Adaptation

    Aaron Defazio and Konstantin Mishchenko. Learning-Rate-Free Learning by D-Adaptation. InInternational Conference on Machine Learning (ICML), volume 202 ofProceedings of Machine Learning Research, pages 7449–7479, 2023.https://proceedings.mlr.press/v202/ defazio23a.html

    2023

  22. [22]

    Prodigy: An Expeditiously Adaptive Parameter- Free Learner

    Konstantin Mishchenko and Aaron Defazio. Prodigy: An Expeditiously Adaptive Parameter- Free Learner. InInternational Conference on Machine Learning (ICML), volume 235 of Proceedings of Machine Learning Research, pages 35779–35804, 2024.https://proceedings. mlr.press/v235/mishchenko24a.html

    2024

  23. [23]

    Mechanic: A Learning Rate Tuner

    Ashok Cutkosky, Aaron Defazio, and Harsh Mehta. Mechanic: A Learning Rate Tuner. InAdvances in Neural Information Processing Systems (NeurIPS), volume 36, pages 24 47828–47848, 2023. https://proceedings.neurips.cc/paper_files/paper/2023/hash/ 955499a8e2860ed746717c1374224c43-Abstract-Conference.html

    2023

  24. [24]

    The Road Less Scheduled

    Aaron Defazio, Xingyu Alice Yang, Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, and Ashok Cutkosky. The Road Less Scheduled. InAdvances in Neural Information Processing Systems (NeurIPS), volume 37, pages 9974– 10007, 2024. https://proceedings.neurips.cc/paper_files/paper/2024/hash/ 136b9a13861308c8948cd308ccd02658-Abstract-Conference.html

    2024

  25. [25]

    Edge of Stochastic Stability: Revisiting the Edge of Stability for SGD.arXiv preprint arXiv:2412.20553, 2024.https://doi.org/10.48550/ arXiv.2412.20553

    Alexander Andreyev and Paolo Beneventano. Edge of Stochastic Stability: Revisiting the Edge of Stability for SGD.arXiv preprint arXiv:2412.20553, 2024.https://doi.org/10.48550/ arXiv.2412.20553

    doi:10.48550/arxiv.2412.20553 2024

  26. [26]

    Stepping on the Edge: Curvature Aware Learning Rate Tuners

    Vincent Roulet, Atish Agarwala, Jean-Bastien Grill, Grzegorz Swirszcz, Mathieu Blon- del, and Fabian Pedregosa. Stepping on the Edge: Curvature Aware Learning Rate Tuners. InAdvances in Neural Information Processing Systems (NeurIPS), volume 37, pages 47708–47740, 2024. https://proceedings.neurips.cc/paper_files/paper/2024/ hash/555479a201da27c97aaeed842d...

    2024

  27. [27]

    Imagenette: A Smaller Subset of 10 Easily Classified Classes from ImageNet

    Jeremy Howard. Imagenette: A Smaller Subset of 10 Easily Classified Classes from ImageNet. https://github.com/fastai/imagenette, 2019

    2019

  28. [28]

    Understanding Gradient Descent on the Edge of Stability in Deep Learning

    Sanjeev Arora, Zhiyuan Li, and Abhishek Panigrahi. Understanding Gradient Descent on the Edge of Stability in Deep Learning. InInternational Conference on Machine Learning (ICML), volume 162 ofProceedings of Machine Learning Research, pages 948–1024, 2022. https://proceedings.mlr.press/v162/arora22a.html A The negative result on the Golden-Section presele...

    2022