Pith. sign in

REVIEW 3 major objections 3 minor 3 references

Online Learning-guided Learning Rate Adaptation via Gradient Alignment

T0 review · 3 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proposes GALA, an online-learning rule that adjusts the learning rate up or down from gradient-alignment signals, and proves a data-adaptive convergence rate for normalized SGD with momentum under it.

desk verdict A promising learning-rate adaptation scheme whose advertised convergence theorem relies on an unstated almost-sure per-sample smoothness assumption that fails for ReLU-type losses; the experiments also test a heuristic variant not covered by the analysis. read the letter →

arxiv 2506.08419 v1 pith:WQPLEK4S submitted 2025-06-10 cs.LG math.OCstat.ML

Ruichen Jiang , Ali Kavis , Aryan Mokhtari This is my paper
classification cs.LGmath.OCstat.ML MSC 90C2690C15
keywords learningrateadaptationonlinegradientalignmentnonconvexoptimizationstochasticdescentadaptivestepsizeFollow-the-Regularized-Leadermomentum
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the learning rate can be removed from the hyperparameter tuning loop: instead of fixing a base rate and an external schedule, the optimizer itself can choose the step size online from the geometry of the iterates. The proposed mechanism, GALA, watches the alignment between consecutive stochastic gradients and a local curvature estimate, and feeds that signal into a one-dimensional online learning problem solved by Follow-the-Regularized-Leader. The paper proves that normalized SGD with momentum equipped with this rule converges in smooth nonconvex stochastic optimization at a data-adaptive rate, and shows empirically that SGD and Adam augmented with GALA stay competitive across initial learning rates spanning many orders of magnitude. If correct, this gives a principled replacement for grid search and hand-designed schedules.

What carries the argument

The load-bearing object is the surrogate online loss $\ell_t(\eta)$ built from gradient alignment and local curvature. The alignment term $\langle g'_t(w_t), m_t/\|m_t\|\rangle$ arises from the fundamental theorem of calculus: the decrease in $F$ along the step equals the integral of gradients along the segment, and a stochastic gradient at a random interpolation point is an unbiased estimate of that integral. The quadratic term, weighted by the local Lipschitz estimates $L_t$ and $\tilde L_t$, regularizes the rate and makes the loss strongly convex, so Follow-the-Regularized-Leader attains logarithmic regret; the closed-form $\eta$ update is just the ratio of accumulated alignment to accumulated curvature. This machinery converts step-size selection into a one-dimensional online learning problem whose regret plugs directly into a nonconvex SGD convergence bound.

What would settle it

Compute the ratios $L_t = \|g'_t(w_t)-g'_t(x_t)\|/\|w_t-x_t\|$ and $\tilde L_t$ while running Algorithm 1 on a ReLU network; if any trajectory shows these ratios exceeding every finite $L_{\max}$ or their cumulative average dropping to zero on a long flat plateau, the logarithmic-regret bound in Lemma 4.2 fails for that run, and with it the Theorem 4.1 rate.

Watch

Extended reading notes

Core claim

GALA treats the learning rate itself as the decision variable of a one-dimensional online convex problem. At each step it forms a surrogate loss $\ell_t(\eta) = -\eta\langle g'_t(w_t), m_t/\|m_t\|\rangle + (L_t/2 + 4(1-\alpha)\tilde L_t/(3\alpha))\eta^2$, where $g'_t(w_t)$ is a stochastic gradient evaluated at a random point between $x_t$ and $x_{t+1}$, $m_t$ is an exponential moving average of stochastic gradients, and $L_t,\tilde L_t$ are local Lipschitz estimates built from mini-batch gradient differences. An optimistic Follow-the-Regularized-Leader update on this loss yields the closed-form learning rate $\eta_{t+1} = \mathrm{clip}_{[0,\eta_{\max}]}\big((\sum_{s=0}^t \langle g'_s(w_s), m_s/\|m_s\|\rangle)/(\delta+\sum_{s=0}^t (L_s + 8(1-\alpha)\tilde L_s/(3\alpha)))\big)$. Under the bounded-Lipschitz assumptions, Theorem 4.1 shows the average gradient norm of normalized SGD with momentum using this rule is $O\big(\sigma^{1/2}(L^{\mathrm{avg}}_T(\Delta_F+\mathrm{Reg}_T))^{1/4}/T^{1/4} + \sigma^2/\sqrt{L^{\mathrm{avg}}_T(\Delta_F+\mathrm{Reg}_T)T} + \sqrt{L^{\mathrm{avg}}_T(\Delta_F+\mathrm{Reg}_T)}/\sqrt{T} + (\Delta_F+\mathrm{Reg}_T)/(\eta_{\max}T)\big)$, with $\mathrm{Reg}_T = O(\log T)$; so the schedule is selected by the algorithm rather than fixed in advance.

Load-bearing premise

The load-bearing premise is that the local curvature estimates $L_t$ and $\tilde L_t$ — ratios of mini-batch gradient differences over tiny update steps — stay below a finite ceiling and their running average stays above a positive floor for every iteration; on a deep ReLU network with mini-batch noise neither bound is guaranteed.

Editorial extensions

If this is right

  • With GALA, the user no longer needs to pick a learning-rate schedule: the FTRL rule replaces both base-rate tuning and external schedulers, and the resulting convergence rate matches normalized SGD with momentum up to logarithmic factors.
  • The learning rate can rise when consecutive gradients align and fall when they oppose, unlike AdaGrad-style monotone decreases.
  • Adding GALA to SGD or Adam keeps final training and test accuracy competitive with the best-tuned baselines across initial learning rates from $10^{-8}$ to $1$ on CIFAR-10, CIFAR-100, and Flower102.
  • The convergence guarantee is data-adaptive, stated in terms of the average local Lipschitz estimates $L_t$ and $\tilde L_t$, so it can be much sharper than bounds that use a global Lipschitz constant.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the alignment signal is only inner products of directions already computed along the trajectory, the same FTRL wrapper should transfer to any base optimizer that exposes a search direction, such as sign-based or preconditioned directions, not just SGD and Adam.
  • Editorial inference: the theory covers an unbiased interpolation point and separate mini-batches, while the experiments set $w_t = x_{t+1}$ and reuse one mini-batch for both gradients; if that same-batch heuristic is what drives the empirical robustness, proving convergence for that variant would close the gap between analysis and implementation.
  • Editorial inference: a direct empirical test of the mechanism is to log $\eta_t$ and the alignment term during training; the paper's own learning-rate curves predict the rate should settle into a common band across initializations, and monitoring when the sign of alignment flips could give a schedule-switching or early-stopping signal.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes GALA (Gradient Alignment-based Learning rate Adaptation), a framework that treats the learning rate of an optimizer as the decision variable in a one-dimensional online learning problem. The surrogate loss balances gradient alignment between consecutive gradients against a local Lipschitz estimate, and the resulting FTRL update produces a learning-rate schedule that can both increase and decrease. For a normalized SGD-with-momentum variant, the authors prove a data-adaptive convergence rate in the smooth nonconvex stochastic setting, with the rate expressed in terms of an online regret and data-dependent Lipschitz estimates. Experiments on CIFAR-10, CIFAR-100, and Flowers102 with ResNet-18 show that heuristic GALA-augmented SGD and Adam are robust to the initial learning rate and competitive with existing baselines.

Significance. If the theoretical guarantee held in the stated generality, this would be a useful contribution: it offers a principled, non-monotone, data-adaptive learning-rate schedule that avoids manual schedulers and has a convergence rate depending on local rather than global smoothness. The surrogate-loss construction is elegant, and the algebraic core of the proof—Lemmas 3.1, B.1–B.3 and the regret-to-convergence conversion in Theorem 4.1—is internally coherent under the stated extra hypotheses. The empirical study is also thoughtfully conducted, with multiple datasets, seeds, and baselines, and it convincingly demonstrates the practical robustness of the heuristic variants. However, the paper's central theorem relies on almost-sure uniform bounds on the local Lipschitz estimates that are neither derived from Assumptions 2.1–2.2 nor satisfied by natural smooth stochastic problems, and the experiments implement a different algorithm from the one analyzed. These gaps currently limit the significance of the theoretical claim.

major comments (3)
  1. [§4, Lemma 4.2 and Theorem 4.1] The advertised convergence rate depends on Lemma 4.2, whose proof assumes, with probability one, that max{L_t, \tilde L_t} ≤ L_max and that (1/(t+1))∑_{s=0}^t L_s ≥ M_avg. These bounds are not consequences of Assumptions 2.1–2.2. For the scalar problem f(x;ξ)=max(0,x+ξ) with ξ uniform on [-1,1], F is C^1 with L-Lipschitz gradient and bounded stochastic-gradient variance, so Assumptions 2.1–2.2 hold. Yet whenever the segment [x_t, x_t-η_t] crosses the kink of the realized sample ξ'_t, the realized L_t equals 1/(λ_t η_t), which exceeds any fixed L_max with positive probability; in flat regions the running average of L_t can fall below any fixed M_avg. Thus the conclusion Reg_N^T=O(log T) does not follow from the stated assumptions, and the convergence rate in Theorem 4.1 is not established in the claimed generality.
  2. [§5.2, Eq. (11) and the Adam update] The experiments do not implement the algorithm analyzed in Theorem 4.1. SGD-GALA sets w_t=x_{t+1}, reuses the same mini-batch for both gradients, removes the clipping step, and uses standard SGD rather than the normalized momentum update (8); Adam-GALA further replaces the direction with the Adam preconditioned direction d_t. The same-mini-batch reuse breaks the independence used to obtain E[⟨g'_t(x_t),g_t(x_t)⟩]=E[∥∇F(x_t)∥²] in Lemma 3.1, and Theorem 4.1 covers neither the heuristic SGD variant nor Adam. The empirical robustness results therefore do not validate the theoretical guarantee; the authors should either analyze a variant close to the tested heuristic or report experiments on the normalized-momentum algorithm from Section 4.
  3. [§4, Theorem 4.1 and the paragraph after Lemma 4.2] The statement 'if we choose α = min{ sqrt(L_avg^T(ΔF+Reg_N^T))/(σ√T), 1 }' is not an implementable parameter selection, because Reg_N^T and L_avg^T depend on the trajectory and Reg_N^T itself depends on α through η_max and through the surrogate loss (9). No fixed-point argument or bound using only quantities available before the run is provided. Consequently, the sentence after Lemma 4.2 claiming a rate O(σ^{1/2}T^{-1/4}+T^{-1/2}) combines a nonconstructive choice of α with the regret bound; as written, this is an existence argument rather than a guarantee for a fully specified adaptive algorithm.
minor comments (3)
  1. [Appendix B.2, proof of Lemma B.2] The displayed chain E[∥∇F(x_{s-1})-∇F(x_s)∥] ≤ E[∥∇f(x_s;ξ_s)-∇f(x_{s-1};ξ_s)∥] = \tilde L_{s-1}∥x_s-x_{s-1}∥ omits an expectation on the right-hand side; \tilde L_{s-1} is defined with an independent sample ξ'_{s-1}, while the expression uses ξ_s. The conclusion is correct because both expectations agree, but the notation should be fixed.
  2. [§3, Lemma 3.1 and §4, Lemma 4.2] The almost-sure bound L_t ≤ L_max is introduced as an extra condition inside Lemma 3.1 and again in Lemma 4.2, but it is not listed among the paper's main assumptions and no discussion is given of when it holds; this should be an explicit, named assumption with a statement about its scope.
  3. [§5.2, Remark 3.3] The remark that the FTRL update is 'independent of the initialization' is only true for t ≥ 1, since the first step explicitly uses the initial learning rate η_0; the wording should be adjusted.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: GALA's convergence analysis is self-contained, with data-dependent quantities appearing as derived bounds rather than fitted inputs.

full rationale

The paper's main derivation chain is not circular. Theorem 4.1 converts a regret bound on the surrogate losses into a convergence rate for normalized SGD with momentum, and Lemma 4.2 supplies that regret bound through a standard optimistic FTRL analysis invoking [Ora19, Theorem 7.39], an external result. The local Lipschitz quantities L_t and \tilde L_t appear as measured, data-dependent objects; the final rate explicitly depends on their average L_avg^T, which is a derived quantity of the trajectory, not a parameter fitted to match the claimed convergence. The proof of the key structural step in Lemma B.3 relies on [CM20, Lemma 2], which is an external, machine-independent published result, and the paper's own authors are not the source of that load-bearing lemma. The few self-citations (e.g., [FTCMSW22], [KLC22], [RKWAC24]) are background comparisons in the related-work section and are not used to justify the central theorem. The choice of alpha in Theorem 4.1 is an analysis artifact for optimizing the explicit bound, not a fitted constant used to produce the advertised rate. Concerns raised about the almost-sure uniform bounds on L_t and \tilde L_t failing for ReLU-type stochastic realizations are assumptions-risk or correctness issues, not circularity: the paper states these bounds as hypotheses in Lemma 4.2, and if they fail the bound does not apply, but the argument does not reduce to its conclusion by construction. Likewise, the difference between the theory (independent samples for the two gradients) and the implementation (same mini-batch) is a scope gap, not a circular step. The empirical evaluation compares against external baselines without fitting constants to obtain the claimed convergence result. Overall, no step was found where a predicted result is identical to an input by definition, and no load-bearing self-citation chain appears.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central claim rests on standard smoothness and variance assumptions plus two ad hoc uniform bounds on data-dependent local Lipschitz estimates that are not verified in the experiments. The algorithm has user-set hyperparameters (alpha, delta, eta_max), and the implemented version drops clipping and uses biased gradient pairs, so the theory does not cover the exact tested method.

free parameters (3)
  • alpha (momentum parameter) = 0.9 in experiments; theorem chooses an oracle value
    User-defined in the normalized SGD update (8); the theorem's optimal choice depends on unknown sigma, Delta F, Reg_T and L_avg_T, so in practice it must be set by hand.
  • delta (FTRL regularization constant) = not reported; omitted in the implemented update (11)
    Appears in the denominator of the FTRL update (7); stabilizes the learning-rate formula; in the experiments the update rule (11) drops it, so its role in practice is unclear.
  • eta_max (maximum learning rate) = not reported; clipping omitted in experiments
    Clips the adaptive learning rate in (7); the regret bound in Lemma 4.2 sets eta_max = sqrt(alpha) eta_bar; experiments omit clipping, leaving effective eta_max unbounded.
assumptions (6)
  • domain assumption F has L-Lipschitz gradients (Assumption 2.1)
    Standard smoothness assumption used throughout the proofs.
  • domain assumption Stochastic gradients have bounded variance sigma^2 (Assumption 2.2)
    Used to control noise in Lemmas 3.1, B.2 and 4.2.
  • ad hoc to paper Data-dependent local Lipschitz estimates are uniformly upper bounded: L_t <= L_max a.s.
    Stated in Lemma 3.1 and Lemma 4.2; not guaranteed for neural networks and not verified experimentally.
  • ad hoc to paper Running averages of local Lipschitz estimates are lower bounded: (1/(t+1)) sum_{s=0}^t L_s >= M_avg a.s.
    Stated in Lemma 4.2; needed for the O(log T) regret bound; hard to verify in practice.
  • standard math The average gradient along the update segment is unbiasedly estimated by the stochastic gradient at a random interpolation point (Eq. 4)
    Follows from the fundamental theorem of calculus and unbiasedness of stochastic gradients; used in Lemma 3.1 and B.1.
  • standard math FTRL regret bound for strongly convex losses (cited as [Ora19, Theorem 7.39])
    Invoked in the proof of Lemma 4.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Online Learning-guided Learning Rate Adaptation via Gradient Alignment." pith.science (2026). https://pith.science/paper/WQPLEK4S

@misc{pith2026250608419,
  author       = {Pith},
  title        = {Pith review of: Online Learning-guided Learning Rate Adaptation via Gradient Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WQPLEK4S}},
  note         = {Machine review of arXiv:2506.08419}
}
read the original abstract

The performance of an optimizer on large-scale deep learning models depends critically on fine-tuning the learning rate, often requiring an extensive grid search over base learning rates, schedules, and other hyperparameters. In this paper, we propose a principled framework called GALA (Gradient Alignment-based Learning rate Adaptation), which dynamically adjusts the learning rate by tracking the alignment between consecutive gradients and using a local curvature estimate. Guided by the convergence analysis, we formulate the problem of selecting the learning rate as a one-dimensional online learning problem. When paired with an online learning algorithm such as Follow-the-Regularized-Leader, our method produces a flexible, adaptive learning rate schedule that tends to increase when consecutive gradients are aligned and decrease otherwise. We establish a data-adaptive convergence rate for normalized SGD equipped with GALA in the smooth, nonconvex setting. Empirically, common optimizers such as SGD and Adam, when augmented with GALA, demonstrate robust performance across a wide range of initial learning rates and perform competitively without the need for tuning.

Figures

Figures reproduced from arXiv: 2506.08419 by the authors.

Figure 1
Figure 1. Training loss comparison for standalone SGD and Adam versus [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison of final training loss values obtained from different initial learning rates for the CIFAR-10, CIFAR-100, and Flower102 datasets. We compare the performance of SGD-GALA, ADAM-GALA against SGD, Adam, AdGD, and Mechanic. We initialize each algorithm with learning rates [1, 10−1 , 10−2 , 10−3 , 10−4 , 10−5 , 10−8 ] and execute 3 seeded runs for each. • An internal online learner selects a learning rate st+1;… view at source ↗
Figure 3
Figure 3. Comparison of final training accuracy obtained from different initial learning rates for the CIFAR-10, CIFAR-100, and Flower102 datasets. We compare the performance of SGD-GALA, ADAM-GALA against SGD, Adam, AdGD, and Mechanic. We initialize each algorithm with learning rates [1, 10−1 , 10−2 , 10−3 , 10−4 , 10−5 , 10−8 ] and execute 3 seeded runs for each. 10 7 10 5 10 3 10 1 Initial Learning Rate 20 40 60 80 100 Fin… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison of final test accuracy values obtained from different initial learning rates for the CIFAR-10, CIFAR-100, and Flower102 datasets. We compare the performance of SGD-GALA, ADAM-GALA against SGD, Adam, AdGD, and Mechanic. We initialize each algorithm with learn…
Figure 5
Figure 5. Figure 5: Comparison of learning rate evolution for SGD-GALA, ADAM-GALA and AdGD on the CIFAR-100 dataset, averaged over 3 runs. Learning rate evolution To better understand the convergence behavior of our method, we visualize the learning rate dynamics during optimization in […

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

3 extracted references · 1 canonical work pages

  1. [1]

    Parameter adaptation in stochastic optimization

    [ALAP99] L. B. Almeida, T. Langlois, J. D. Amaral, and A. Plakhov. “Parameter adaptation in stochastic optimization”. In: On-line learning in neural networks. 1999, pp. 111–134 (page 4). [AVCLM22] K. Antonakopoulos, D. Q. Vu, V . Cevher, K. Y. Levy, and P . Mertikopoulos. “Un- derGrad: A universal black-box optimization method with almost dimension-free c...

    arXiv 2023

  2. [32]

    High Probability Bounds for a Class of Nonconvex Algorithms with AdaGrad Stepsize

    2019, pp. 6260–6269 (page 3). [KLC22] A. Kavis, K. Y. Levy, and V . Cevher. “High Probability Bounds for a Class of Nonconvex Algorithms with AdaGrad Stepsize”. In: International Conference on Learning Representations. 2022 (page 3). [KMJ23] A. Khaled, K. Mishchenko, and C. Jin. “DoWG unleashed: An efficient univer- sal parameter-free gradient descent met...

    work page 2023

  3. [2024]

    A High Probability Analysis of Adaptive SGD with Mo- mentum

    arXiv: 2310.10082 [math.OC] (page 4). [LO20] X. Li and F. Orabona. “A High Probability Analysis of Adaptive SGD with Mo- mentum”. In: Workshop on Beyond First Order Methods in ML Systems at ICML’20. 2020 (page 3). [LO19] X. Li and F. Orabona. “On the convergence of stochastic gradient descent with adaptive stepsizes”. In: The 22nd international conference...

    arXiv 2010

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.