pith. sign in

arxiv: 2606.30455 · v1 · pith:QLKE6GRTnew · submitted 2026-06-29 · 💻 cs.LG · math.OC· stat.ML

Curvature-Weighted Gradient Diversity: A Noise Measure for Geometry-Adaptive SGD Schedules

Muhammad Hamza (1) , Ayush Goel (1) ((1) Indian Institute of Technology Kharagpur) This is my paper

Pith reviewed 2026-06-30 07:28 UTC · model grok-4.3

classification 💻 cs.LG math.OCstat.ML
keywords SGDlearning rate schedulegradient noiseHessian estimationcurvature weightingoptimization error floorHutchinson estimatorquadratic objectives
0
0 comments X

The pith

Curvature-Weighted Gradient Diversity lets a modulated cosine schedule cut the asymptotic SGD error floor by up to half on strongly convex quadratics with diagonal Hessians and isotropic noise.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper defines Curvature-Weighted Gradient Diversity as a measure of per-sample gradient diversity scaled by the inverse square root of the Hessian, so that noise in high-curvature directions receives less weight. It proves that feeding this measure into a cosine learning-rate schedule lowers the long-run error floor by a factor of two relative to ordinary cosine annealing when the objective is a strongly convex quadratic whose Hessian is diagonal and whose noise is isotropic. Experiments across condition numbers, batch sizes, and noise structures show the resulting CWGD-Cosine method reaches roughly 20 percent lower final error with negligible added cost on quadratics. The work also fixes a degenerate curvature estimator and notes that Hessian staleness limits direct use outside the quadratic case.

Core claim

For strongly convex quadratic objectives with diagonal Hessians and isotropic noise, a CWGD-modulated cosine learning-rate schedule reduces the asymptotic optimization error floor by up to a factor of two compared with standard cosine annealing, because the curvature weighting supplies a tighter proxy for the effective noise that actually limits convergence.

What carries the argument

Curvature-Weighted Gradient Diversity (CWGD), which computes a geometry-aware noise proxy by weighting per-sample gradient diversity with the inverse square root of the Hessian.

If this is right

  • CWGD-Cosine reaches approximately 20 percent lower final optimization error than standard cosine annealing across tested condition numbers and batch sizes.
  • The Hutchinson diagonal Hessian estimator is exact under the quadratic setting and adds negligible overhead.
  • The same modulated schedule works for a range of noise structures while preserving the factor-of-two error reduction in the stated quadratic case.
  • A degenerate curvature estimator is identified and replaced to keep the weighting stable.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same directional weighting idea could be tested on non-quadratic problems by replacing the exact Hessian with periodic diagonal estimates, provided staleness is controlled.
  • Standard variance-only analyses of SGD noise may systematically overestimate the harm of curvature-aligned noise, suggesting similar corrections for other adaptive methods.
  • If the factor-of-two gain survives modest Hessian approximation error, CWGD-style modulation could be combined with existing second-order preconditioners without doubling their cost.

Load-bearing premise

The objective must be a strongly convex quadratic whose Hessian is exactly diagonal and whose gradient noise is isotropic.

What would settle it

Construct a strongly convex quadratic with diagonal Hessian and isotropic gradient noise, run both the CWGD-modulated cosine schedule and standard cosine annealing to steady state, and check whether the final error of the modulated schedule is half or less that of the baseline.

Figures

Figures reproduced from arXiv: 2606.30455 by Ayush Goel (1) ((1) Indian Institute of Technology Kharagpur), Muhammad Hamza (1).

Figure 1
Figure 1. Figure 1: Convergence trajectories on synthetic quadratics for [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Left: effect of modulation strength α on final suboptimality (κ = 20). Loss decreases monotonically with α; the grey dashed line marks the theoretically optimal α ∗ = 0.95. Right: percentage improvement of CWGD-Cosine over Cosine across batch sizes (κ = 20). The benefit is stable at 20–24% regardless of B. to replace the periodic Hutchinson refresh with an online EMA of per-coordinate gradient second momen… view at source ↗
Figure 3
Figure 3. Figure 3: Theoretical improvement factor vs. observed empirical improvement across condition [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
read the original abstract

The standard convergence analysis of mini-batch stochastic gradient descent (SGD) models gradient noise using a single variance term that treats all parameter directions equally, ignoring the fact that noise in high-curvature directions has less impact because learning rates are already constrained there. We introduce Curvature-Weighted Gradient Diversity (CWGD), a geometry-aware measure that weights per-sample gradient diversity by the inverse square root of the Hessian, providing a tighter proxy for the effective optimization noise. For strongly convex quadratic objectives with diagonal Hessians and isotropic noise, we prove that a CWGD-modulated cosine learning-rate schedule can reduce the asymptotic optimization error floor by up to a factor of two compared with standard cosine annealing. We implement this idea as CWGD-Cosine using a Hutchinson-based diagonal Hessian estimator that is exact for quadratic objectives. Across a range of condition numbers, batch sizes, and noise structures, CWGD-Cosine consistently achieves approximately 20% lower final optimization error than standard cosine annealing while incurring negligible overhead in the quadratic setting. We also identify and correct a degenerate curvature estimator, analyze the robustness of the proposed estimator, and explicitly discuss the limitations of the method, including Hessian staleness in non-convex optimization. These results establish CWGD as a principled geometry-aware measure of optimization noise and motivate future extensions to more general learning problems.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 2 minor

Summary. The manuscript introduces Curvature-Weighted Gradient Diversity (CWGD), a geometry-aware measure of per-sample gradient diversity weighted by the inverse square root of the Hessian. For strongly convex quadratic objectives with diagonal Hessians and isotropic gradient noise, it proves that modulating a cosine learning-rate schedule by CWGD reduces the asymptotic optimization error floor by up to a factor of two relative to standard cosine annealing. The method is realized as CWGD-Cosine via a Hutchinson-based diagonal Hessian estimator (exact under the quadratic assumptions), with experiments across condition numbers, batch sizes, and noise structures reporting approximately 20% lower final error and negligible overhead. The paper also corrects a degenerate curvature estimator and explicitly discusses limitations including Hessian staleness in non-convex settings.

Significance. If the result holds, the work supplies an explicit, parameter-free derivation of an improved error floor under clearly stated quadratic assumptions together with consistent empirical gains in that regime. The strengths include the machine-checkable-style proof under the diagonal-Hessian/isotropic-noise model, the correction of the degenerate estimator, and the open statement of scope limitations. These elements make the contribution a useful reference point for geometry-aware noise measures, even if immediate extension beyond quadratics remains open.

minor comments (2)
  1. [Abstract] Abstract: the claim that CWGD-Cosine 'consistently achieves approximately 20% lower final optimization error' is presented without error bars, standard deviations, number of independent runs, or mention of any post-hoc data exclusions. Adding these details would strengthen the empirical support for the reported gains across condition numbers and batch sizes.
  2. [Experiments / quadratic-setting paragraph] The abstract and quadratic-setting paragraph correctly restrict both the theorem and the exactness of the Hutchinson estimator to diagonal Hessians and isotropic noise; a short cross-reference in the experimental section reminding readers of this scope would improve readability.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for the positive and constructive review, including the accurate summary of our contributions on CWGD and the CWGD-Cosine schedule, as well as the recommendation for minor revision. No specific major comments were raised in the report.

Circularity Check

0 steps flagged

No significant circularity; derivation is self-contained proof under stated assumptions

full rationale

The central claim is an explicit proof, for strongly convex quadratics with exactly diagonal Hessians and isotropic noise, that a CWGD-modulated cosine schedule reduces the asymptotic error floor by up to a factor of two. This follows directly from the quadratic dynamics and the CWGD definition without any fitted parameters renamed as predictions, without self-citation chains, and without ansatzes smuggled via prior work. The Hutchinson estimator is stated to be exact precisely under these conditions. The derivation therefore does not reduce to its inputs by construction and remains independent of any external or self-referential load-bearing steps.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The central claim depends on the quadratic model, diagonal Hessian, and isotropic noise; these are domain assumptions rather than free parameters or new entities.

axioms (1)
  • domain assumption Objective is strongly convex quadratic with exactly diagonal Hessian and isotropic gradient noise
    Invoked to obtain the factor-of-two error reduction and to guarantee the Hutchinson estimator is exact.
invented entities (1)
  • CWGD no independent evidence
    purpose: Geometry-aware proxy for effective optimization noise
    Defined in the paper as per-sample gradient diversity weighted by inverse square root of the Hessian.

pith-pipeline@v0.9.1-grok · 5786 in / 1303 out tokens · 34105 ms · 2026-06-30T07:28:18.162904+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 2 canonical work pages · 1 internal anchor

  1. [1]

    E., and Nocedal, J

    Bottou, L., Curtis, F. E., and Nocedal, J. (2018). Optimization methods for large-scale machine learning. SIAM Review, 60(2):223--311

    2018

  2. [2]

    Duchi, J., Hazan, E., and Singer, Y. (2011). Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121--2159

    2011

  3. [3]

    K., Paul, M., Kharaghani, S., Roy, D

    Fort, S., Dziugaite, G. K., Paul, M., Kharaghani, S., Roy, D. M., and Ganguli, S. (2019). Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. In NeurIPS

    2019

  4. [4]

    Ghorbani, B., Krishnan, S., and Xiao, Y. (2019). An investigation into neural net optimization via Hessian eigenvalue density. In ICML

    2019

  5. [5]

    Gupta, V., Koren, T., and Singer, Y. (2018). Shampoo: Preconditioned stochastic tensor optimization. In ICML

    2018

  6. [6]

    Hutchinson, M. F. (1989). A stochastic estimator of the trace of the influence matrix for Laplacian smoothing splines. Communications in Statistics --- Simulation and Computation, 18(3):1059--1076

    1989

  7. [7]

    Kingma, D. P. and Ba, J. (2015). Adam: A method for stochastic optimization. In ICLR

    2015

  8. [8]

    Li, Y., Wei, C., Ma, T., and Ma, Y. (2021). Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning. In ICLR

    2021

  9. [9]

    Liu, H., Li, Z., Hall, D., Liang, P., and Ma, T. (2023). Sophia: A scalable stochastic second-order optimizer for language model pre-training. arXiv preprint arXiv:2305.14342

    work page arXiv 2023

  10. [10]

    and Hutter, F

    Loshchilov, I. and Hutter, F. (2017). SGDR: Stochastic gradient descent with warm restarts. In ICLR

    2017

  11. [11]

    and Hutter, F

    Loshchilov, I. and Hutter, F. (2019). Decoupled weight decay regularization. In ICLR

    2019

  12. [12]

    Martens, J. (2010). Deep learning via Hessian-free optimization. In ICML

    2010

  13. [13]

    and Grosse, R

    Martens, J. and Grosse, R. (2015). Optimizing neural networks with Kronecker-factored approximate curvature. In ICML

    2015

  14. [14]

    Needell, D., Ward, R., and Srebro, N. (2014). Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm. In NeurIPS

    2014

  15. [15]

    Nesterov, Y. (1983). A method for unconstrained convex minimization problem with the rate of convergence O(1/k^2) . Doklady AN USSR, 269(3):543--547

    1983

  16. [16]

    Pethick, T., Xie, W., and Cevher, V. (2025). LANTON: Learning-adaptive noise-tolerant optimization. In ICML

    2025

  17. [17]

    Polyak, B. T. (1964). Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics, 4(5):1--17

    1964

  18. [18]

    and Sridharan, K

    Rakhlin, A. and Sridharan, K. (2012). Making gradients small in stochastic subgradient descent. Technical report, University of Pennsylvania

    2012

  19. [19]

    J., Kale, S., and Kumar, S

    Reddi, S. J., Kale, S., and Kumar, S. (2018). On the convergence of Adam and beyond. In ICLR

    2018

  20. [20]

    and Stern, M

    Shazeer, N. and Stern, M. (2018). Adafactor: Adaptive learning rates with sublinear memory cost. In ICML

    2018

  21. [21]

    Sutskever, I., Martens, J., Dahl, G., and Hinton, G. (2013). On the importance of initialization and momentum in deep learning. In ICML

    2013

  22. [22]

    and Hinton, G

    Tieleman, T. and Hinton, G. (2012). Lecture 6.5-RMSProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26--31

    2012

  23. [23]

    Yao, Z., Gholami, A., Keutzer, K., and Mahoney, M. W. (2020). PyHessian: Neural networks through the lens of the Hessian. In IEEE BigData

    2020

  24. [24]

    Yao, Z., Gholami, A., Shen, S., Keutzer, K., and Mahoney, M. W. (2021). ADAHESSIAN: An adaptive second order optimizer for machine learning. In AAAI

    2021

  25. [25]

    Yin, D., Pananjady, A., Lam, M., Papailiopoulos, D., Ramchandran, K., and Bartlett, P. (2018). Gradient diversity: A key ingredient for scalable distributed learning. In AISTATS

    2018

  26. [26]

    You, Y., Gitman, I., and Ginsburg, B. (2017). Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  27. [27]

    You, Y., Li, J., Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J., Keutzer, K., and Hsieh, C.-J. (2019). Large batch optimization for deep learning: Training BERT in 76 minutes. In ICLR

    2019

  28. [28]

    E., Shallue, C

    Zhang, G., Li, L., Nado, Z., Martens, J., Sachdeva, S., Dahl, G. E., Shallue, C. J., and Grosse, R. (2019). Which algorithmic choices matter at which batch sizes? Insights from a noisy quadratic model. In NeurIPS

    2019

  29. [29]

    C., Dvornek, N., Papademetris, X., and Duncan, J

    Zhuang, J., Tang, T., Ding, Y., Tatikonda, S. C., Dvornek, N., Papademetris, X., and Duncan, J. (2020). AdaBelief optimizer: Adapting stepsizes by the belief in observed gradients. In NeurIPS

    2020