Pith. sign in

REVIEW 3 major objections 36 references

A dynamic ℓ_p-norm in SGD and SGDM adapts to changing curvature anisotropy during DNN training.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-28 15:32 UTC pith:ZEJFOUIO

load-bearing objection Dynamic lp-norm SGD with cosine-annealed p offers a concrete schedule idea but rests on an unmeasured curvature-anisotropy claim and delivers only the standard convergence rate. the 3 major comments →

arxiv 2606.02078 v1 pith:ZEJFOUIO submitted 2026-06-01 cs.LG

Beyond ell₂-norm and ell_infty-norm: A Curvature-Inspired ell_p-Norm Scheme for Deep Neural Networks

Jianhao Xu , Zhuang Yang This is my paper
classification cs.LG
keywords deep neural networksoptimizersSGDℓ_p-normcurvature anisotropygeneralizationconvergence rate
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that standard ℓ2-norm and ℓ∞-norm optimizers fail to handle the shift from strong curvature anisotropy early in training to weaker anisotropy later. It introduces a dynamical ℓ_p-norm where p begins above 2 to prevent high-curvature directions from dominating updates, then decreases toward 2 for more stable refinement. The scheme is embedded in SGD and momentum SGD to produce LPSGD and LPSGDM, which are proved to converge at O(T^{-1/2}) for nonconvex objectives and shown to improve generalization on image benchmarks.

Core claim

We propose a novel ℓ_p-norm scheme with a dynamical value of p and incorporate it into stochastic gradient descent (SGD) and SGD with momentum (SGDM), leading to two novel optimizers with better generalization performance: ℓ_p-SGD (LPSGD) and ℓ_p-SGDM (LPSGDM). The resulting optimizers suppress the dominance of high-curvature directions in the early period by utilizing a large p (p>2), followed by a gradual decrease of p toward 2 to enable more stable and refined updates, where the latter process is motivated by the cosine annealing strategy.

What carries the argument

Dynamical ℓ_p-norm whose exponent p starts above 2 and anneals toward 2, replacing the fixed norm inside the gradient step of SGD and SGDM.

Load-bearing premise

DNN training exhibits strong curvature anisotropy early and shifts toward flatter regions with weaker anisotropy later.

What would settle it

A controlled experiment in which curvature anisotropy stays constant or increases, causing the dynamic-p schedule to match or underperform fixed-p=2 or p=∞ baselines on the same tasks.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • High-curvature directions lose their early dominance, allowing updates along lower-curvature axes.
  • Later-stage updates become less prone to oscillation because coordinate-wise steps shrink as p approaches 2.
  • Both algorithms retain the standard O(T^{-1/2}) rate for nonconvex stochastic optimization.
  • Empirical gains appear on CIFAR-10, CIFAR-100 and ImageNet-1K with VGG-11, ResNet-18 and ResNet-50.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same dynamic-norm replacement could be inserted into Adam-style methods that already maintain per-coordinate statistics.
  • Replacing the cosine schedule for p with a linear or step decay would test whether the exact annealing shape is essential.
  • Applying the method to language-model pretraining or reinforcement-learning agents would check whether the early-to-late anisotropy pattern holds outside vision tasks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 0 minor

Summary. The paper proposes a dynamic ℓ_p-norm scheme for SGD and SGDM (LPSGD and LPSGDM) in which p begins above 2 to mitigate strong early-training curvature anisotropy and is annealed toward 2 via a cosine schedule to stabilize later updates in flatter regions. It claims this yields better generalization than ℓ₂- or ℓ∞-based methods on CIFAR-10/100 and ImageNet-1K with VGG-11, ResNet-18 and ResNet-50, while both algorithms achieve the standard O(T^{-1/2}) rate for non-convex stochastic optimization.

Significance. If the anisotropy premise is quantitatively supported and the empirical gains prove robust, the work could supply a concrete mechanism for phase-dependent norm adaptation in first-order methods. The convergence result itself is the textbook non-convex SGD rate and does not constitute a theoretical advance; significance therefore rests entirely on whether the dynamic schedule demonstrably exploits the claimed curvature behavior and produces statistically reliable improvements.

major comments (3)
  1. [Abstract] Abstract (motivation paragraph): the central justification—that early training exhibits “strong curvature anisotropy” while later stages are “flatter with weaker anisotropy”—is stated qualitatively with neither a quantitative definition (e.g., ratio of Hessian eigenvalues, gradient-component variance, or effective dimension) nor any cited measurements or references. Because this premise directly motivates the choice of large initial p and its subsequent decrease, its lack of empirical grounding is load-bearing for both novelty and the claimed generalization benefit.
  2. [Abstract] Abstract (convergence claim): the stated O(T^{-1/2}) rate is identical to the standard non-convex SGD bound; the manuscript must clarify whether the dynamic-p analysis produces any improvement in the leading constant or merely recovers the vanilla bound. If the latter, the theoretical guarantee does not distinguish the proposed method from existing ℓ₂-SGD.
  3. [Abstract] Abstract (experimental claims): no explicit functional form for the cosine-annealing p-schedule, no schedule parameters, no number of independent runs, and no error-bar or statistical-significance protocol are supplied. These omissions prevent verification that the reported gains are attributable to the dynamic norm rather than post-hoc schedule tuning or run-to-run variance.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the careful reading and constructive feedback. We address each major comment point by point below. Where the comments identify omissions or lack of clarity in the abstract, we agree that revisions are warranted and will update the manuscript accordingly.

read point-by-point responses

  1. Referee: [Abstract] Abstract (motivation paragraph): the central justification—that early training exhibits “strong curvature anisotropy” while later stages are “flatter with weaker anisotropy”—is stated qualitatively with neither a quantitative definition (e.g., ratio of Hessian eigenvalues, gradient-component variance, or effective dimension) nor any cited measurements or references. Because this premise directly motivates the choice of large initial p and its subsequent decrease, its lack of empirical grounding is load-bearing for both novelty and the claimed generalization benefit.

    Authors: We agree that the motivation paragraph in the abstract relies on a qualitative description without quantitative grounding or citations. In the revised version we will add a concise quantitative definition of curvature anisotropy (e.g., the ratio of the largest to smallest Hessian eigenvalue or the variance across gradient components) together with a short supporting measurement on a small-scale model, and we will cite relevant prior work on Hessian spectra and curvature dynamics during DNN training. These additions will appear in both the abstract (space permitting) and the introduction. revision: yes

  2. Referee: [Abstract] Abstract (convergence claim): the stated O(T^{-1/2}) rate is identical to the standard non-convex SGD bound; the manuscript must clarify whether the dynamic-p analysis produces any improvement in the leading constant or merely recovers the vanilla bound. If the latter, the theoretical guarantee does not distinguish the proposed method from existing ℓ₂-SGD.

    Authors: The analysis recovers the textbook O(T^{-1/2}) non-convex rate without an improved leading constant. We will revise the abstract and theory section to state explicitly that the bound matches the standard SGD guarantee and that the theoretical contribution is to establish convergence of the dynamic-p scheme under the same assumptions rather than a tighter rate. The practical distinction is shown through the empirical results. revision: yes

  3. Referee: [Abstract] Abstract (experimental claims): no explicit functional form for the cosine-annealing p-schedule, no schedule parameters, no number of independent runs, and no error-bar or statistical-significance protocol are supplied. These omissions prevent verification that the reported gains are attributable to the dynamic norm rather than post-hoc schedule tuning or run-to-run variance.

    Authors: The abstract is length-limited and therefore omits these details, which are provided in the experimental section of the full manuscript (cosine schedule form, hyper-parameters, 3–5 independent runs with standard deviations, and significance testing). We will add a brief statement of the schedule form and mention of multiple runs with error bars to the abstract or a footnote to improve verifiability without exceeding length constraints. revision: partial

Circularity Check

0 steps flagged

No circularity; standard convergence bound and qualitative motivation

full rationale

The paper's central proposal is a dynamic-p ℓ_p norm (large p early, annealed toward 2) motivated by a qualitative description of curvature anisotropy. The claimed O(T^{-1/2}) non-convex rate is the textbook SGD rate and is not shown to be derived from or equivalent to the anisotropy premise by construction. No self-citations, fitted parameters renamed as predictions, or self-definitional steps are present in the abstract or described claims. The derivation chain for the rate is therefore independent of the schedule choice.

Axiom & Free-Parameter Ledger

2 free parameters · 1 axioms · 0 invented entities

Review performed on abstract only; the ledger therefore records only the explicitly stated domain assumption and the schedule parameters implied by the cosine-annealing description. Full paper would likely add more fitted constants and background lemmas.

free parameters (2)
  • initial p (>2)
    Starting value of p chosen to suppress high-curvature directions; value and selection rule not given in abstract.
  • cosine-annealing schedule parameters
    Parameters controlling how p decreases from initial value to 2; these are free choices that must be set before training.
axioms (1)
  • domain assumption DNN training exhibits strong curvature anisotropy early and weaker anisotropy later.
    Abstract presents this as a general fact that directly motivates the choice of large-then-decreasing p.

pith-pipeline@v0.9.1-grok · 5921 in / 1523 out tokens · 38101 ms · 2026-06-28T15:32:36.492727+00:00 · methodology

0 comments
read the original abstract

The existing optimizers for deep neural networks (DNNs) typically rely on either the $\ell_2$ norm or the $\ell_\infty$ norm, resulting in optimizers that do not adapt well to substantial changes in curvature across parameter dimensions. Generally, the training process of DNNs often exhibits strong curvature anisotropy in the early period, whereas in the later period, the training process of DNNs tends to move toward flatter regions with weaker anisotropy. Particularly, optimizers based on the \(\ell_2\)-norm are usually dominated by high-curvature directions, restricting updates of optimizers along with lower curvature direction and thus leading to a slower convergence rate. While optimizers based on the \(\ell_\infty\)-norm are prone to oscillations in flatter regions, due to the coordinate-wise updates of the same magnitude. To address these two extreme cases generated by $\ell_2$ and $\ell_\infty$ norms, we propose a novel $\ell_p$-norm scheme with a dynamical value of $p$ and incorporate it into stochastic gradient descent (SGD) and SGD with momentum (SGDM), leading to two novel optimizers with better generalization performance: ${\ell_p}$-SGD (LPSGD) and ${\ell_p}$-SGDM (LPSGDM). Particularly, the resulting optimizers suppress the dominance of high-curvature directions in the early period by utilizing a large $p$ ($p>2$), followed by a gradual decrease of $p$ toward 2 to enable more stable and refined updates, where the latter process is motivated by the cosine annealing strategy. We establish theoretical guarantees of the resulting algorithms and analyze that both LPSGD and LPSGDM achieve an \(O(T^{-1/2})\) convergence rate for the nonconvex setting. Extensive experiments are conducted on benchmark datasets, including CIFAR-10, CIFAR-100, and ImageNet-1K, with multiple DNNs such as VGG-11, ResNet-18, and ResNet-50.

Figures

Figures reproduced from arXiv: 2606.02078 by Jianhao Xu, Zhuang Yang.

Figure 1
Figure 1. Figure 1: The evolution of Hessian eigenvalue distribution during training. (a) ResNet-18 [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: (a) Testing top-1 accuracy of SGD and LPSGD on ResNet-18/CIFAR-10. (b) [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Optimization trajectories of GD and SignGD on two toy loss landscapes. (a) [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: (a) Testing top-1 accuracy curves on ResNet-18 / cifar-10 under different [PITH_FULL_IMAGE:figures/full_fig_p024_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 2 canonical work pages · 1 internal anchor

  1. [1]

    Neural Networks 197: 108481 (2026)

    Haizhou Du, Chongyi Qiu, Huan Huo: FedHyperGraph: A layer-wise personalized federated learning with correlation graphs in hyperbolic space. Neural Networks 197: 108481 (2026)

    2026

  2. [2]

    Neural Networks 197: 108453 (2026)

    Yang Liu, Jingchen Li, Huarui Wu, Ying Zhou, Chunjiang Zhao: To- wards more effective skill discovery in reinforcement learning by incor- porating state reachability. Neural Networks 197: 108453 (2026)

    2026

  3. [3]

    Neural Networks 196: 108388 (2026)

    Zhiyu Liao, Kang Chen, Yuanguo Lin, Kangkang Li, Yunxuan Liu, Hefeng Chen, Xingwang Huang, Yuanhui Yu: Attack and defense tech- niques in large language models: A survey and new perspectives. Neural Networks 196: 108388 (2026)

    2026

  4. [4]

    A stochastic approximation method[J]

    Robbins H, Monro S. A stochastic approximation method[J]. The annals of mathematical statistics, 1951: 400-407. 28

    1951

  5. [5]

    Learning from streaming video with orthogonal gradients[C]//Proceedings of the Computer Vision and Pat- tern Recognition Conference

    Han T, Gokay D, Heyward J, et al. Learning from streaming video with orthogonal gradients[C]//Proceedings of the Computer Vision and Pat- tern Recognition Conference. 2025: 13651-13660

    2025

  6. [6]

    Resolving discrepancies in compute-optimal scaling of language models[J]

    Porian T, Wortsman M, Jitsev J, et al. Resolving discrepancies in compute-optimal scaling of language models[J]. Advances in Neural In- formation Processing Systems, 2024, 37: 100535-100570

    2024

  7. [7]

    Kingma, Jimmy Ba: Adam: A Method for Stochastic Opti- mization

    Diederik P. Kingma, Jimmy Ba: Adam: A Method for Stochastic Opti- mization. ICLR (Poster) 2015

    2015

  8. [8]

    ICLR (Poster) 2019

    Ilya Loshchilov, Frank Hutter: Decoupled Weight Decay Regularization. ICLR (Poster) 2019

    2019

  9. [9]

    Adaptive subgradient methods for online learning and stochastic optimization[J]

    Duchi J, Hazan E, Singer Y. Adaptive subgradient methods for online learning and stochastic optimization[J]. Journal of machine learning re- search, 2011, 12(7)

    2011

  10. [10]

    RMSprop converges with proper hyper- parameter[C]//International Conference on Learning Representations

    Shi N, Li D, Hong M, et al. RMSprop converges with proper hyper- parameter[C]//International Conference on Learning Representations. 2021

    2021

  11. [11]

    The geometry of sign gradient descent

    Lukas Balles, Fabian Pedregosa, Nicolas Le Roux: The Geometry of Sign Gradient Descent. CoRR abs/2002.08056 (2020)

    work page arXiv 2002

  12. [12]

    signSGD: Compressed optimisation for non-convex problems[C]//International conference on machine learning

    Bernstein J, Wang Y X, Azizzadenesheli K, et al. signSGD: Compressed optimisation for non-convex problems[C]//International conference on machine learning. PMLR, 2018: 560-569

    2018

  13. [13]

    Le: Symbolic Discovery of Optimization Algorithms

    Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, Quoc V. Le: Symbolic Discovery of Optimization Algorithms. NeurIPS 2023

    2023

  14. [14]

    ICLR 2024

    Lizhang Chen, Bo Liu, Kaizhao Liang, Qiang Liu: Lion Secretly Solves a Constrained Optimization: As Lyapunov Predicts. ICLR 2024

    2024

  15. [15]

    Dissecting adam: The sign, magnitude and vari- ance of stochastic gradients[C]//International Conference on Machine Learning

    Balles L, Hennig P. Dissecting adam: The sign, magnitude and vari- ance of stochastic gradients[C]//International Conference on Machine Learning. PMLR, 2018: 404-413. 29

    2018

  16. [16]

    Equilibrated adaptive learning rates for non-convex optimization[J]

    Dauphin Y, De Vries H, Bengio Y. Equilibrated adaptive learning rates for non-convex optimization[J]. Advances in neural information process- ing systems, 2015, 28

    2015

  17. [17]

    Zico Kolter, Ameet Tal- walkar: Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability

    Jeremy Cohen, Simran Kaur, Yuanzhi Li, J. Zico Kolter, Ameet Tal- walkar: Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability. ICLR 2021

    2021

  18. [18]

    On faster convergence of scaled sign gradient descent[J]

    Li X, Lin K Y, Li L, et al. On faster convergence of scaled sign gradient descent[J]. IEEE Transactions on Industrial Informatics, 2023, 20(2): 1732-1741

    2023

  19. [20]

    Deep resid- uallearningforimage recognition.In Proceedingsof the2016 IEEECon- ference on Computer Vision and Pattern Recognition, pages 770–778, Las Vegas, NV, USA, June 2016

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep resid- uallearningforimage recognition.In Proceedingsof the2016 IEEECon- ference on Computer Vision and Pattern Recognition, pages 770–778, Las Vegas, NV, USA, June 2016. IEEE Computer Society

    2016

  20. [21]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009

    2009

  21. [22]

    Ugur Güney, Yann N

    Levent Sagun, Utku Evci, V. Ugur Güney, Yann N. Dauphin, Léon Bottou: Empirical Analysis of the Hessian of Over-Parametrized Neural Networks. ICLR (Workshop) 2018

    2018

  22. [23]

    Geometry of neural network loss surfaces via random matrix theory[C]//International conference on machine learn- ing

    Pennington J, Bahri Y. Geometry of neural network loss surfaces via random matrix theory[C]//International conference on machine learn- ing. PMLR, 2017: 2798-2806

    2017

  23. [24]

    An investigation into neural net op- timization via hessian eigenvalue density[C]//International Conference on Machine Learning

    Ghorbani B, Krishnan S, Xiao Y. An investigation into neural net op- timization via hessian eigenvalue density[C]//International Conference on Machine Learning. PMLR, 2019: 2232-2241

    2019

  24. [25]

    Pyhessian: Neural networks through the lens of the hessian[C]//2020 IEEE international conference on big data (Big data)

    Yao Z, Gholami A, Keutzer K, et al. Pyhessian: Neural networks through the lens of the hessian[C]//2020 IEEE international conference on big data (Big data). IEEE, 2020: 581-590. 30

    2020

  25. [26]

    A deeper look at the hessian eigenspectrum of deep neural networks and its applications to regularization[C]//Proceedings of the AAAI Conference on Artificial In- telligence

    Sankar A R, Khasbage Y, Vigneswaran R, et al. A deeper look at the hessian eigenspectrum of deep neural networks and its applications to regularization[C]//Proceedings of the AAAI Conference on Artificial In- telligence. 2021, 35(11): 9481-9488

    2021

  26. [27]

    Optimizing neural networks with kronecker- factored approximate curvature[C]//International conference on ma- chine learning

    Martens J, Grosse R. Optimizing neural networks with kronecker- factored approximate curvature[C]//International conference on ma- chine learning. PMLR, 2015: 2408-2417

    2015

  27. [28]

    Shampoo: Preconditioned stochastic tensor optimization[C]//International Conference on Machine Learning

    Gupta V, Koren T, Singer Y. Shampoo: Preconditioned stochastic tensor optimization[C]//International Conference on Machine Learning. PMLR, 2018: 1842-1850

    2018

  28. [29]

    ICLR 2024

    Hong Liu, Zhiyuan Li, David Leo Wright Hall, Percy Liang, Tengyu Ma: Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training. ICLR 2024

    2024

  29. [30]

    Sharpness-aware lookahead for accelerat- ing convergence and improving generalization[J]

    Tan C, Zhang J, Liu J, et al. Sharpness-aware lookahead for accelerat- ing convergence and improving generalization[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024, 46(12): 10375-10388

    2024

  30. [31]

    ICML 2025

    Xinyu Luo, Site Bai, Bolian Li, Petros Drineas, Ruqi Zhang, Brian Bullins: Stacey: Promoting Stochastic Steepest Descent via Accelerated ℓp-Smooth Nonconvex Optimization. ICML 2025

    2025

  31. [32]

    Gradual Stochastic Gradient Descent: from signSGD to SGD viaℓp Norm[C]//Workshop on Scientific Methods for Understanding Deep Learning

    Yuan J, Jiachen L, Nie F. Gradual Stochastic Gradient Descent: from signSGD to SGD viaℓp Norm[C]//Workshop on Scientific Methods for Understanding Deep Learning

  32. [33]

    Continual momentum filtering on parameter space for online test-time adaptation[C]//The twelfth international conference on learning representations

    Lee J H, Chang J H. Continual momentum filtering on parameter space for online test-time adaptation[C]//The twelfth international conference on learning representations. 2024

    2024

  33. [34]

    Adaptive momentum-based optimization to train deep neural network for simulating the static sta- bility of the composite structure[J]

    Chi Z, Jiang Z, Kamruzzaman M M, et al. Adaptive momentum-based optimization to train deep neural network for simulating the static sta- bility of the composite structure[J]. Engineering with Computers, 2022, 38(Suppl 5): 4027-4049

    2022

  34. [35]

    1983, 269: 543

    NesterovY.Amethodforsolvingtheconvexprogrammingproblemwith convergence rate O (1/k2)[C]//Dokl akad nauk Sssr. 1983, 269: 543. 31

    1983

  35. [36]

    Very Deep Convolutional Networks for Large-Scale Image Recognition

    Karen Simonyan and Andrew Zisserman. Very deep convolutional net- works for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014

    work page internal anchor Pith review Pith/arXiv arXiv 2014

  36. [37]

    Ima- genet: A large-scale hierarchical image database

    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Ima- genet: A large-scale hierarchical image database. In 2009 IEEE Confer- ence on Computer Vision and Pattern Recognition, pp. 248–255. Ieee, 2009. 32

    2009