ZENITH: Automated Gradient Norm Informed Stochastic Optimization

Dhrubo Saha

REVIEW 5 major objections 4 minor 4 references

ZENITH: Automated Gradient Norm Informed Stochastic Optimization

T0 review · 5 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new learning-rate scheduler, ZENITH, sets the step size at each iteration to the ratio of the current gradient's L2 norm (smoothed over a window) to the largest such windowed norm seen so far.

desk verdict ZENITH is a simple, low-overhead learning-rate scheduler with a novel norm-ratio rule, but the convergence proof is unsupported and the empirical claims overreach what single-run experiments show. read the letter →

arxiv 2601.15212 v2 pith:2UVZSZZV submitted 2026-01-21 cs.LG cs.CV

Dhrubo Saha This is my paper

classification cs.LGcs.CV

keywords learning-rateschedulinggradientnormadaptiveoptimizerstochasticoptimizationdeeplearningimageclassificationobjectdetectionregularization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ZENITH claims that a single scalar rule can replace hand-tuned learning-rate schedules: at each step the learning rate is the initial rate times the sliding-window mean of recent gradient norms divided by the historical peak of that mean. Across 6 image-classification tasks and COCO detection/segmentation, the paper reports higher test accuracy in less wall-clock time than 11 published automatic optimizers, with zero per-weight memory overhead and roughly 1.3% per-iteration time cost. The rule is scale-invariant by construction, which the paper argues makes it robust across tasks and compatible with L2 regularization. The paper's convergence proof, however, rests on an asserted equivalence between the window mean and the current gradient norm (Appendix B, after Eq. 21), and it analyzes a deterministic regime that does not match the mini-batch training used in the experiments; the paper states that proportionality rather than deriving it.

What carries the argument

The ratio μ_t/Z_t — the sliding-window mean of the L2 gradient norm divided by its historical maximum ('zenith'). This is a scalar, model-dimension-independent statistic that converts the gradient norm's temporal decay into a learning-rate multiplier between 0 and 1, enforcing scale-invariance and eliminating the need for per-weight state.

What would settle it

A multi-seed rerun of the six classification benchmarks with confidence intervals would test the empirical claim; a constructed gradient sequence where the windowed mean decays to zero while the true gradient norm does not would directly refute the claimed convergence guarantee.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the temporal evolution of the gradient L2 norm contains enough information to schedule the learning rate: η_t = η_0 · μ_t / Z_t, where μ_t is the mean of the W most recent gradient norms and Z_t is the running maximum of μ_t. This automatically anneals the learning rate as gradients shrink, mimics exponential, polynomial, or cosine decay depending on the task, requires no auxiliary per-weight buffers, and does not depend on the absolute scale of the loss. Empirically, ZENITH is reported to beat all 11 compared optimizers on the joint objective of higher test accuracy and lower wall-clock time on MNIST, CIFAR-10, CIFAR-100, Food-101, Tin

Load-bearing premise

The proof assumes, without derivation, that the sliding-window mean gradient norm is proportional to the current gradient norm (μ_t ≈ ||∇L(θ_t)||); if that proportionality fails, Theorem B.5's conclusion that the gradient norm tends to zero does not follow.

Editorial extensions

If this is right

Practitioners using SGD for vision can drop manual or pre-scheduled learning-rate decay and use this rule instead, if the empirical claims replicate.
Because ZENITH adds no per-weight buffers, it can scale to models where memory-hungry optimizers like Adam or D-Adaptation become expensive.
Unlike several distance-aware and Polyak-style baselines, ZENITH is reported to work together with L2 regularization, so explicit weight decay remains available.
The scale-invariance of the rule suggests it should transfer across datasets and architectures without re-tuning the initial learning rate, though the paper only varies some settings.
If the convergence theorem can be repaired, the rule would come with a parameter-free descent guarantee for smooth non-convex objectives.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: applying the same ratio rule to Adam-style optimizers is a natural extension; the paper scopes itself to SGD, so this is not its claim.
Beyond the paper: the flat-minima explanation is an interpretation, not a proven mechanism; a controlled experiment with varying batch size and label noise could separate the scheduling effect from the flatness story.
Beyond the paper: the paper does not report multiple seeds or confidence intervals, so the margins of the reported gains remain unquantified; a multi-seed replication would settle it.
Beyond the paper: W-sensitivity is demonstrated only on CIFAR-100; checking it on the other benchmarks is a cheap robustness test.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

ZENITH is a simple and cheap learning-rate scheduler with a clean idea: scale the LR by the ratio of a rolling mean of gradient norms to its historical maximum. The specific rule doesn't show up in the cited literature, and the overhead is genuinely low — about 1.3% per-iteration time and no per-weight memory. The empirical sweep is broad: six classification settings, COCO detection/segmentation, and eleven baselines. The analysis of LR trajectories and sharpness is a reasonable addition.

The problem is the theory. Theorem B.5 does not follow from its proof. After Eq. (21) you have (μ_t/Z_t)||∇L||^2 → 0. To conclude ||∇L|| → 0, the proof asserts μ_t ≈ ||∇L(θ_t)||. That proportionality is never derived, and a sliding-window mean doesn't automatically track the instantaneous norm. If μ_t/Z_t decays (which the scheduler is designed to do), the product can go to zero with the gradient norm bounded away from zero. The theorem also restricts to a deterministic gradient, so it doesn't cover the mini-batch training used in every experiment.

The empirical section has its own softening. The abstract's 'higher test accuracy in lower wall-clock time' is not true on all six classification runs: ALIG and SPS are faster on Food-101, Tiny ImageNet, and ImageNet-100, though at lower accuracy; D-Adaptation is faster in Table 5. There are no error bars or seeds anywhere, and code is only promised upon acceptance. Detection results rely on mAP50 and on the five baselines that didn't diverge under fixed settings.

None of this means the paper is dishonest. The mechanics are clear, the method is coherent, and the baselines are external. But the load-bearing proof gap kills the theoretical pillar, and the empirical claims overstate what a single-run comparison supports. As written, it's not ready for publication.

The paper is worth a serious referee — the idea is simple and plausible, and the experiments are extensive enough to justify careful review. But the referee should require a corrected or removed theorem, multi-seed error bars, released code, and a more careful claim about joint accuracy/time. If those land, this could become a solid contribution. My recommendation: engage with it, but don't accept in current form.

Referee Report

5 major / 4 minor

Summary. The paper proposes ZENITH, a learning-rate scheduler that sets the step size as η_t = η0·(μ_t/Z_t), where μ_t is a sliding-window mean of recent gradient L2 norms and Z_t is the running maximum of μ_t. The algorithm stores no per-weight state and is claimed to add only ~1.3% per-iteration overhead. The central claims are that ZENITH outperforms 11 published automatic optimizers in test accuracy and wall-clock time across six image-classification setups, and also gives superior mAP on MS COCO keypoint detection and instance segmentation, while a convergence theorem in Appendix B guarantees convergence to a stationary point. I find the convergence proof invalid as written and the empirical evidence lacking in statistical rigor, so the paper's central claims are not currently established.

Significance. If the empirical claims were reproducible, ZENITH would be a useful, memory-free alternative to existing automatic LR schedulers. The core idea is simple and the benchmark breadth is a strength: the paper compares against many external, independently published optimizers with standard settings, and the LR is a deterministic function of measured gradient norms rather than being fitted to test data. The overhead measurement in Table 1 is also a useful contribution. However, the formal convergence result is unsupported, the experiments are single runs without error bars or significance tests, code is only promised upon acceptance, and the detection/segmentation evaluation uses non-standard metrics. As submitted, the paper does not establish its headline claims.

major comments (5)

[Appendix B.2, Eq. (21)-(24)] Theorem B.5 is not proven. From the finite-sum inequality (19) the proof obtains lim_t (μ_t/Z_t)||∇L(θ_t)||^2 = 0. To conclude lim_t ||∇L(θ_t)|| = 0, the proof asserts after Eq. (21) that μ_t ≈ ||∇L(θ_t)||, but this proportionality is never derived and is not a consequence of Assumptions B.3-B.4. μ_t is the arithmetic mean of the previous W gradient norms, not the instantaneous norm. Moreover, the statement that Z_t is bounded is also merely asserted. Since η_t = η0 μ_t/Z_t, the vanishing product in Eq. (21) can in principle be driven by the ratio μ_t/Z_t changing over time or by Z_t growing, with ||∇L|| staying bounded away from zero. The final conclusion Eq. (24) therefore does not follow from the supplied argument.
[Appendix B.1] The convergence theorem is stated only for a deterministic regime where the gradient signal dominates stochastic noise. All experiments in Sections 3 and 4 use mini-batch training with data augmentation and stochastic gradients. Even if Theorem B.5 were correct, it would not apply to the actual experimental setting. A convergence guarantee for the stochastic mini-batch case is needed to support the claim that the algorithm is principled in the settings where it is evaluated.
[Tables 3 and 5] All reported benchmark results are single runs with no error bars, confidence intervals, or significance tests. Some differences are small (e.g., ImageNet-100: ZENITH 78.2% vs Prodigy 78.0%; MNIST: ZENITH 99.57% vs DAdapt 99.58%), so the claims that ZENITH is uniformly superior and that no baseline beats it on both objectives are not supported by the data as presented. Multiple seeds and a defined comparison procedure are necessary before these head-to-head claims can be assessed.
[Section 4, Table 5] The COCO experiments report only mAP50 (Box mAP50, Keypoint mAP50, Mask mAP50). The standard COCO metric is mean AP averaged over IoU thresholds [0.5:0.95], which is more discriminative. The text claims 'superior test performance across all evaluation metrics,' but only mAP50 is reported. Additionally, several baselines are said to have 'failed' or produced invalid results, which is not documented in detail. This weakens the detection/segmentation evidence.
[Section 3.1, Table 3] The 'Time' column is not clearly defined. Training durations are set per benchmark (Table 2), but the reported times differ widely across optimizers. If the reported time is the time to reach the reported accuracy, the stopping criterion and its uniform application must be specified. If it is the total training time, the values should match the fixed durations. Without this definition, the joint accuracy-time comparison is ambiguous.

minor comments (4)

[Eq. (15)-(16)] Theorem B.5 assumes η0 ≤ 2/M, but the proof requires C = 1 - (M/2)η0 > 0, i.e., η0 < 2/M. The boundary case η0 = 2/M is not handled.
[Table 3, ImageNet-100 row] LQA reports 1.00% accuracy in 0.20 hr. This appears to be a failed run and should be explained or excluded, rather than listed as a baseline result.
[Section 3.1] The code is only 'promised upon acceptance.' For a paper whose main claim is empirical superiority, the reference implementation and notebooks should be provided as supplementary material at submission.
[Appendix C] Figure 6 shows that larger window sizes improve accuracy and that W = 500 is not the best choice. The text explains the default was kept for fairness, but this should be stated more explicitly in the main text, since it is relevant to the 'no tuning' claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: ZENITH's empirical comparisons are external and its LR is a deterministic function of measured gradient norms; the proof gap at Eq. (21) is an unsupported assumption, not a circular reduction.

full rationale

The central empirical claim is not circular. ZENITH's learning rate is computed as η_t = η0 μ_t/Z_t from measured gradient L2 norms (Eq. 3), with no parameter fitted to test accuracy and no use of the test set in the scheduler; comparisons are against independently published optimizers with their recommended settings. There are no self-citations, imported uniqueness theorems, or renamed known results that carry load. The only notable derivation concern is in the proof of Theorem B.5: after obtaining lim_t (μ_t/Z_t)||∇L(θ_t)||^2 = 0 (Eq. 21), the proof asserts 'The sliding window mean μ_t scales with the local gradient magnitude: μ_t ≈ ||∇L(θ_t)||' without derivation. If this proportionality were true the cubic limit would follow, but the assertion is neither implied by Assumptions B.3–B.4 nor by the definition of μ_t as a mean of past norms. This is an unsupported step in the proof (a soundness gap), but it is not a circular reduction of the theorem to its inputs: μ_t is not defined as ||∇L(θ_t)||, and the paper does not fit the conclusion into the definition. Under the rule that only definitional equivalences, fitted-parameter predictions, or self-citation chains count as circularity, no circular step is established; the empirical contribution remains independently benchmarked.

Assumptions & free parameters 2 free parameters · 8 assumptions · 0 invented entities

The paper's contribution is a schedule with two user-set scalars (η0, W), both fixed at standard values and shown to be mildly sensitive. The mathematical cost is concentrated in the convergence appendix: besides standard smoothness assumptions, the proof needs an unverified deterministic regime, boundedness of the norm maximum, and the asserted equivalence μ_t ≈ ||∇L|| — the last of which is effectively the conclusion.

free parameters (2)

η0 (initial learning rate) = 0.1 (fixed; not tuned per task)
Initial LR in Eq. (3); fixed at 0.1 for ZENITH and the other optimizers that need one (§3.1). Fig. 3g-h shows mild sensitivity (74.3% at 0.2 vs 73.8% at 0.1).
W (window size) = 500 (fixed; not tuned per task)
Sliding-window length for the rolling mean in Eq. (1); fixed at 500. Appendix C shows W=500 is not the best choice — larger windows do slightly better — demonstrating sensitivity to the choice, though modest.

assumptions (8)

standard math L is bounded below by L*
Assumption B.3 in Appendix B; standard for non-convex convergence analysis.
standard math Gradient of L is M-Lipschitz
Assumption B.4; standard smoothness condition used in the Descent Lemma (Eq. 11).
domain assumption η0 < 2/M holds
Eqs. (15)–(16): the proof needs C = 1 − M·η0/2 > 0, but M (the smoothness constant) is unknown in practice, so the precondition of Theorem B.5 cannot be checked for real models.
domain assumption Deterministic regime: gradient signal dominates stochastic noise
Appendix B.1 states the deterministic regime is analyzed; all experiments are mini-batch (stochastic) training, so the theorem does not cover the reported runs.
ad hoc to paper μ_t ≈ ||∇L(θ_t)|| in the limit
Appendix B, after Eq. (21): 'The sliding window mean μ_t scales with the local gradient magnitude.' Asserted without derivation; the convergence conclusion (Eq. 24) depends entirely on it.
domain assumption Z_t is bounded and converges to a finite Z_∞ > 0
Appendix B, after Eq. (20): the proof needs the running max of gradient norms bounded and positive in the limit; boundedness follows from no stated assumption under B.3–B.4 alone.
domain assumption Higher LRs evade local minima and converge to flatter/global minima
Section 3.2: 'It is already well-known in the ML community that higher LRs evade local minima and converge to global minima' — used to attribute ZENITH's generalization to flat minima; no citation or derivation.
domain assumption Training loss can be near zero at both sharp and flat minima
Section 3.2 / Fig. 3c: the sharp-vs-flat test-loss penalty argument assumes training loss is near zero at both minima, stated explicitly in the text.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ZENITH: Automated Gradient Norm Informed Stochastic Optimization." pith.science (2026). https://pith.science/paper/2UVZSZZV

@misc{pith2026260115212,
  author       = {Pith},
  title        = {Pith review of: ZENITH: Automated Gradient Norm Informed Stochastic Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2UVZSZZV}},
  note         = {Machine review of arXiv:2601.15212}
}

read the original abstract

Training deep computer vision models requires manual oversight or hyperparameter tuning of the learning rate (LR) schedule. While existing adaptive optimizers schedule the LR automatically, they suffer from computational and memory overhead, incompatibility with regularization, and suboptimal LR choices. In this work, we introduce the ZENITH (Zero-overhead Evolution using Norm-Informed Training History) optimizer, which adapts the LR using the temporal evolution of the gradient norm. Image classification experiments spanning 6 CNN architectures and 6 benchmarks demonstrate that ZENITH achieves higher test accuracy in lower wall-clock time than baselines. It also yielded superior mAP in object detection, keypoint detection, and instance segmentation on MS COCO using the R-CNN family of models. Furthermore, its compatibility with regularization enables even better generalization.

Figures

Figures reproduced from arXiv: 2601.15212 by the authors.

**Figure 1.** Training loss and test accuracy curves against wall-clock time for the 6 image classification experiments. The color-coding for each optimizer is as follows: ALIG, COCOB, D-Adaptation, DoG, DoWG, GeN, L4, LQA, PAL, Prodigy, SPS, and ZENITH. This color scheme is consistent across all other figures as well. Training loss magnitudes in this figure are scaled by a factor of 10 or 100 for visual clarity. This linear tran… view at source ↗

**Figure 2.** (a) Evolution of the L2 gradient norm and (b) the corresponding LR used by ZENITH across iterations. 3.2. Experimental Results and Discussion [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** (a–b) Influence of the LR on minima sharpness and test accuracy for vanilla SGD; (c) conceptual explanation of the generalization benefits of flat minima; (d) loss landscapes around the minima in the CIFAR-100 experiment; (e) accuracy versus minima sharpness across 4 experiments; (f) LR trajectories for distance-aware and Polyak-style baselines; (g–h) sensitivity of optimizers to the choice of initial LR. ima reache… view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Impact of regularization of strength 2e-4 on LR trajectories in (a) CIFAR-100 and (b) Food-101. regularization boosts test accuracy compared to not using regularization, but for an artificial reason. Without regularization, the scale-sensitivity of these Polyak-style…

**Figure 5.** Figure 5: Training loss and test mAP50 curves against wall-clock time for detection and segmentation experiments. The color-coding for each optimizer is as follows: COCOB, D-Adaptation, DoG, PAL, and ZENITH. Training loss magnitudes in this figure are scaled by a factor of 10 or…

**Figure 6.** Figure 6: Effect of the window size hyperparameter on the (a) LR trajectory, (b) test accuracy evolution, and (c) convergence test accuracy. To evaluate the impact of the window size W, the CIFAR-100 experiment was repeated across six window sizes: 50, 150, 500, 1500, 5000, and …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

4 extracted references · 4 linked inside Pith

[2014]

and Xu, S

Bu, Z. and Xu, S. Gradient descent with generalized new- ton’s method.arXiv preprint arXiv:2407.02772,

arXiv
[2016]

and Defazio, A

Mishchenko, K. and Defazio, A. Prodigy: An expedi- tiously adaptive parameter-free learner.arXiv preprint arXiv:2306.06101,

arXiv
[2018]

and Zisserman, A

Simonyan, K. and Zisserman, A. Very deep convolu- tional networks for large-scale image recognition.arXiv preprint arXiv:1409.1556,

arXiv
[2021]

and Hutter, F

Loshchilov, I. and Hutter, F. Sgdr: Stochastic gra- dient descent with warm restarts.arXiv preprint arXiv:1608.03983,

arXiv

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [2014]

and Xu, S

Bu, Z. and Xu, S. Gradient descent with generalized new- ton’s method.arXiv preprint arXiv:2407.02772,

arXiv

[2] [2016]

and Defazio, A

Mishchenko, K. and Defazio, A. Prodigy: An expedi- tiously adaptive parameter-free learner.arXiv preprint arXiv:2306.06101,

arXiv

[3] [2018]

and Zisserman, A

Simonyan, K. and Zisserman, A. Very deep convolu- tional networks for large-scale image recognition.arXiv preprint arXiv:1409.1556,

arXiv

[4] [2021]

and Hutter, F

Loshchilov, I. and Hutter, F. Sgdr: Stochastic gra- dient descent with warm restarts.arXiv preprint arXiv:1608.03983,

arXiv