REVIEW 3 major objections 5 minor 1 cited by
Adaptive Batch Sizes Using Non-Euclidean Gradient Noise Scales for Stochastic Sign and Spectral Descent
T0 review · 3 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read This paper derives gradient noise scales for sign-based and spectral optimizers from the dual norms of their geometries, and shows that adaptive batch sizes using these metrics match constant-batch validation loss while reducing training st
desk verdict Non-Euclidean GNS theory is clean and worth taking seriously; the headline step-reduction claim for Signum/Muon is not yet supported by the experiments. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the dual-norm gradient noise scale, a ratio of two dual norms: the expected gradient error (component-wise standard deviation for signSGD; row-wise covariance nuclear norm for specSGD) over the true gradient norm, both measured in the optimizer's dual norm (ℓ1 for sign, Schatten-1 for spectral). Setting the batch size to θ^{-2} times this ratio keeps the noise-to-signal ratio constant. Supporting machinery includes Lemma 3.1 (a Hölder-based lower bound on progress), convergence theorems in the appendix, and a distributed estimator that uses per-rank mini-batch gradients to compute the variance statistics without extra data passes.
What would settle it
Measure, for Signum or Muon on a fixed model, the optimal batch size at several training steps by scanning B and computing per-sample progress; if the maximizing batch size does not track θ^{-2} times the ℓ1 (or S1) GNS across training, the transfer from the signSGD/specSGD theory to stateful optimizers fails.
Extended reading notes
Core claim
For stochastic steepest descent under a general norm, the expected inner product of the true gradient with the stochastic descent direction is bounded below by the dual norm of the true gradient minus the expected dual norm of the gradient error (Lemma 3.1). The bias introduced by sign and spectral updates therefore scales with the dual norm of the noise, and the critical batch size becomes the squared ratio of noise to signal in that dual norm: B_ℓ1 = ||σ||_1² / ||∇L||_1² for signSGD and B_S1 = ||C_row^{1/2}||_{S1}² / ||∇L||_{S1}² for specSGD. The paper proves convergence for the resulting adaptive schedule and validates it empirically, reporting up to 66.77% step reduction for Muon on the
Load-bearing premise
The load-bearing premise is that the GNS measured on raw mini-batch gradients still governs the optimal batch size after the optimizer adds momentum and semi-orthogonalization, since the theory is derived for plain signSGD and specSGD while the headline results use Signum and Muon.
Editorial extensions
If this is right
- Adaptive batch schedules for signSGD, Signum, specSGD, and Muon can be set from a single scalar per step instead of hand-tuned ramp schedules.
- Step-count savings can be large for momentum and orthogonalized variants: up to 66.61% for Signum and 66.77% for Muon on the 160M Llama model, with no loss in validation loss.
- The critical batch size is always a constant multiple of the corresponding GNS under all three turning-point definitions, so tuning θ is equivalent to choosing an operating point on a fixed saturation curve.
- The distributed variance estimator gives unbiased population noise estimates using existing AllReduce or ReduceScatter traffic, adding little extra memory or compute.
- Convergence holds with the batch-size rule: sublinear in the non-strongly-convex setting and linear in the strongly-convex setting, with the (1−θ)² factor controlling the noise term.
Reading between the lines
- If the noise scale transfers from plain signSGD/specSGD to momentum and orthogonalized variants, the same dual-norm GNS could yield adaptive batch sizes for other preconditioned optimizers such as Adam, Shampoo, or SOAP—a direction the paper explicitly leaves open.
- The GNS formulas suggest a diagnostic: monitoring ||σ||_1 / ||g||_1 over training could flag when coordinate sign flips are dominated by noise, which may be useful for learning-rate tuning or detecting distribution shift.
- A direct test of the mechanism would be to compare adaptive batching against a well-tuned ramp schedule for each optimizer; if the GNS schedule only matches, rather than beats, a tuned heuristic, the practical gain is mainly in removing tuning cost, not in raw step savings.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes a non-Euclidean generalization of gradient noise scale (GNS) for stochastic generalized steepest descent. It derives an ℓ1 GNS for signSGD (Eq. 13) and a Schatten-1 GNS for specSGD (Eq. 17), together with a distributed variance estimator (Eqs. 20–21) and an adaptive batch-size scheduler (Algorithm 1). Experiments on Llama-160M/1B, SimpleViT/Imagewoof, and ResNet/CIFAR-10 report that the adaptive schedule matches or improves constant-batch baselines while reducing optimizer steps by up to 66.77% (Table 2). The appendix supplies convergence theorems (Theorems A.4–A.5) for the idealized signSGD/specSGD settings.
Significance. The dual-norm framework is a natural and potentially useful extension of the McCandlish et al. (2018) GNS formalism, and the distributed rank-based variance estimator is practical. The appendix lemmas and theorems are internally consistent given their assumptions, and the paper is transparent about the lower-bound nature of the derivation. However, the central empirical claim—that the non-Euclidean GNS metric, rather than the accompanying monotonic schedule and sqrt learning-rate scaling, drives the reported step reductions—is not isolated. Moreover, the theory covers plain signSGD/specSGD while the headline gains are for Signum/Muon, which the paper itself admits are not covered. The contribution is therefore significant if the empirical attribution can be established, but at present it remains conditional.
major comments (3)
- [§5.1 / Algorithm 1 / Tables 2–3] The central attribution claim is not experimentally isolated. Algorithm 1 changes three quantities simultaneously: the batch size (line 9), the learning rate via sqrt-batch scaling (line 10), and the monotonic/warm-up policy (lines 8–12). All baselines are constant-batch runs. There is no control with a blind monotonic schedule (e.g., doubling every F steps) using the same initial batch, warm-up, and sqrt LR scaling. Without such a control, the up-to-66% reductions could be due to the schedule heuristic rather than the geometry-aware GNS. The concern is amplified by per-task tuning of θ (θ=0.6 for language; swept 0.25–2 for vision; θ=0.3 for ℓ2 in Table 9) and by the paper's own statement in §3.3 that the CBS-to-GNS constant is loose.
- [§2.1 / §6 / Tables 2–3] The theory is derived for plain signSGD and specSGD, but the headline experiments use Signum and Muon, which add momentum and (for Muon) Newton–Schulz semi-orthogonalization. Section 6 explicitly states that extending the framework to stateful/preconditioned optimizers remains open. The remark that Signum/Muon reduce to signSGD/specSGD as special cases is not a proof that the noise-versus-batch-size relationship is invariant under momentum or preconditioning. Consequently, the empirical gains for Signum/Muon lack theoretical grounding; the revised paper should either provide such grounding, report headline results for the optimizers actually covered by the theory, or explicitly label the Signum/Muon results as heuristics.
- [§D.1 / Table 9] The ℓ1-versus-ℓ2 GNS comparison is confounded by different θ values: θ=0.6 for ℓ1 (fixed in Table 5) and θ=0.3 for ℓ2 ('after tuning'). Since θ determines the target noise-to-signal ratio and hence the batch-size trajectory, the comparison conflates metric geometry with schedule aggressiveness. A fair comparison should tune θ for both metrics or use a matched criterion. As presented, the table supports only the weaker statement that ℓ1 GNS at θ=0.6 outperforms ℓ2 GNS at θ=0.3 on this benchmark.
minor comments (5)
- [Eqs. (12), (16)] The phrase 'optimal learning rate' should be qualified as 'maximizer of the lower bound under the quadratic approximation', since the derivation in §3 maximizes a lower bound and omits the smoothness constant. The convergence theorems later include the smoothness constant, but the text should not present the earlier expressions as globally optimal.
- [Table 2] For AdamW the adaptive run ends at validation loss 3.3031, slightly worse than the B=64 baseline (3.2991), yet a 67.13% step reduction is reported. The definition of 'steps reduction' should be clarified: does it measure the step at which the adaptive run first reaches the baseline's minimum loss? This metric can overstate efficiency if the final loss is worse.
- [Theorem A.5, Eq. (30)] The expectation uses x_k instead of X_k in the left-hand side; the variable is a matrix. Please fix the typo.
- [Algorithm 1] It is not stated how the real-valued target batch size from line 9 is rounded to an integer that is a multiple of the number of ranks/GPUs. This detail matters for reproducibility in distributed training.
- [Figure 1b] The caption is too terse. The geometric sketch should be explained in the text: what is the 'allowable region', and why does the non-Euclidean norm improve sign alignment for signSGD?
Circularity Check
No significant circularity: non-Euclidean GNS is derived from explicit bounds; the one self-citation is not load-bearing.
full rationale
The central derivation is self-contained. The l1 GNS (Eq. 13) and S1 GNS (Eq. 17) are obtained by combining the descent inequality (Eq. 9) with external, cited error bounds for signSGD and specSGD, then maximizing the resulting lower bound. The CBS-to-GNS proportionality in Eq. (19) is an algebraic consequence of the paper's own improvement formula, not a fitted quantity masquerading as a prediction; the paper explicitly states that the theoretical constant is loose and that theta is a tunable hyperparameter (Section 3.3, Appendix C). The experiments report measured step reductions rather than predictions from a fitted parameter. The one self-citation (Sato et al., 2025, co-authored by Naganuma) appears only as a background remark about prior CBS analysis of Muon and is not load-bearing for the new derivation. The acknowledged gap between the signSGD/specSGD theory and the Signum/Muon experiments (Section 6) is a validity limitation, not a circular structure. Likewise, the absence of a blind batch-schedule control is an empirical confounding concern, not an instance of a claim reducing to its own inputs. No circular step meets the required quote-and-reduction bar.
Assumptions & free parameters
free parameters (4)
- tolerance θ =
0.6 (LM); swept {0.25,0.5,1,2} for vision; 0.3 for Euclidean ℓ2 baseline
- EMA coefficients β_N, β_M =
0.9, 0.9
- update frequency F =
100 iterations (LM), 1-3 epochs (vision)
- optimizer learning rates and momenta =
per-optimizer sweep (Tables 5-7)
assumptions (6)
- domain assumption The loss is well approximated by a quadratic with identity Hessian in the update direction (Eqs. 3 and 8, following McCandlish et al. 2018).
- standard math Gradient estimates are unbiased, samples are i.i.d., and the noise bounds E∥∇L−g∥_1 ≤ ∥σ∥_1/√B (Lemma A.2) and the nuclear-norm analog (Lemma A.3) hold.
- domain assumption Objective is ℓ∞-smooth (or spectral-smooth) for the convergence theorems (Theorems A.4, A.5).
- ad hoc to paper Signum and Muon can be treated as signSGD/specSGD for GNS purposes; momentum/preconditioning does not change the noise-vs-batch relationship.
- domain assumption Distributed local mini-batch gradients across ranks are independent samples from the gradient distribution (Eqs. 20-21).
- ad hoc to paper Monotonic batch-size increase and warm-up prevent instability (Algorithm 1).
Cite this review
Pith. "Pith review of Adaptive Batch Sizes Using Non-Euclidean Gradient Noise Scales for Stochastic Sign and Spectral Descent." pith.science (2026). https://pith.science/paper/XGCUEABZ
@misc{pith2026260203001,
author = {Pith},
title = {Pith review of: Adaptive Batch Sizes Using Non-Euclidean Gradient Noise Scales for Stochastic Sign and Spectral Descent},
year = {2026},
howpublished = {\url{https://pith.science/paper/XGCUEABZ}},
note = {Machine review of arXiv:2602.03001}
}
abstract
To maximize hardware utilization, modern machine learning systems typically employ large constant or manually tuned batch size schedules, relying on heuristics that are brittle and costly to tune. Existing adaptive strategies based on gradient noise scale (GNS) offer a principled alternative. However, their assumption of SGD's Euclidean geometry creates a fundamental mismatch with popular optimizers based on generalized norms, such as signSGD / Signum ($\ell_\infty$) and stochastic spectral descent (specSGD) / Muon ($\mathcal{S}_\infty$). In this work, we derive gradient noise scales for signSGD and specSGD that naturally emerge from the geometry of their respective dual norms. To practically estimate these non-Euclidean metrics, we propose an efficient variance estimation procedure that leverages the local mini-batch gradients on different ranks in distributed data-parallel systems. Our experiments demonstrate that adaptive batch size strategies using non-Euclidean GNS enable us to match the validation loss of constant-batch baselines while reducing training steps by up to 66\% for Signum and Muon on a 160 million parameter Llama model.
Figures
Forward citations
Cited by 1 Pith paper
-
Convergence Bound and Critical Batch Size of Muon Optimizer
Muon's four momentum and weight-decay variants are proven to converge with bounded gradient norms under weight decay, and a lower-bound formula links critical batch size to momentum and weight decay.
Reference graph
Works this paper leans on
-
[1]
If the learning rate is chosen asη k =η= 1√L∞K , E " 1 K K−1X k=0 ∥∇L(xk)∥1 # ≤ √L∞ (1−θ) √ K L(x0)− L∗ + 1 2 ,(25) and{x k}converges to the neighborhood of a stationary point in expectation at a sublinear rate
-
[2]
If the learning rate is chosen such that P∞ k=0 ηk =∞and P∞ k=0 η2 k <∞, lim inf k→∞ E[∥∇L(xk)∥1] = 0.(26)
-
[3]
K−1X k=0 ηk ∥∇L(xk)∥1 # − L∞ 2 K−1X k=0 η2 k, where rearranging the in above using1−θ >0yields, E
If the objective function is alsoµ ∞−strongly convex, i.e., 2µ∞ (L(x)− L∗)≤ ∥∇L(x)∥2 1 ∀x∈R d, withµ ∞ >0and the learning rate is chosen asη k = (1−θ)∥∇L(xk)∥1 L∞ , then E[L(x k+1)]− L∗ ≤ 1−(1−θ) 2 µ∞ L∞ (E[L(x k)]− L∗)(27) and{x k}converges to a unique optimal point in expectation at a linear rate. Proof. Under the stated smoothness assumption, the chang...
2020
-
[5]
14 Adaptive Batch Sizes Using Non-Euclidean Gradient Noise Scales for Stochastic Sign and Spectral Descent
If the learning rate is chosen asη k =η= 1√ LS∞ K , E " 1 K K−1X k=0 ∥∇L(xk)∥S1 # ≤ p LS∞ (1−θ) √ K L(x0)− L∗ + 1 2 ,(30) and{X k}converges to the neighborhood of a stationary point in expectation at a sublinear rate. 14 Adaptive Batch Sizes Using Non-Euclidean Gradient Noise Scales for Stochastic Sign and Spectral Descent
-
[6]
If the learning rate is chosen such that P∞ k=0 ηk =∞and P∞ k=0 η2 k <∞, lim inf k→∞ E[∥∇L(Xk)∥S∞ ] = 0.(31)
-
[7]
K−1X k=0 ηk ∥∇L(Xk)∥S1 # ≤ 1 (1−θ)
If the objective function is alsoµ S∞ −strongly convex, i.e., 2µS∞ (L(X)− L∗)≤ ∥∇L(X)∥2 S1 ∀X∈R m×n, withµ S∞ >0and the learning rate is chosen asη k = (1−θ)∥∇L(Xk)∥S1 LS∞ , then E[L(X k+1)]− L∗ ≤ 1−(1−θ) 2 µS∞ LS∞ (E[L(X k)]− L∗)(32) and{X k}converges to a unique optimal point in expectation at a linear rate. Proof.Under the stated smoothness assumption,...
2018
-
[2019]
Jordan, K., Jin, Y ., Boza, V ., Jiacheng, Y ., Ce- sista, F., Newhouse, L., and Bernstein, J
URL https://github.com/fastai/ imagenette#imagewoof. Jordan, K., Jin, Y ., Boza, V ., Jiacheng, Y ., Ce- sista, F., Newhouse, L., and Bernstein, J. Muon: An optimizer for hidden layers in neural net- works. https://kellerjordan.github.io/ posts/muon/, 2024. Accessed: 2025-7-3. Karimireddy, S. P., Rebjock, Q., Stich, S., and Jaggi, M. Error feedback fixes ...
arXiv 2024
-
[2022]
Second, we train ResNet-18 over the CIFAR-10 dataset to cover CNNs with residual connections
over the Imagewoof dataset (Howard, 2019) which isolates the transformer setting in a vision task. Second, we train ResNet-18 over the CIFAR-10 dataset to cover CNNs with residual connections. Across both workloads, we compare constant batch training to the adaptive strategy in Algorithm 1. For ResNet-18 over the CIFAR-10, we train for 100 epochs, and for...
arXiv 2019
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.