{"id":"55ba66ee-9247-4bb5-8e40-a8f34309eea3","arxiv_id":"2602.03001","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Adaptive batch sizes driven by non-Euclidean gradient noise scales reduce training steps of Signum, Muon, signSGD, and specSGD while matching validation loss.","lead":"This paper derives batch-size scheduling rules from noise metrics tailored to sign-based and spectral optimizers, then tests them on language and vision models. The method matches constant-batch validation loss while cutting optimizer steps by up to roughly two-thirds for Signum and Muon on a 160M-parameter language model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing no-GNS control: the 66% step reductions for Signum/Muon may be due to monotonic batch growth + sqrt LR scaling, not the non-Euclidean GNS metric.","rationale":"The reader's weakest assumption—theory for plain signSGD/specSGD transferred to Signum/Muon—is a genuine gap and is explicitly conceded in Section 6. I did not make it the headline because even for the optimizers the theory does cover (signSGD, specSGD), the experiments would not establish the causal role of the GNS: Algorithm 1 confounds the GNS batch rule with monotonic batch growth and sqrt-LR scaling. Table 8 shows LR scaling alone changes validation loss and step counts for signSGD, so its effect must be separated. Table 9 compares ℓ1 vs ℓ2 GNS but uses different tuned θ values (ℓ2 θ=0.3 vs ℓ1 θ=0.6), so it is not a matched comparison. The theory itself is a lower-bound derivation whose constant is acknowledged loose (Section 3.3), meaning a tuned θ can absorb miscalibration. A blind schedule control is the single cheapest experiment that would determine whether the non-Euclidean metric, as opposed to the schedule scaffolding, is responsible for the 66% reductions. The paper is otherwise careful: the variance estimator is unbiased, the convergence theorems are plausible, and the core GNS formulas follow from the stated dual-norm arguments. Those strengths make the missing control the decisive gap rather than a fatal flaw.","tokens_in":22361,"tokens_out":12751,"duration_ms":140723,"concrete_test":"Re-run the 160M Llama 3 Signum experiment (Table 2) with Algorithm 1 modified so line 9 sets B_{k+1} = 2*B_k every F=1000 steps (or any fixed doubling schedule that consumes the same 3.2B token budget), keeping warmup, initial B=64, monotonicity, and the line-10 sqrt learning-rate scaling; tune nothing. Compare median validation loss and median steps-to-baseline to the GNS-adaptive run over 10 seeds. If the blind schedule matches the GNS run, the metric is not load-bearing; if it is significantly worse, the attribution is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Headline experiments (Tables 2–3, Fig. 2) claim that non-Euclidean GNS drives the step reductions, but Algorithm 1 changes three things at once: monotonic batch growth (line 9), learning-rate scaling with sqrt batch ratio (line 10), and the GNS estimate that sets the growth rate. The baselines are constant-batch runs only. There is no control where batch size follows a blind schedule (e.g., doubling every F steps) with the same warmup, initial batch, and sqrt LR scaling. Without that control, the reductions are not attributable to the geometry-aware GNS; a tuned, monotonic batch ramp could produce the same numbers. This is especially acute because θ is tuned per task/optimizer (θ=0.6 for LLMs; swept 0.25–2 for vision; Table 9 uses θ=0.3 for ℓ2), and the paper concedes the theoretical constant linking GNS to CBS is loose (Section 3.3) and the extension to stateful optimizers like Signum/Muon is open (Section 6). The stateful-optimizer gap alone would not sink the empirical claim, but the missing causal control means the central claim—that the non-Euclidean metric, not just a schedule heuristic, is what enables the gains—is unverified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a non-Euclidean generalization of gradient noise scale (GNS) for stochastic generalized steepest descent. It derives an ℓ1 GNS for signSGD (Eq. 13) and a Schatten-1 GNS for specSGD (Eq. 17), together with a distributed variance estimator (Eqs. 20–21) and an adaptive batch-size scheduler (Algorithm 1). Experiments on Llama-160M/1B, SimpleViT/Imagewoof, and ResNet/CIFAR-10 report that the adaptive schedule matches or improves constant-batch baselines while reducing optimizer steps by up to 66.77% (Table 2). The appendix supplies convergence theorems (Theorems A.4–A.5) for the idealized signSGD/specSGD settings.","tokens_in":22757,"tokens_out":8038,"duration_ms":84952,"significance":"The dual-norm framework is a natural and potentially useful extension of the McCandlish et al. (2018) GNS formalism, and the distributed rank-based variance estimator is practical. The appendix lemmas and theorems are internally consistent given their assumptions, and the paper is transparent about the lower-bound nature of the derivation. However, the central empirical claim—that the non-Euclidean GNS metric, rather than the accompanying monotonic schedule and sqrt learning-rate scaling, drives the reported step reductions—is not isolated. Moreover, the theory covers plain signSGD/specSGD while the headline gains are for Signum/Muon, which the paper itself admits are not covered. The contribution is therefore significant if the empirical attribution can be established, but at present it remains conditional.","major_comments":[{"comment":"The central attribution claim is not experimentally isolated. Algorithm 1 changes three quantities simultaneously: the batch size (line 9), the learning rate via sqrt-batch scaling (line 10), and the monotonic/warm-up policy (lines 8–12). All baselines are constant-batch runs. There is no control with a blind monotonic schedule (e.g., doubling every F steps) using the same initial batch, warm-up, and sqrt LR scaling. Without such a control, the up-to-66% reductions could be due to the schedule heuristic rather than the geometry-aware GNS. The concern is amplified by per-task tuning of θ (θ=0.6 for language; swept 0.25–2 for vision; θ=0.3 for ℓ2 in Table 9) and by the paper's own statement in §3.3 that the CBS-to-GNS constant is loose.","section":"§5.1 / Algorithm 1 / Tables 2–3"},{"comment":"The theory is derived for plain signSGD and specSGD, but the headline experiments use Signum and Muon, which add momentum and (for Muon) Newton–Schulz semi-orthogonalization. Section 6 explicitly states that extending the framework to stateful/preconditioned optimizers remains open. The remark that Signum/Muon reduce to signSGD/specSGD as special cases is not a proof that the noise-versus-batch-size relationship is invariant under momentum or preconditioning. Consequently, the empirical gains for Signum/Muon lack theoretical grounding; the revised paper should either provide such grounding, report headline results for the optimizers actually covered by the theory, or explicitly label the Signum/Muon results as heuristics.","section":"§2.1 / §6 / Tables 2–3"},{"comment":"The ℓ1-versus-ℓ2 GNS comparison is confounded by different θ values: θ=0.6 for ℓ1 (fixed in Table 5) and θ=0.3 for ℓ2 ('after tuning'). Since θ determines the target noise-to-signal ratio and hence the batch-size trajectory, the comparison conflates metric geometry with schedule aggressiveness. A fair comparison should tune θ for both metrics or use a matched criterion. As presented, the table supports only the weaker statement that ℓ1 GNS at θ=0.6 outperforms ℓ2 GNS at θ=0.3 on this benchmark.","section":"§D.1 / Table 9"}],"minor_comments":[{"comment":"The phrase 'optimal learning rate' should be qualified as 'maximizer of the lower bound under the quadratic approximation', since the derivation in §3 maximizes a lower bound and omits the smoothness constant. The convergence theorems later include the smoothness constant, but the text should not present the earlier expressions as globally optimal.","section":"Eqs. (12), (16)"},{"comment":"For AdamW the adaptive run ends at validation loss 3.3031, slightly worse than the B=64 baseline (3.2991), yet a 67.13% step reduction is reported. The definition of 'steps reduction' should be clarified: does it measure the step at which the adaptive run first reaches the baseline's minimum loss? This metric can overstate efficiency if the final loss is worse.","section":"Table 2"},{"comment":"The expectation uses x_k instead of X_k in the left-hand side; the variable is a matrix. Please fix the typo.","section":"Theorem A.5, Eq. (30)"},{"comment":"It is not stated how the real-valued target batch size from line 9 is rounded to an integer that is a multiple of the number of ranks/GPUs. This detail matters for reproducibility in distributed training.","section":"Algorithm 1"},{"comment":"The caption is too terse. The geometric sketch should be explained in the text: what is the 'allowable region', and why does the non-Euclidean norm improve sign alignment for signSGD?","section":"Figure 1b"}],"recommendation":"major_revision","confidential_remarks":"The main reason for major revision is the missing causal control, not the lower-bound derivation (which the authors acknowledge) or the difference from current consensus. The theory is sound in its limited scope, and the practical estimator is a genuine strength. I would be satisfied if the revision adds (a) a blind-schedule control, (b) a careful statement that Signum/Muon results are heuristic, or a theoretical extension, and (c) a fairer ℓ1-vs-ℓ2 comparison. These are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new part is the ℓ1 and S1 gradient noise scales and the distributed rank-based estimator. The derivation via dual norms is the right way to extend McCandlish, the math in Lemmas A.1–A.3 and Theorems A.4–A.5 is internally consistent, and the paper is honest that the CBS-to-GNS constant is loose and that stateful optimizers like Signum/Muon remain an open extension. That honesty is real credit.\n\nWhat the paper does well: it gives a principled, cheap way to estimate non-Euclidean noise in DDP/FSDP, and the lower-bound analysis correctly shows that the natural noise scale for signSGD is an ℓ1 quantity and for specSGD a nuclear-norm quantity. Those formulas are the contribution.\n\nWhere it gets soft. First, the headline 66% step reductions come from Algorithm 1, which changes three things at once: monotonic batch growth, square-root learning-rate scaling, and the GNS-driven growth rate. The baselines are constant-batch only. There is no control with a blind schedule — say, doubling the batch every F steps with the same warmup and LR scaling. Without that, you cannot attribute the gains to the geometry-aware GNS rather than to a generic batch ramp. Table 9's ℓ1-vs-ℓ2 comparison helps for signSGD, but not for the Signum/Muon numbers in Tables 2–3. Second, the theory covers plain signSGD/specSGD, not the momentum/preconditioned variants used in the headline experiments; the \"reduce to special cases\" line is approximate and the paper concedes this. Third, θ is tuned separately per metric (0.6 for ℓ1, 0.3 for ℓ2), which weakens the comparison, and some key results lack error bars or released code.\n\nNone of this sinks the theory. But the empirical claim is overstated as written. A serious referee should ask for a blind-schedule control, a clearer separation of the heuristic extension from the proven part, and ideally code/data.\n\nWho is this for: people working on adaptive batching, GNS, sign/spectral optimizers, or distributed training. I'd bring it to a reading group and I'd likely cite the ℓ1/S1 formulas. It deserves peer review — the theory is a real contribution, and the empirical question is important enough to warrant the extra experiments.","headline":"Non-Euclidean GNS theory is clean and worth taking seriously; the headline step-reduction claim for Signum/Muon is not yet supported by the experiments.","tokens_in":23229,"tokens_out":4204,"would_cite":true,"duration_ms":44472,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper derives gradient noise scales for sign-based and spectral optimizers from the dual norms of their geometries, and shows that adaptive batch sizes using these metrics match constant-batch validation loss while reducing training st","keywords":["gradient noise scale","adaptive batch size","signSGD","Signum","spectral descent","Muon","dual norm","distributed training"],"falsifier":"Measure, for Signum or Muon on a fixed model, the optimal batch size at several training steps by scanning B and computing per-sample progress; if the maximizing batch size does not track θ^{-2} times the ℓ1 (or S1) GNS across training, the transfer from the signSGD/specSGD theory to stateful optimizers fails.","tokens_in":1415,"feed_emoji":"⚡","tokens_out":2193,"duration_ms":60531,"temperature":0.7,"pith_summary":"This paper argues that the standard gradient noise scale (GNS), which guides adaptive batch sizing for SGD, is the wrong ruler for optimizers that update in non-Euclidean geometries. For sign-based methods like signSGD and Signum the right ruler is the ℓ1 norm of the gradient error; for spectral methods like specSGD and Muon it is the nuclear (Schatten-1) norm. The authors derive GNS formulas in those dual norms, prove convergence for the resulting adaptive schedule, and design a distributed variance estimator that reuses local gradient statistics already computed by data-parallel training. On a 160M Llama 3 model, their adaptive batching matches the validation loss of constant-batch baselines while cutting optimizer steps by up to 66% for Signum and Muon.","feed_headline":"Dual-norm noise metric cuts sign-optimizer training steps by 66%","feed_subtitle":"Adaptive batch sizes tuned to ℓ1 and nuclear-norm noise match validation loss on a 160M Llama model.","key_machinery":"The central object is the dual-norm gradient noise scale, a ratio of two dual norms: the expected gradient error (component-wise standard deviation for signSGD; row-wise covariance nuclear norm for specSGD) over the true gradient norm, both measured in the optimizer's dual norm (ℓ1 for sign, Schatten-1 for spectral). Setting the batch size to θ^{-2} times this ratio keeps the noise-to-signal ratio constant. Supporting machinery includes Lemma 3.1 (a Hölder-based lower bound on progress), convergence theorems in the appendix, and a distributed estimator that uses per-rank mini-batch gradients to compute the variance statistics without extra data passes.","core_discovery":"For stochastic steepest descent under a general norm, the expected inner product of the true gradient with the stochastic descent direction is bounded below by the dual norm of the true gradient minus the expected dual norm of the gradient error (Lemma 3.1). The bias introduced by sign and spectral updates therefore scales with the dual norm of the noise, and the critical batch size becomes the squared ratio of noise to signal in that dual norm: B_ℓ1 = ||σ||_1² / ||∇L||_1² for signSGD and B_S1 = ||C_row^{1/2}||_{S1}² / ||∇L||_{S1}² for specSGD. The paper proves convergence for the resulting adaptive schedule and validates it empirically, reporting up to 66.77% step reduction for Muon on the","pith_inferences":["If the noise scale transfers from plain signSGD/specSGD to momentum and orthogonalized variants, the same dual-norm GNS could yield adaptive batch sizes for other preconditioned optimizers such as Adam, Shampoo, or SOAP—a direction the paper explicitly leaves open.","The GNS formulas suggest a diagnostic: monitoring ||σ||_1 / ||g||_1 over training could flag when coordinate sign flips are dominated by noise, which may be useful for learning-rate tuning or detecting distribution shift.","A direct test of the mechanism would be to compare adaptive batching against a well-tuned ramp schedule for each optimizer; if the GNS schedule only matches, rather than beats, a tuned heuristic, the practical gain is mainly in removing tuning cost, not in raw step savings."],"forward_implications":["Adaptive batch schedules for signSGD, Signum, specSGD, and Muon can be set from a single scalar per step instead of hand-tuned ramp schedules.","Step-count savings can be large for momentum and orthogonalized variants: up to 66.61% for Signum and 66.77% for Muon on the 160M Llama model, with no loss in validation loss.","The critical batch size is always a constant multiple of the corresponding GNS under all three turning-point definitions, so tuning θ is equivalent to choosing an operating point on a fixed saturation curve.","The distributed variance estimator gives unbiased population noise estimates using existing AllReduce or ReduceScatter traffic, adding little extra memory or compute.","Convergence holds with the batch-size rule: sublinear in the non-strongly-convex setting and linear in the strongly-convex setting, with the (1−θ)² factor controlling the noise term."],"fun_headline_variants":["Non-Euclidean noise scales adapt batch sizes to cut steps 66%","Dual-norm noise metric adapts batch size, matching loss with fewer steps","Adaptive batch sizes via dual-norm noise cut training steps by 66%","Non-Euclidean noise scales for sign/spectral optimizers: 66% fewer steps","Batch size tuning with non-Euclidean noise scales: 66% step cut"],"cache_read_input_tokens":24448,"weakest_assumption_plain":"The load-bearing premise is that the GNS measured on raw mini-batch gradients still governs the optimal batch size after the optimizer adds momentum and semi-orthogonalization, since the theory is derived for plain signSGD and specSGD while the headline results use Signum and Muon.","fun_headline_variants_meta":{"raw":{"variants":["Non-Euclidean noise scales adapt batch sizes to cut steps 66%","Dual-norm noise metric adapts batch size, matching loss with fewer steps","Adaptive batch sizes via dual-norm noise cut training steps by 66%","Non-Euclidean noise scales for sign/spectral optimizers: 66% fewer steps","Batch size tuning with non-Euclidean noise scales: 66% step cut"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001166,"raw_usage":{"total_tokens":4676,"prompt_tokens":770,"completion_tokens":3906,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":3798}},"tokens_in":514,"tokens_out":3906,"duration_ms":26004,"temperature":1.0,"reasoning_tokens":3798,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T05:08:23.503218+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure, for Signum or Muon on a fixed model, the optimal batch size at several training steps by scanning B and computing per-sample progress; if the maximizing batch size does not track θ^{-2} times the ℓ1 (or S1) GNS across training, the transfer from the signSGD/specSGD theory to stateful optimizers fails.","supporting_citations":[],"review_version":1}