Pith. sign in

REVIEW 3 major objections 6 minor 17 references

Hyperball-style optimizers are best understood as implicit learning-rate schedulers, not providers of a superior update direction; the paper's angular effective learning rate explains their slow-then-fast behavior.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 04:42 UTC pith:7JESJFX7

load-bearing objection Useful, honest paper with a clean angular effective LR and a plausible scheduler interpretation, but the negative claim about update direction outruns the evidence. the 3 major comments →

arxiv 2607.22444 v1 pith:7JESJFX7 submitted 2026-07-24 cs.LG cs.AI

Hyperball May Not Be a Free Lunch

Yihao Xiao , Jialong Sun , Zitian Gao , Zeming Wei , Chutian Wang , Ran Tao , Jiaye Teng , Bryan Dai This is my paper
classification cs.LG cs.AI
keywords Hyperball optimizationangular effective learning ratescale-invariant networksimplicit learning-rate schedulerMuon optimizerradial updateslearning-rate scheduling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to pin down why Hyperball-style optimizers, which fix parameter norms and normalize updates, show slow-then-fast training behavior in scale-invariant networks. It derives an angular effective learning rate that measures how far a single update rotates the parameter direction, and shows that the standard norm-based measure is only valid when parameter and update are orthogonal. Controlled experiments indicate that the radial part of the update has little direct effect on this angular rate, so it cannot explain the phase transition. Instead, aligning the angular effective learning rates of MuonH and MuonWD by per-step learning-rate changes makes their validation-loss trajectories largely reproduce one another. The paper concludes that Hyperball's main effect is an implicit state-dependent learning-rate schedule, not a superior update direction, and that careful scheduling remains necessary.

Core claim

The central claim is that the apparent advantage of Hyperball-style optimizers in scale-invariant deep networks is not an intrinsically better update direction but an altered effective step size. Starting from the angular displacement between consecutive parameter states, the authors define an angular effective learning rate that depends on the learning rate, parameter norm, update norm, and the cosine similarity between parameter and update. They show the usual norm-based effective learning rate is a special case under orthogonality. They then decompose updates into radial and tangential components and find, in the configurations studied, that the radial component only weakly affects the on

What carries the argument

The central object is the angular effective learning rate, defined from the small-angle approximation of the angular displacement Δφ_t between consecutive parameter states: η^φ_eff,t ≈ η_t sqrt(1-c_t^2)/(α_t ||W_t||_F - η_t c_t ||U_t||_F), where c_t is the cosine similarity between the parameter W_t and the optimizer update U_t. It simultaneously accounts for the learning rate, parameter norm, update norm, and parameter–update angle; when c_t = 0 it reduces to the conventional norm-based rate. This scalar is load-bearing because it is used both to attribute the evolution of effective step size to different factors and to construct the per-step learning-rate alignment procedure that transform

Load-bearing premise

The load-bearing premise is that the angular effective learning rate is a sufficient summary of what differs between the two optimizers; if the real difference lives in update direction, or if the small-angle and scale-invariance approximations fail, the scheduler conclusion collapses.

What would settle it

A training run where MuonH and MuonWD are matched in angular effective learning rate at every step and their validation-loss trajectories nevertheless diverge would falsify the scheduler claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If Hyperball's main effect is an implicit learning-rate schedule, then fixed-norm constraints alone do not promise faster or better convergence; the schedule they induce must be deliberately managed.
  • The angular effective learning rate generalizes norm-based effective-learning-rate analysis, so future analyses of optimizers with momentum, preconditioning, or matrix constraints should track the parameter–update angle rather than norms alone.
  • Because Hyperball keeps the effective learning rate proportional to the scheduled learning rate, Hyperball-style optimizers can serve as a cleaner testbed for studying learning-rate schedules without the confounding coupling to weight decay.
  • More aggressive early learning-rate decay can accelerate MuonH's early convergence but can hurt later performance, so scaling studies using Hyperball should include a schedule search rather than reuse conventional schedules.
  • The mutual alignment of MuonH and MuonWD by matching angular effective learning rates provides a direct experimental method for separating step-size effects from update-direction effects in other optimizer comparisons.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same alignment procedure could be applied to AdamW versus AdamH or to mixture-of-experts training; if the scheduler explanation generalizes, matched effective learning rates should again collapse the loss trajectories.
  • If Hyperball is essentially a state-dependent schedule, then 'schedule-free' methods or warmup-stable-decay recipes may need to be re-derived for Hyperball rather than transferred from conventional optimizers.
  • A testable extension is to use the angular effective learning rate as a target for closed-loop schedulers that actively maintain a desired angular velocity, and check whether that removes the residual phase-dependent gap.
  • The paper's finding suggests that reported gains from Hyperball-style optimizers could partly reflect an accidental annealing effect; comparing optimizers at matched angular effective learning rates would separate accidental scheduling from genuine update-direction advantages.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies why Hyperball-style constrained optimizers (e.g., MuonH) exhibit a slow-then-fast training dynamic relative to non-Hyperball counterparts (e.g., MuonWD). Starting from a generic update rule, the authors derive an angular effective learning rate, η^φ_eff, that depends on the learning rate, parameter norm, update norm, and parameter–update cosine c_t (Eq. 7), recovering the conventional norm-based effective learning rate when c_t = 0 (Eq. 8). For Hyperball updates, they derive a reduced expression depending only on η_t and c_t (Eq. 12). They then decompose updates into radial and tangential components (Eq. 13) and use a log-derivative analysis (Eq. 15) with numerical experiments (Fig. 1) to argue that the radial component's direct effect on the angular effective learning rate is limited. They propose a state-dependent learning-rate alignment procedure (Sec. 4.2, Eq. 19) that matches η^φ_eff between MuonH and MuonWD, and report that this substantially reproduces the other optimizer's validation-loss trajectory. They also test alternative learning-rate schedules for MuonH (Sec. 4.3) and conclude that Hyperball's phase-dependent behavior stems from the evolution of effective step size rather than an intrinsically superior update direction, calling the scheduler interpretation a 'working hypothesis.' The manuscript includes public code.

Significance. If the central claim holds, the paper provides a useful conceptual reframing: Hyperball-style optimizers may act primarily as implicit learning-rate schedulers, rather than as methods that improve the update direction. The angular effective learning rate is a clean, parameter-free object, and the derivation of Eq. (12) is a genuine contribution that unifies and extends norm-based effective-learning-rate analyses. The paper is also honest about its limitations and explicitly labels the scheduler interpretation as a working hypothesis. However, the main negative claim—'rather than an intrinsically superior update direction'—is not directly tested: the alignment experiment matches only a scalar angular magnitude and discards the tangential orientation of the update. The numerical sensitivity analysis in Sec. 3 also lacks formal guarantees and error bars, and the abstract overstates the strength of the evidence. These issues are load-bearing for the paper's central conclusion, so the manuscript needs revision before publication.

major comments (3)
  1. [§4.2, Eq. (19), Abstract] The alignment experiment matches only the scalar η^φ_eff, which by Eq. (7) depends solely on η_t, ||U_t||_F, c_t, and ||W_t||_F. It therefore discards the orientation of the tangential component of U_t. If Hyperball's benefit came from rotating the update within the tangent space—through orthogonalization, momentum structure, or preconditioning—η^φ_eff would be unchanged and the Sec. 4.2 procedure would never detect the difference. The conclusion in the abstract that the behavior stems from effective step size 'rather than an intrinsically superior update direction' is a claim about a quantity that is never measured. I recommend adding a direct direction-similarity check, e.g., the cosine between the normalized tangential components of MuonH and MuonWD updates (or an ablation that randomizes tangential orientation while preserving c_t and norms), and reporting whether the alignment still
  2. [§3.1–3.2, Eq. (15), Fig. 1, Abstract] The manuscript's own §3.1 states that a purely radial update can affect future optimization indirectly through changes in ||W_t||_F, gradient scale, and optimizer state, and that the analysis 'does not rule out longer-term indirect effects.' Yet the abstract and conclusion say the radial component 'therefore cannot explain why MuonH converges more slowly...' This is a logical leap: the numerical analysis only bounds the direct one-step contribution (via Fig. 1 and R_j), not the indirect path through parameter-norm dynamics. Moreover, Fig. 1 appears to be based on single runs with no error bars, and the metric R_j = 100(1 − η^(j)/η^(j−1)) is not fully defined regarding how the counterfactual η^(j−1) is computed and how negative contributions are handled. Please either provide multi-seed confidence intervals and a precise definition, or explicitly scope the conclusion to the direct effect.
  3. [§4.3, Fig. 4] The pretraining experiments introduce two hand-designed schedules, 'MuonH-Z' (square-root decay) and 'MuonH-Ours' (power-0.4 decay), with free exponents (p and the sqrt exponent). The claim that 'more aggressive learning-rate decay can accelerate MuonH early but may impair its later performance' is based on these specific schedules, and Fig. 4 again shows single runs without error bars. Since the paper's practical message is that Hyperball requires careful scheduling and that there is 'no direct evidence' of consistent improvement, the sensitivity to the schedule exponent and to random seeds should be reported. A small grid over p (e.g., 0.2, 0.4, 0.6) and at least three seeds per configuration would substantially strengthen the empirical basis. The current presentation makes it difficult to distinguish a robust phenomenon from schedule-specific overfitting.
minor comments (6)
  1. [§2.3, Eq. (7)] The definition η^φ_eff,t ≜ Δφ_t / ||U_t||_F has units of inverse norm, while the approximation on the right-hand side has different scaling. Please clarify that the division by ||U_t||_F is a deliberate normalization choice and that the small-angle approximation tan Δφ ≈ Δφ is what makes the two sides commensurate.
  2. [§3.2, Eq. (15)] The derivative notation ¤η_t, ¤c_t, etc., is introduced without formally defining the continuous-time interpolation of the discrete step index t. Please specify how the discrete-time quantities are extended to continuous time and state any smoothness assumptions.
  3. [Fig. 1] The caption says 'The decline in learning rate caused by different factors,' but the metric R_j is expressed as a percentage of relative change. Please spell out that R_j measures the instantaneous contribution of each term to the decline in η^φ_eff and indicate how the four contributions sum (or do not sum) to 100%.
  4. [§4, Eqs. (16)–(17)] Equations (16) and (17) are rendered with garbled line breaks and extraneous whitespace in the manuscript, making the derivation hard to follow. Please rewrite these equations in a cleanly formatted form.
  5. [Fig. 4] The legend in Fig. 4(a) and (c) uses 'Muon' while the text discusses 'MuonWD.' Please unify the notation so that the baseline is unambiguously named.
  6. [§5.1] The limitations paragraph already concedes that the theoretical analysis is limited to numerical sensitivity and that the river-valley landscape is interpretive. I would recommend moving some of this caution into the abstract, where the claim is currently stated more definitively than the evidence supports.

Circularity Check

0 steps flagged

No circularity: the angular effective LR is derived from the update rule, and the alignment experiment checks an independent observable rather than re-labeling a fitted quantity as a prediction.

full rationale

The paper's derivation chain is self-contained. Proposition 1 and Eq. (7) follow from an exact trigonometric decomposition of the generic update rule W_{t+1} = α_t W_t − η_t U_t, with no fitted constants. Equation (12) is obtained by substituting the Hyperball normalization and projection into the same formula, and Eq. (8) is the c_t = 0 special case. The radial/tangential decomposition in Section 3 is also algebra from the same update rule, and the numerical sensitivity claim is an empirical measurement rather than a re-labeling of inputs. The Section 4.2 alignment experiment does not fit a parameter and then call it a prediction: it records the target angular effective LR from MuonH, solves Eq. (19) for a per-step LR of the other optimizer, and then compares the external observable—validation loss. Matching the scalar is by construction, but the validation-loss comparison is an independent check, and the conclusion is explicitly qualified as a 'working hypothesis' and as reproducing only 'part of' Hyperball's behavior. The caveat that η^φ_eff discards tangential orientation is a scope limitation, not a circular step, and the paper's limitations section acknowledges the need for further theoretical characterization. Citations to prior work are background or empirical support and are not used as the load-bearing derivation of the central result. No circular step was found.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

No new physical or algorithmic entities are introduced. The angular effective learning rate is a derived scalar measure, not an invented entity. The free parameters are all experimental schedule choices, not fitted constants in the derivation.

free parameters (2)
  • MuonH-Ours power exponent p = 0.4
    Hand-selected power-law decay schedule in Section 4.3, used to show early acceleration and later degradation. It is a search outcome on the same target validation loss, not a free prediction.
  • MuonH-Z square-root schedule exponent = minus square-root (PR #343)
    Baseline schedule imported from a GitHub PR; exact schedule parameters are not given in the text, so the numerical comparison depends on external PR specifics.
axioms (5)
  • domain assumption Scale invariance L(ρW)=L(W) and consequent gradient-parameter orthogonality
    Invoked in Section 2.2 (Eqs. 3-4); it underlies the classical norm-based effective LR and the c_t=0 special case. The paper admits in Section 5.1 that scale invariance does not strictly hold in practical training.
  • standard math Small-angle approximation tan(Δφ)≈Δφ
    Used to define the angular effective learning rate in Eq. (7). Reasonable for small single-step angles, but the paper does not verify it for early large steps.
  • domain assumption Positivity condition α_t||W_t|| - η_t c_t||U_t|| > 0
    Needed for Proposition 1 (Eq. 6) and for the angular effective LR to be finite. Assumed to hold along the training trajectories studied.
  • ad hoc to paper Angular effective learning rate is a sufficient scalar descriptor of optimizer dynamics
    The Section 4.2 alignment experiment matches only η^φ_eff and interprets the resulting loss similarity as evidence that effective step size is the main difference. The paper labels this a working hypothesis, not a proven equivalence.
  • domain assumption Continuous-step differentiability in the log-derivative decomposition
    Eq. (15) treats the discrete step index as a continuous variable and sums additive contributions from learning rate, c_t, parameter norm, and update norm. This is used in Fig. 1 to attribute radial effects as small.

pith-pipeline@v1.3.0-alltime-deepseek · 10487 in / 12574 out tokens · 145338 ms · 2026-08-01T04:42:34.165180+00:00 · methodology

0 comments
read the original abstract

For scale-invariant deep networks, Hyperball-style optimizers have shown strong performance in large-scale training by fixing the norms of matrix-valued parameters and normalizing updates. However, the source of their advantage remains unclear. Starting from the angular displacement between consecutive parameter states, we derive an angular effective learning rate that accounts for the parameter-update angle, parameter norm, and update norm. We also show that the conventional norm-based measure is a special case under parameter-update orthogonality. We then decompose optimizer updates into radial and tangential components and analyze how radial updates affect one-step angular displacement. Under the training configurations considered, numerical results show that the radial component has only a limited direct effect on the angular effective learning rate. It therefore cannot explain why MuonH converges more slowly than MuonWD early in training but overtakes it later. To further isolate the underlying mechanism, we devise a heuristic experiment that modifies only the learning-rate schedule so that the dynamics of each optimizer reproduce those of the other. The results suggest that their main difference stems from the evolution of the effective step size rather than an intrinsically superior update direction induced by Hyperball. Our pretraining experiments further show that more aggressive learning-rate decay can accelerate MuonH early in training but may impair its later performance. Thus, maintaining a constant angular velocity does not eliminate the learning-rate-scheduling problem; careful scheduling remains essential to realizing the potential of Hyperball-style optimizers. Our code is publicly available at https://github.com/mangocrazz/hyperball-may-not-be-a-free-lunch.

Figures

Figures reproduced from arXiv: 2607.22444 by Bryan Dai, Chutian Wang, Jialong Sun, Jiaye Teng, Ran Tao, Yihao Xiao, Zeming Wei, Zitian Gao.

Figure 1
Figure 1. Figure 1: The decline in learning rate caused by different factors. (a) MuonWD. (b) MuonH. [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Learning-rate schedules and the resulting effective learning rates. (a) Constant [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Mutual alignment of Hyperball and non-Hyperball dynamics. (a) Validation loss [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Validation loss and learning-rate schedules under different decay strategies. (a) [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

17 extracted references · 11 linked inside Pith

  1. [4]

    J. M. Cohen, A. Damian, A. Talwalkar, J. Z. Kolter, and J. D. Lee. Understanding optimization in deep learning with central flows.arXiv preprint arXiv:2410.24206,

    arXiv

  2. [6]

    S. Deng, Z. Ouyang, T. Pang, Z. Liu, R. Jin, S. Yu, and Y. Yang. Rmnp: Row-momentum normal- ized preconditioning for scalable matrix-based optimization.arXiv preprint arXiv:2603.20527,

    Pith/arXiv arXiv

  3. [8]

    URL https: //arxiv.org/abs/2510.19093. B. Li, Z. Wang, F. Chen, S. Zhao, R. Zheng, and L. Wu. Optimal learning-rate schedules under functional scaling laws: Power decay and warmup-stable-decay.arXiv preprint arXiv:2602.06797,

    arXiv

  4. [11]

    URLhttps://arxiv.org/abs/1804.04235. J. Su. Thinking about spectral norm gradient and spectral weight decay,

    Pith/arXiv arXiv

  5. [12]

    URL https: //kexue.fm/archives/10648. J. Su. Why Adam’s update RMS is 0.2?, 2025a. URLhttps://kexue.fm/archives/11267. J. Su. AdamW weight RMS asymptotics (part I), 2025b. URL https://kexue.fm/archives/ 11307. J. Su. AdamW weight RMS asymptotics (part II), 2025c. URL https://kexue.fm/archives/ 11404. T. Van Laarhoven. L2 regularization versus batch and we...

    Pith/arXiv arXiv

  6. [14]

    J. Wang, B. Li, Z. Zhou, M. Wang, Y. Sun, J. Zhang, X. Cai, and L. Wu. Fast catch-up, late switching: Optimal batch size scheduling via functional scaling laws.arXiv preprint arXiv:2602.14208,

    arXiv

  7. [15]

    K. Wen, Z. Li, J. Wang, D. Hall, P . Liang, and T. Ma. Understanding warmup-stable-decay learning rates: A river valley loss landscape view. InInternational Conference on Learning Representations, volume 2025, pages 42840–42885,

    2025

  8. [16]

    K. Wen, X. Dang, K. Lyu, T. Ma, and P . Liang. Fantastic pretraining optimizers and where to find them ii: Hyperball optimization.arXiv preprint arXiv:2606.16899,

    arXiv

  9. [17]

    T. Xie, H. Luo, H. Tang, Y. Hu, J. K. Liu, Q. Ren, Y. Wang, W. X. Zhao, R. Yan, B. Su, et al. Controlled llm training on spectral sphere.arXiv preprint arXiv:2601.08393,

    arXiv

  10. [18]

    G. Yuan. Optmuon: Closed-loop orthogonalized momentum methods for stochastic optimization with zero-noise optimality.arXiv preprint arXiv:2606.08783,

    Pith/arXiv arXiv

  11. [19]

    Zhang, C

    G. Zhang, C. Wang, B. Xu, and R. Grosse. Three mechanisms of weight decay regularization. arXiv preprint arXiv:1810.12281,

    Pith/arXiv arXiv

  12. [2016]

    URLhttps://arxiv.org/abs/1602.07868. 13 N. Shazeer and M. Stern. Adafactor: Adaptive learning rates with sublinear memory cost,

    Pith/arXiv arXiv

  13. [2017]

    R. Wan, Z. Zhu, X. Zhang, and J. Sun. Spherical motion dynamics: Learning dynamics of neural network with normalization, weight decay, and sgd.arXiv preprint arXiv:2006.08419,

    Pith/arXiv arXiv 2006

  14. [2020]

    arXiv:2010.02916

    URLhttps://arxiv.org/abs/2010.02916. arXiv:2010.02916. Z. Li, L. Liu, C. Liang, W. Chen, and T. Zhao. NorMuon: Making Muon more efficient and scalable.arXiv preprint arXiv:2510.05491,

    Pith/arXiv arXiv 2010

  15. [2021]

    A. Defazio. Schedulefree+: Scaling learning-rate-free & schedule-free learning to large language models.arXiv preprint arXiv:2605.19095,

    Pith/arXiv arXiv

  16. [2024]

    URLhttps://arxiv.org/abs/2407.17465. F. L. Cesista. Heuristic solutions for steepest descent on the stiefel manifold,

    Pith/arXiv arXiv

  17. [2025]

    URL https://leloykun.github.io/ponder/steepest-descent-stiefel/. H. Chen, H. Zhang, X. Li, Y. Dong, K. Shen, and J. Zhu. Nexus: Same pretraining loss, better downstream generalization via common minima.arXiv preprint arXiv:2604.09258,

    Pith/arXiv arXiv