Pith. sign in

REVIEW 4 major objections 5 minor 34 references

The Loss Does Not See the Basis, but Adam Does

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Gradient flow's low-rank bias passes to an optimizer only if the optimizer respects the factorization gauge; this one property separates five equivariant optimizers from five coordinate-wise ones.

desk verdict A genuinely useful classification of optimizers by gauge equivariance, with an abstract that overstates the necessity direction; the theory and control battery largely deliver. read the letter →

arxiv 2608.05136 v1 pith:T2KHGAD3 submitted 2026-08-05 cs.LG

Devender Singh This is my paper
classification cs.LG
keywords implicitbiaslow-rankgaugesymmetrymatrixfactorizationoptimizerequivarianceAdamgradientflowinterpolantselection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Training a factored model $W=UV^\top$ to interpolation, gradient descent settles on low-rank solutions while Adam, from the same small initialization, does not. This paper traces that difference to the gauge symmetry of the loss, its invariance under $(U,V)\mapsto(UQ,VQ)$ for orthogonal $Q$: an optimizer can inherit gradient flow's low-rank mechanism only if its update is gauge-equivariant, a condition necessary but not sufficient for low-rank recovery. The paper classifies nine update rules by this test, proves that memoryless equivariant updates are exactly Gram-determined left preconditioners, and shows on matrix sensing, attention heads, and hyperspectral completion that the equivariant optimizers recover planted or structured low-rank signals at lower error while coordinate-wise optimizers select worse interpolants at matched training loss. If the claim is right, basis choice inside a factorization is not a tuning detail but a decision about which solution the optimizer selects.

What carries the argument

The central object is gauge equivariance under the orthogonal group $O(k)$ acting on the latent factors of $W=UV^\top$. An optimizer's update commutes with right multiplication by $Q$ when the state transforms covariantly; this is what allows a trajectory to reproduce gradient flow's dynamics on the quotient. Theorem 4.5 pins down the memoryless equivariant update as $\Phi(G)=H(GG^\top)G$ with $H$ a function of the Gram matrix, so the only freedom inside the class is the spectral transfer function $h(\sigma)$ of Proposition A.1, with GD's $h(\sigma)=\eta\sigma$ (greedy) at one end and the polar map's $h\equiv\eta$ (equal-rate, as in Muon) at the other. Theorem 4.6 transfers gradient-flow path and limit results to common-scalar flows through a time rescaling. Together these give two axes: equivariance decides whether the low-rank mechanism is accessible, and the spectral schedule decides whether it is actually used within a budget.

What would settle it

On the paper's 40×40 rank-3 matrix-sensing task, train Adam from two gauge-equivalent initializations $(U_0,V_0)$ and $(U_0Q,V_0Q)$ using the same schedule and run both to interpolation loss $10^{-7}$ across the full learning-rate grid; the claim predicts a substantially different recovery error between the twins, and any setting where they agree to within float precision while recovering below the equivariant class's worst error would refute it.

Watch

Extended reading notes

Core claim

For a factored objective $L(U,V)=f(UV^\top)$, the loss is unchanged by the gauge transformation $(U,V)\mapsto(UQ,VQ)$ for any $Q\in O(k)$. Gradient flow on the factors has a well-studied implicit bias toward low-rank solutions, and the paper's thesis is that this bias is available to an optimizer only when the optimizer's update rule is gauge-equivariant: trajectories from gauge-equivalent initializations must remain gauge-equivalent, with covariant state. Gradient descent, momentum, shared-scalar Adam, Muon, and Shampoo pass the test; Adam, RMSProp, signSGD, Lion, and Adafactor fail it, because any fixed entrywise nonlinearity at zero optimizer state breaks equivariance. The structure theorem (Theorem 4.5) makes the equivariant class exact: a memoryless update is equivariant if and only if it has the form $\Phi(G)=H(GG^\top)G$, a left preconditioner depending only on the gauge-invariant Gram matrix. A transfer theorem shows common-scalar flows follow gradient-flow paths under time reparameterization, so pathwise low-rank results carry over. The paper also identifies the spectral schedule—how the update's spectral transfer function $h(\sigma)$ weighs singular directions—as what determines behavior inside the equivariant class, and reports experiments where the equivariant/coordinate-wise split predicts recovery error, attention-head invariant drift, and hyperspectral completion performance at matched training loss.

Load-bearing premise

The argument depends on the assumption that the relevant symmetry is the orthogonal one: an optimizer must be invariant under rotations of the factor basis to inherit gradient flow's low-rank bias, even though the paper itself shows that this invariance is neither necessary nor sufficient for low-rank recovery.

Editorial extensions

If this is right

  • Nine deployed optimizers separate into two classes on underdetermined matrix sensing at interpolation: equivariant methods reach recovery error at most 0.286, coordinate-wise methods at least 0.42, at the fixed budget.
  • Adam's per-coordinate denominator is the specific culprit: interpolating the denominator from coordinate-wise (p=1) to shared-scalar (p=0) monotonically improves recovery and lowers effective rank, leaving adaptivity and momentum fixed.
  • Attention heads carry the same gauge symmetry, so Adam separates two identical functions written in different bases at the first step, ending with per-head invariants $W_Q^\top W_K$ 56% apart in relative Frobenius distance; no per-head rotation can close that gap.
  • Muon recovers exactly low-rank targets but loses that edge once the target's spectral-tail energy exceeds about 4%, at which point GD's greedy schedule takes over, reconciling conflicting reports about Muon.
  • On two hyperspectral image-completion benchmarks at matched training loss, GD beats Adam's held-out error by 43–44% at the lowest sampling density and at lower effective rank.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the criterion generalizes, any factored layer trained with per-coordinate preconditioning—LoRA adapters, low-rank heads—is silently choosing an interpolant that depends on the arbitrary basis of the factors; swapping the basis changes the learned function even when the training objective is identical.
  • The dial parameter $p$ suggests a practical, continuous control of implicit bias strength inside Adam: interpolating toward a shared scalar trades speed for low-rank recovery without any weight decay, which could be useful when the regularization direction is known.
  • The twin protocol (gauge twin, identity twin, noise twin) could be adopted as a standard diagnostic: the ratio between the gauge and noise splits at step 1 distinguishes structural basis-dependence from numerical chaos for any optimizer in a factored architecture.
  • Model merging and re-basin methods should check gauge-invariant products like $W_Q^\top W_K$ before alignment; Adam-trained runs that start as the same function may not be reconcilable by any rotation, which would explain some merge failures.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies the implicit low-rank bias of optimizers on factored models W = UV^T, attributing the difference between gradient descent and Adam to the gauge symmetry (U,V) -> (UQ,VQ) with Q in O(k). It proves a structure theorem characterizing memoryless gauge-equivariant updates as Gram-determined left preconditioners (Theorem 4.5), a transfer theorem for common-scalar flows (Theorem 4.6), and a first-step defect formula for Adam (Proposition 4.4). Empirically it sorts nine optimizers on underdetermined matrix sensing, introduces an anisotropy dial interpolating between coordinate-wise and shared-scalar preconditioning, shows gauge-equivalent transformer twins diverge under Adam but not under equivariant optimizers, maps a spectral-tail phase diagram for Muon, and reports matched-loss benefits for GD over Adam on two hyperspectral datasets. The paper is carefully qualified in several places, especially Remark 4.7 and Appendix D.10, but the abstract and introduction state a stronger 'only if' claim than the proofs and the paper's own counterexamples support.

Significance. If the central claim is appropriately qualified, this is a useful contribution: Theorem 4.5 gives a clean algebraic characterization of memoryless equivariant updates, Theorem 4.6 makes precise which gradient-flow results transfer to common-scalar flows, and the dial experiment in §7 isolates coordinate-wise anisotropy as a causal mechanism. The control battery in Appendix D is extensive and unusually transparent, including schedule symmetrization, learning-rate curves, initialization sweeps, and ten-seed ladders, and the code is provided. The attention-twin diagnostic (§6) is a concrete and falsifiable test of basis dependence. However, the headline dichotomy is stated more strongly than the evidence: the paper's own Remark 4.7 and Appendix D.10 show that equivariance is not necessary for low-rank recovery, and Table 2's three-seed means without dispersion carry a claim that is only properly supported in an appendix. With a revised statement of the claim as a budget-level and mechanism-level classification, the paper would make a solid contribution.

major comments (4)
  1. [Abstract; §1; Remark 4.7; Appendix D.10] The abstract's 'only if' claim — that gradient flow's low-rank mechanism is available to an optimizer only if it is gauge-equivariant — is not proven and is contradicted by the paper's own Appendix D.10. Theorem 4.6 proves only the transfer direction for common-scalar flows; there is no theorem proving necessity. Remark 4.7 explicitly states that equivariance is 'neither necessary nor sufficient for low-rank recovery,' and Appendix D.10 shows long-anneal signum, a non-equivariant coordinate-wise method, reaching recovery below 1e-5 at 4e4 steps and 0.000 at 6e4 steps. The main text's careful qualifiers ('this budget,' 'less direct access,' 'empty gap (this budget)') are more accurate than the abstract. The central claim should be restated as a budget-relative and mechanism-relative classification: common-scalar equivariant flows inherit gradient-flow's path, while the coordinate-wise methods tested here do not, as a rule, within the stated budgets, with documented exceptions.
  2. [§5, Table 2] The two-class empirical separation is presented in Table 2 using three-seed means with no dispersion, while the main text says the separation is 'cleanly separated' and 'leaving a 0.14 gap.' The supporting dispersion is deferred to Appendix D.9, where Table 11 in fact shows two flagged exceptions: signum reaches 0.000 at every size n≥64, and Shampoo sits above Adam at n≥64. These exceptions are disclosed in §11, but the main-text claim should either include the dispersion and exceptions in Table 2 itself or explicitly carry the qualifier into the abstract's taxonomy. As written, a reader of the main text gets a stronger assertion than the data support.
  3. [§9; Appendix D.6] The headline real-data result — GD cuts held-out error by 43–44% — depends on the train-only learning-rate selection rule, and the paper itself reports in Appendix D.6 that test-informed selection reduces the advantage to about 13%. The protocol is defensible, and the ordering and rank profile are reported as stable, but the abstract and §9 should state this sensitivity prominently rather than only in the appendix. The current wording ('GD cuts held-out error by 43–44%') is presented as a property of the optimizers, when it is in part a property of the evaluation protocol.
  4. [§4, Proposition 4.3 and Proposition 4.4] The classification of Adam, RMSProp, signSGD, and Lion as gauge-breaking is proven only for the first update from zero optimizer state. Proposition 4.3 assumes a memoryless entrywise map, and Proposition 4.4 explicitly computes the first-step defect; the paper notes that checking subsequent updates would require reasoning about state evolution. The abstract and contribution list, however, state flatly that these methods 'break' equivariance. The paper should state in the theory section that the proved statement concerns zero-state first updates, with the caveat that later-step behavior is an empirical matter. This does not undermine the experiments, but it is a mismatch between theorem and summary that should be corrected.
minor comments (5)
  1. [§5, Table 1 caption] The caption of Table 2 notes 'no dispersion quoted'; since Appendix D.9 contains ten-seed data, the main table should at least reference the ladder explicitly in the same paragraph as the two-class claim, rather than only in the deferred appendix.
  2. [§8, Table 4] The phase diagram table reports ten-seed results but omits the ±1 s.d. bands, referring to Figure 4; a reader relying on the table alone cannot assess the boundary at τ*≈0.2, where the GD–Muon difference is described as within seed variance.
  3. [§3, Protocol] The term 'dof' is used before it is defined in §9; a one-sentence definition at first use in §3 would help readability.
  4. [§10 and Appendix C] The FlowAdam comparison is careful about ε conventions and the dial endpoint, but the sentence 'the p=0 endpoint reads 0.201' vs. the geometric-mean endpoint 0.229 could be stated even more plainly in the main text to avoid an apparent inconsistency with the 0.169 FlowAdam-p figure.
  5. [Appendix D.10] The long-anneal signum result is reported as an empirical fact without a mechanistic account, which is honest; adding a sentence in §11 or the contribution list pointing to this as an open problem would strengthen the paper's framing of the two-class claim.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation; the only questionable step is an unproven 'only if' claim, which is an overstatement rather than a circular reduction.

full rationale

The paper's main derivation is self-contained. The equivariance classification (Definition 3.1, Propositions 4.2–4.4, Theorem 4.5) is proven directly from the definition of gauge equivariance and the covariance of gradients (Lemma 4.1); it does not presuppose which optimizers recover low-rank solutions. The transfer theorem (Theorem 4.6) is a mathematical time-reparameterization result and imports only external gradient-flow results (Gunasekar et al., Arora et al., Li et al.), not results of this paper. The zoo experiments use a planted ground truth and an independent minimum-nuclear-norm baseline, report full learning-rate curves, and are explicitly budget-qualified; selection of learning rate by best recovery is disclosed and applied symmetrically, and it favors the coordinate-wise class, so it cannot manufacture the equivariant advantage. The one-parameter dial (§7) changes only the denominator anisotropy and is empirical. The attention twin experiment is deterministic with an A=I control and a noise twin, so the measured Adam split is not definitionally forced. The only self-cited work, FlowAdam (Singh & Sheel, 2026), is used as a case study to test the criterion and is itself shown to have a gauge-breaking clip; it is not cited as evidence for the main claim. The abstract's 'only if' necessity claim for gauge equivariance is asserted, not proven: no theorem states the converse of Theorem 4.6, and Remark 4.7 plus Appendix D.10 explicitly provide a non-equivariant long-anneal signum counterexample that recovers low-rank targets. That is a genuine overstatement/correctness risk, but it is not circular because the empirical two-class separation is presented as a budget-level observation and the paper itself flags the exception. No fitted parameter is renamed as a prediction, and no self-citation carries a load-bearing uniqueness or ansatz argument.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central classification theorem is parameter-free. The experimental program introduces no new physical or model entities; its free parameters are standard optimizer hyperparameters plus the empirically estimated phase boundary. The main axioms are the prior gradient-flow results being transferred, the choice of O(k) as the operative gauge group, and the zero-state initialization convention for optimizer states.

free parameters (5)
  • Per-optimizer learning rate (best-recovery selection) = Varies by method; examples include Adam 0.01, GD 0.001 to 0.03, Muon 0.03 to 0.1
    Selected on recovery outcome in the zoo experiments. The paper argues the two-class split is robust across rate grids, but individual recovery numbers are rate-dependent.
  • Initialization scale = 1e-3
    Small initialization is required for the low-rank implicit bias regime; at 1e-2 the split fades for most methods (Appendix D.4).
  • Shampoo damping lambda = 1
    Kept untuned at all problem sizes, which the authors flag as the reason Shampoo degrades at large n.
  • Muon Newton-Schulz coefficients = 3.4445, -4.7750, 2.0315
    Standard published coefficients used in experiments; they affect finite msign amplification and the chaos observed in Muon twins.
  • Phase boundary tau-star = ~0.2, about 4 percent tail energy
    Empirically estimated crossover where Muon cedes to GD. The estimate moved from 0.35 to 0.2 when the seed count increased from 3 to 10, so it is not a derived constant.
assumptions (5)
  • domain assumption Gradient flow on underdetermined matrix sensing from small initialization converges to minimum-nuclear-norm or greedy low-rank solutions (Gunasekar et al. 2017, Arora et al. 2019, Li et al. 2021).
    The transfer theorem imports these restricted-regime results rather than re-deriving them.
  • domain assumption The orthogonal group O(k) is the correct gauge group for optimizer equivariance because it is the maximal isometric symmetry of the factored parameter space (Lemma 3.2).
    The full function-preserving group is GL(k), and restricting to O(k) is justified by isometry, but this restriction is a substantive modeling choice for the criterion.
  • ad hoc to paper Optimizer states are zero at initialization and map to zero under the gauge (sigma_Q(0)=0).
    Used in Definition 3.1 and Proposition 4.3 to analyze first updates. This is standard for Adam, RMSProp, signSGD, and Lion, but the equivariance definition is tied to this convention.
  • domain assumption Full-batch deterministic training with small enough step sizes approximates gradient flow for the transfer conclusions.
    Remark A.5 concedes that discrete GD tracks flow only to O(eta) error, yet many conclusions are read off finite-budget discrete runs.
  • domain assumption The matrix sensing task with Gaussian measurements, m=2*dof, rank-3 target, and n=40 is representative of factored-model implicit bias.
    The ladder to n=256 and the hyperspectral experiments support this, but the signum long-anneal exception shows the two-class story is not universal.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Loss Does Not See the Basis, but Adam Does." pith.science (2026). https://pith.science/paper/T2KHGAD3

@misc{pith2026260805136,
  author       = {Pith},
  title        = {Pith review of: The Loss Does Not See the Basis, but Adam Does},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T2KHGAD3}},
  note         = {Machine review of arXiv:2608.05136}
}
abstract

Gradient descent on a factored model $W = UV^\top$ is implicitly biased toward low-rank solutions, while Adam, starting from the same small initialization, is not. We trace the difference to the gauge symmetry of the loss, its invariance under $(U, V) \mapsto (UQ, VQ)$. Gradient flow's low-rank mechanism is available to an optimizer only if that optimizer is gauge-equivariant, a condition necessary for the transfer but not sufficient for low-rank recovery. Gradient descent, momentum, "shared-scalar" Adam, Muon, and Shampoo satisfy it. Adam, RMSProp, and the other coordinate-wise methods do not. A structure theorem characterizes the memoryless equivariant rules as exactly the Gram-determined left preconditioners, and a transfer theorem carries gradient flow's pathwise properties to common-scalar flows. We then sort nine update rules on underdetermined matrix sensing by recovery error against the planted ground truth. A one-parameter family from coordinate-wise to shared-scalar preconditioning restores the bias monotonically, isolating anisotropy as the cause. A "spectral schedule" reconciles two opposing reports about Muon: equal-rate updates recover exactly low-rank targets but lose their edge as the spectral tail grows. In transformers, Adam separates two gauge-equivalent initializations at the first step, where the equivariant optimizers stay at float precision, and ends with the per-head invariants $W_Q^\top W_K$ 56% apart in relative Frobenius distance, a gap no per-head rotation can close. On two hyperspectral datasets at matched training loss, gradient descent cuts held-out error by 43-44% at the lowest sampling density, and at lower effective rank. Basis choice is therefore not a tuning detail but a decision about which interpolant the optimizer selects.

Figures

Figures reproduced from arXiv: 2608.05136 by the authors.

Figure 1
Figure 1. Equivariance structures the set of available optimizers under a fixed computational budget. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Adam is basis-dependent in attention, shown here by the relative logit distance between twins in the gauge and noise experiments. In the former the two copies are initialized as the same function, while in the latter they differ by a small (10−7 ) perturbation in the same basis. Adam’s gauge twins (solid vermilion line) split structurally in one step, to a value four orders of magnitude greater than its own noise tw… view at source ↗
Figure 3
Figure 3. The dial. As Adam’s preconditioner changes shape from coordinate-wise (p=1) towards isotropic (p=0), the effective rank of the solution gracefully decreases towards that of standard gradient descent (GD). The plotted curve is the “envelope arm,” in which for each p we pick the learning rate giving the best recovery from a shared grid. A “fixed-step arm” that instead varies only p at each individual rate of that grid… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The spectral-tail phase diagram. Recovery vs. the target’s tail-amplitude parameter τ (the tail carries τ 2 of the target’s energy). Adam has the worst recovery error at every τ plotted. Within the equivariant class, Muon’s aggressive equal-rate schedule is exact at τ=…
Figure 5
Figure 5. Figure 5: Performance as a function of training loss on real data [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

34 extracted references · 20 canonical work pages

  1. [1]

    =X(G2), and H(GG⊤) :=X(G) is well defined, giving Φ(G) =H(GG⊤)G. In the square invertible case, the polar decomposition G= (GG⊤)1/2O with O∈O(n) gives directly Φ(G) =Φ (GG⊤)1/2 O= H(GG⊤)GwithH(P) =Φ(P 1/2)P −1/2; uniqueness follows by evaluating atG=P 1/2. B.7 PROPOSITIONA.1 (SPECTRAL TRANSFER) Proof. With the compact SVD G= ∑i σiuiv⊤ i we have GG⊤ = ∑i σ...

    work page 2017

  2. [3]

    https://doi.org/10.4231/R7RX991C,

    doi:10.4231/r7rx991c

  3. [4]

    E EXTENDED RELATED WORK Transformation invariance in optimizer design.The engineering literature has grappled with similar design choices at a lower level; for instance, Ling et al

    Thus, we view signum as an exceptional case that helps define boundaries: non-equivariant dynamics can achieve low-rank solutions only in a narrow regime—exactly low-rank settings, large steps, and long anneals—while common-scalar equivariance supports a more robust, generalizable, and analyzable gradient-flow mechanism. E EXTENDED RELATED WORK Transforma...

    work page 2022

  4. [5]

    The Spectral Dynamics and Noise Geometry of Muon

    Pierfrancesco Beneventano, Mahmoud Abdelmoneum, and Tomaso Poggio. The spectral dynamics and noise geometry of Muon.arXiv preprint arXiv:2606.08388,

    work page Pith review arXiv

  5. [6]

    Old optimizer, new norm: An anthology.arXiv preprint arXiv:2409.20325,

    Jeremy Bernstein and Laker Newhouse. Old optimizer, new norm: An anthology.arXiv preprint arXiv:2409.20325,

    arXiv

  6. [8]

    How do simple rotations affect the implicit bias of Adam?arXiv preprint arXiv:2510.23804,

    Adela DePavia, Vasileios Charisopoulos, and Rebecca Willett. How do simple rotations affect the implicit bias of Adam?arXiv preprint arXiv:2510.23804,

    arXiv

  7. [10]

    To use or not to use Muon: How simplicity bias in optimizers matters.arXiv preprint arXiv:2603.00742,

    Sara Dragutinovi´c, Yedi Zhang, and Rajesh Ranganath. To use or not to use Muon: How simplicity bias in optimizers matters.arXiv preprint arXiv:2603.00742,

    arXiv

  8. [11]

    Manuel Graña, Miguel A

    arXiv:2502.04664. Manuel Graña, Miguel A. Veganzons, and Borja Ayerdi. Hyperspectral remote sensing scenes. https://web.archive.org/web/20250115080044/https://www.ehu.eus/ccwintco/index. php/Hyperspectral_Remote_Sensing_Scenes,

    arXiv

Show all 34 references
  1. [12]

    Distributes the corrected Indian Pines (A VIRIS) and Pavia University (ROSIS) cubes

    Grupo de Inteligencia Computacional (GIC), Universidad del País Vasco (UPV/EHU). Distributes the corrected Indian Pines (A VIRIS) and Pavia University (ROSIS) cubes. Page last edited 12 July 2021; the original page now returns HTTP 403, so it is cited via the Internet Archive ...

    2021

  2. [13]

    23 Preprint Suriya Gunasekar, Blake Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nathan Srebro

    arXiv:2602.16340. 23 Preprint Suriya Gunasekar, Blake Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nathan Srebro. Implicit regularization in matrix factorization. InAdvances in Neural Information Processing Systems (NeurIPS),

    arXiv

  3. [15]

    The unreasonable effectiveness of recurrent neural networks

    Andrej Karpathy. The unreasonable effectiveness of recurrent neural networks. http://karpathy.github. io/2015/05/21/rnn-effectiveness/,

    2015

  4. [16]

    com/karpathy/char-rnn/blob/master/data/tinyshakespeare/input.txt

    Origin of the character-level Shakespeare experiment; the tiny-shakespeare corpus is distributed with the accompanying char-rnn release, https://github. com/karpathy/char-rnn/blob/master/data/tinyshakespeare/input.txt. Nitish Shirish Keskar and Richard Socher. Improving genera...

    arXiv

  5. [17]

    Symmetry-compatible principle for optimizer design: Embeddings, LM heads, SwiGLU MLPs, and MoE routers.arXiv preprint arXiv:2605.18106,

    Tim Tsz-Kit Lau and Weijie Su. Symmetry-compatible principle for optimizer design: Embeddings, LM heads, SwiGLU MLPs, and MoE routers.arXiv preprint arXiv:2605.18106,

    arXiv

  6. [21]

    Dead-direction conditioners: Gauge-equivariant preconditioning for deep networks

    Tejas Pradeep Shirodkar. Dead-direction conditioners: Gauge-equivariant preconditioning for deep networks. arXiv preprint arXiv:2606.29176,

    arXiv

  7. [22]

    Symmetry breaking in transformers for efficient and interpretable training.arXiv preprint arXiv:2601.22257,

    Eva Silverstein, Daniel Kunin, and Vasudev Shyam. Symmetry breaking in transformers for efficient and interpretable training.arXiv preprint arXiv:2601.22257,

  8. [23]

    Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro

    arXiv:2604.06652. Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data.Journal of Machine Learning Research, 19(70),

    arXiv

  9. [24]

    How the optimizer shapes learned solutions in equivariant neural networks.arXiv preprint arXiv:2605.27662,

    Teodor-Mihai Stupariu and Andrei Manolache. How the optimizer shapes learned solutions in equivariant neural networks.arXiv preprint arXiv:2605.27662,

    arXiv

  10. [25]

    PolarAdamW: Disentangling spectral control and Schur gauge-equivariance in matrix optimisa- tion.arXiv preprint arXiv:2605.07067,

    Haozhou Zhang. PolarAdamW: Disentangling spectral control and Schur gauge-equivariance in matrix optimisa- tion.arXiv preprint arXiv:2605.07067,

    arXiv

  11. [26]

    aggressiveness

    25 Preprint A INSIDE THE EQUIVARIANT CLASS Theorem 4.5 pins down the shape of an equivariant memoryless rule. This appendix collects what lives inside that shape: the spectral coordinate, the extension to stateful rules and to deeper factorizations, and what happens to balance...

    2026

  12. [28]

    • Target and measurements

    C EXPERIMENTAL DETAILS Sensing task (§5–§7, §8). • Target and measurements. X ∗ =U ∗V ∗⊤, U ∗,V ∗ ∈R 40×3 i.i.d. Gaussian scaled by 1/ √ 3; m= 462=2dof, dof=3(80−3) =231;A i i.i.d. standard Gaussian; loss=mean i(⟨Ai,W ⟩ −y i)2. •Factors. U,V∈R 40×40 initializedN(0,10 −6)entryw...

    2024

  13. [30]

    Two configurations failed to meet that threshold within the allotted steps: FlowAdam-p=0 got to 7×10 −6, and global-norm clipping at p=1 reached 6×10 −7

    We use the same three paired seeds and the same 10−7 interpolation threshold. Two configurations failed to meet that threshold within the allotted steps: FlowAdam-p=0 got to 7×10 −6, and global-norm clipping at p=1 reached 6×10 −7. Appendix D.8 continues the former to a true 1...

  14. [31]

    sc- Adam

    So picking the best rate for each method would have improved GD by13% over Adam, compared to the 43% found with the train-only selection (also consistent across the four paired seeds in the GPU experiment). We report the train-only selection because using held-out statistics t...

    2026

  15. [33]

    incorporate similar domain knowledge for better conditioning; ScaledGD is also our within-class counterexample (Remark 4.7). Independently, Zhang (2026) discovers a similar local phenomenon in a different application domain: in group-equivariant networks, Muon’s polar step res...

    2026

  16. [34]

    Recent theoretical work attributes a spectral max-margin bias to Muon (Fan et al., 2025; Gronich & Vardi, 2026), and Kang et al

    uses Kronecker-factored preconditioning—both methods seeing practical use. Recent theoretical work attributes a spectral max-margin bias to Muon (Fan et al., 2025; Gronich & Vardi, 2026), and Kang et al. (2026) establish equal-rate singular-value growth for Muon when applied t...

    2026

  17. [256]

    • Training.Full-batch optimization over 1500 steps, executed deterministically on CPU (torch.use_deterministic_algorithms)

    The model omits biases and does not apply query–key normalization (qk-norm), so the gauge holds exactly. • Training.Full-batch optimization over 1500 steps, executed deterministically on CPU (torch.use_deterministic_algorithms). • Gauge.An orthogonal transformation Ah per head...

    2021

  18. [2015]

    Scene acquired 12 June 1992; deposited 30 September

    1992

  19. [2018]

    Towards understanding the power and limits of the Muon optimizer: A river-valley perspective.arXiv preprint arXiv:2606.21514,

    Tianqi Shen, Jinji Yang, Runze Shi, Jianhao Ma, Jiaye Teng, and Ziye Ma. Towards understanding the power and limits of the Muon optimizer: A river-valley perspective.arXiv preprint arXiv:2606.21514,

    arXiv

  20. [2019]

    Grokking: Generalization beyond overfitting on small algorithmic datasets.arXiv preprint arXiv:2201.02177,

    Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets.arXiv preprint arXiv:2201.02177,

    arXiv

  21. [2021]

    George E. Dahl, Frank Schneider, Zachary Nado, Naman Agarwal, Chandramouli Shama Sastry, Philipp Hennig, Sourabh Medapati, Runa Eschenhagen, Priya Kasimbeg, Daniel Suo, Juhan Bae, Justin Gilmer, Abel L. Peirson, Bilal Khan, Rohan Anil, Mike Rabbat, Shankar Krishnan, Daniel Sni...

    arXiv

  22. [2022]

    Muon is scalable for LLM training.arXiv preprint arXiv:2502.16982,

    Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al. Muon is scalable for LLM training.arXiv preprint arXiv:2502.16982,

    arXiv

  23. [2023]

    Understanding and inverse design of implicit bias in stochastic learning: a geometric perspective.arXiv preprint arXiv:2601.06597,

    Nicola Aladrah, Emanuele Ballarin, Matteo Biagetti, Alessio Ansuini, Alberto d’Onofrio, and Fabio Anselmi. Understanding and inverse design of implicit bias in stochastic learning: a geometric perspective.arXiv preprint arXiv:2601.06597,

    arXiv

  24. [2024]

    Uniform spectral growth and convergence of Muon in LoRA-style matrix factorization.arXiv preprint arXiv:2602.06385,

    Changmin Kang, Jihun Yun, Baekrok Shin, Yeseul Cho, and Chulhee Yun. Uniform spectral growth and convergence of Muon in LoRA-style matrix factorization.arXiv preprint arXiv:2602.06385,

  25. [2025]

    Muonp: Muon with fractional spectral powers.arXiv preprint arXiv:2606.13867,

    Yihe Dong and Will Sawin. Muonp: Muon with fractional spectral powers.arXiv preprint arXiv:2606.13867,

  26. [2026]

    Marion F

    arXiv:2510.26303. Marion F. Baumgardner, Larry L. Biehl, and David A. Landgrebe. 220 band A VIRIS hyperspectral image data set: June 12, 1992 Indian Pine test site

    1992

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.