Pith. sign in

REVIEW 3 major objections 5 minor 64 references

PoLoRA reaches tuned Adam's final held-out loss in 1.2–1.7× fewer steps while adding at most 3% per-step overhead, on LoRA finetuning across models and datasets.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 17:27 UTC pith:MBCAV6GV

load-bearing objection A genuinely new LoRA optimizer with a clean derivation and a broad but single-seed, same-split evaluation; the speedup numbers need an independent test set and seed variance before they can be believed, but the work deserves serious peer review. the 3 major comments →

arxiv 2607.17620 v1 pith:MBCAV6GV submitted 2026-07-20 cs.LG cs.CLmath.OC

PoLoRA: A Preconditioned Orthogonalized LoRA Optimizer

Nikhil Ghosh , Tetiana Parshakova , Robert M. Gower This is my paper
classification cs.LG cs.CLmath.OC
keywords low-rank adaptationLoRA optimizationpreconditioningmatrix signKronecker-factored curvatureper-sample loss controlspectral norminstruction tuning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper's central claim is that how you optimize the two low-rank LoRA factors matters as much as which loss you minimize. Rather than treating all adapter parameters as one flat vector like the standard Adam recipe, PoLoRA derives an update direction from controlling the change in the per-sample loss, then applies it to the merged product of the two factors. The authors report that this consistently beats Adam on instruction-tuning of code and math tasks, reaching Adam's final held-out loss in 1.2–1.7× fewer steps while adding at most 3% per-step overhead. They also report that PoLoRA is easier to tune than Adam: its optimal learning rate is stable across ranks and its performance is less sensitive to the learning rate. If these claims hold, adapter finetuning becomes cheaper and hyperparameter search simpler.

Core claim

The central discovery is that a steepest-descent direction for a LoRA step should be chosen by bounding the change in the per-sample loss, not merely the spectral norm of the weight update. This leads to a preconditioned matrix-sign update of the form ΔW = −τ P^(-1/2) msign(P^(-1/2)GQ^(-1/2))Q^(-1/2), where P and Q are diagonal curvature estimates fit to a Kronecker approximation of the per-sample gradient covariance. The direction is then applied to the LoRA factors with a magnitude rule ρ = η/(‖A‖ + ‖B‖) that controls both the per-factor step size and the merged update. In the authors' experiments, this combination consistently outperforms Adam across four base models, code and math datase

What carries the argument

The engine is a product-aware linear minimization oracle (LMO): a subroutine that picks the steepest descent direction within a constraint set. PoLoRA's LMO minimizes the linearized loss subject to a bound on per-sample loss change, which is approximated by replacing the exact per-sample gradients with a centrally symmetric outer-gradient set and a Kronecker factorization Σ ≈ Q⊗P. That factorization turns the constraint into a preconditioned spectral norm, and the minimizing direction becomes a preconditioned matrix sign. A separate magnitude rule sets the sizes of the factor updates while preserving the bound on the merged product, addressing the instability that arises when A or B has smal

Load-bearing premise

The speedup rests on the assumption that the per-sample gradient covariance is well approximated by a Kronecker product of two matrices, each taken to be diagonal; if that approximation does not capture the true loss geometry, the curvature preconditioner is not actually controlling per-sample loss changes and the advantage over Adam could vanish.

What would settle it

Measure the actual per-sample loss changes under PoLoRA steps and compare them with the bound τ implied by the derivation: if individual examples' losses frequently move by more than the bound, the linearized per-sample control is not operative. A cheaper check is to replace the diagonal P and Q with full Kronecker factors; if the speedup over Adam does not improve, the curvature component is not responsible for the gain.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • PoLoRA reaches tuned Adam's final held-out loss in 1.2–1.7× fewer steps across four base models (1B to 8B) on code and math instruction tuning, with wall-clock speedup nearly unchanged after per-step overhead.
  • The per-step overhead is at most 3% at the tested batch size, and the overhead fraction shrinks as batch size grows because the optimizer FLOPs are independent of batch size.
  • Curvature preconditioning and magnitude control each contribute roughly half of the speedup over the product-aware but curvature-blind baseline; removing both reduces PoLoRA to that baseline and eliminates the speedup.
  • PoLoRA's optimal learning rate is stable across ranks r = 32–256, whereas Adam's drifts, so PoLoRA can transfer a tuned learning rate across rank settings.
  • The speedup over Adam is larger on a low-resource-language dataset than on code, consistent with the method's advantage growing when finetuning must move the model further from its pretraining distribution.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the same per-sample loss-control derivation does not rely on low rank, so it could plausibly be applied to full finetuning or pretraining; if the diagonal Kronecker approximation holds there, matrix-aware optimizers could gain similar speedups.
  • Editorial extension: the rank-stable learning rate suggests PoLoRA may remove the need for explicit rank-dependent rescaling schemes; a direct test would compare PoLoRA against such rescaling methods on downstream tasks, not just held-out loss.
  • Editorial extension: the magnitude rule's dependence on the factorization (A,B) ↦ (cA, c⁻¹B) means balanced factors give larger steps; a testable extension is to run PoLoRA from deliberately imbalanced non-zero initializations and measure whether self-balancing still settles near ‖A‖ ≈ ‖B‖.
  • Editorial extension: the larger speedup on low-resource language data suggests a natural test in other distribution-shift settings, such as domain adaptation or continual finetuning, where moving far from the pretraining distribution is common.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. PoLoRA is a new optimizer for LoRA adapters. It combines three ingredients: a product-aware spectral update direction (Product Muon), curvature preconditioning derived from a per-sample loss-change LMO with a Kronecker-factored (and in practice diagonal) approximation to the gradient covariance, and a magnitude rule that controls both factor-step and merged-update spectral norms. The paper derives the update from first principles (Section 2, proofs in Appendix B), gives Algorithm 1, and reports experiments on instruction tuning for code and math across models from 1B to 8B. The central empirical claim is that PoLoRA reaches the final held-out loss of tuned Adam in 1.2–1.7× fewer steps, with at most 3% per-step overhead, and that its optimal learning rate is stable across ranks. The authors provide open-source code, a component ablation, and a rank sweep.

Significance. If the empirical claims hold, PoLoRA would be a practically useful LoRA optimizer with a clean conceptual derivation. The paper's strengths are the self-contained derivation of the preconditioned spectral direction, the explicit product-aware LMO, the supplied proofs in Appendix B, and the open-source implementation. The ablation attributing the speedup roughly equally to curvature and magnitude is informative, though underpowered. However, the quantitative central claim is currently not established: the evaluation selects learning rates and reports final losses on the same 1% split, with one seed per setting. The headline speedup is therefore a selected-on-validation number, not a test-set estimate. The diagonal-Kronecker approximation used in the implementation is also not independently validated, leaving a gap between the stated mechanism and the observed speedup. The method is promising, but the empirical evidence needs strengthening before the claims can be accepted.

major comments (3)
  1. [§3.1 / Appendix A] The evaluation protocol is the main load-bearing issue. Appendix A states: 'We hold out 1% of each dataset for evaluation, and use this split for both learning-rate selection and all reported losses.' With one run per (model, dataset, rank) cell, every speedup in Tables 1–2 and Figures 1–4 is the best-of-sweep value on the split used to choose the run, not an unbiased test estimate. The learning-rate grid is spaced by factors of 3, so selection noise can materially affect both the reported final loss and the steps-to-Adam ratio. This directly undermines the headline '1.2–1.7× fewer steps' claim. Please report results on a separate test split (or use nested validation), run at least 3–5 seeds per setting, and report mean/standard error or the worst-case speedup. If compute is limited, at least give the seed that is median by final loss and show the separation between selection and reporti
  2. [§2.3, Eq. (11); Algorithm 1, line 2] The theoretical derivation hinges on the Kronecker approximation Σ ≈ Q⊗P and, in the implementation, on P and Q being diagonal. The Conclusion explicitly acknowledges that the preconditioners are diagonal for efficiency, yet no experiment validates that this diagonal Kronecker approximation faithfully represents the loss geometry. The component ablation (Figure 2) shows that removing curvature preconditioning halves the speedup, but it does not compare the diagonal update against the idealized P,Q or against a richer approximation (e.g., rank-1 or block-diagonal), nor does it test whether the per-sample bound (7) is actually better controlled by PoLoRA than by Adam or Product Muon. Without such evidence, the mechanism in Section 2.3 is not tied to the observed speedup. Please add a small-scale diagnostic — for example, measure max_i |<G_i, ΔW>| before and after the update, or compare dia
  3. [§3.3 / Figure 2] The claim that 'curvature and magnitude each account for about half the speedup gap' rests on a single run on one model/dataset (Llama-3.2-1B, math, r=256) with no seed variance. Given the evaluation protocol issue in Appendix A, this attribution is not quantitative. Additionally, the text says removing magnitude control sets τ = η/2 in (16), but Algorithm 1 always normalizes the direction to spectral norm ρ; it is unclear what exact update the 'w/o magnitude' variant performs. Please specify the exact variant (for example, keep the product-aware direction but disable the ρ normalization, or use the Product Muon step) and provide error bars or at least multiple seeds for the ablation.
minor comments (5)
  1. [Appendix A / throughout] The term 'held-out' is misleading because the same split is used for learning-rate selection and reporting. Consider renaming it 'validation' to make clear that reported losses are not independent test losses.
  2. [Table 2] The entries '1 .48×' and '1 .54×' contain stray spaces; please fix the formatting. Similar spacing appears in Figure captions.
  3. [Section 4] The method name 'Riemannion' appears to be a typo for 'Riemannian' or the actual name used in the cited work. Please verify and correct.
  4. [Algorithm 1, line 1] The line 'cMA ← β1MA + (1 − β1)GA' is written twice; the intent is likely that the second is a look-ahead expression using the updated buffer. Please clarify the notation so the update is unambiguous.
  5. [Appendix G / Figure 5] The statement that 'the ratio settles near 1' is based on a single run per rank. If this is a load-bearing observation for the magnitude rule, please specify that it is unseeded or provide multiple runs.

Circularity Check

0 steps flagged

No circular derivation: PoLoRA's update follows from its stated LMOs; the only self-citation is a non-load-bearing lemma, and the same-split LR selection is an evaluation limitation, not circularity.

full rationale

The derivation chain is self-contained. Lemma 2 solves the spectral LMO (2) by norm duality; Lemma 4 solves the preconditioned LMO used for the factor updates (16)/(19); Lemma 3 evaluates the support function of the outergradient set and converts the per-sample loss constraint (8) into the preconditioned spectral constraint (13). The magnitude rule (20)-(22) is an algebraic bound on ||B ΔA + ΔB A||_2, not an equality fitted to a target. The curvature preconditioners P and Q are estimated online from factor-gradient moments (Appendix D, eqs. (28)-(32)) rather than fitted to the reported losses, so the claimed speedup is a measured outcome, not a value enforced by construction. The only self-citation in the proof chain is 'The minimum-norm statement follows from Lemma C.2 [42]' (Appendix B); [42] shares author Parshakova, but the cited lemma only establishes a least-Frobenius-norm uniqueness property that the normalized direction (22) does not require, so it is not load-bearing. The paper's own limitation statement (Section 5) correctly notes the diagonal/Kronecker approximation and linearizations; these weaken the loss-control guarantee but are not circular substitutions. The same-split evaluation protocol (Section A: 'use this split for both learning-rate selection and all reported losses') is a genuine methodological risk—reported losses are best-of-sweep values on the selection split and no seed variance is given—but it does not reduce the speedup to the hyperparameter fit by any equation; it is an empirical-validity concern, not circularity.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 0 invented entities

The derivation rests on standard linear algebra and on a series of approximations (Kronecker-factored curvature, diagonal preconditioners, linearized losses). These approximations are transparently stated and are typical for practical optimizers, but they are load-bearing for the method's empirical success.

free parameters (6)
  • learning rate η = swept per optimizer/dataset (e.g., 1e-2 for PoLoRA on Llama-3.2-1B math)
    All optimizers are given their best learning rate from a sweep; the reported speedup depends on this tuning.
  • momentum decay β1 = 0.9
    Fixed by hand, standard for Adam-style optimizers; not searched.
  • curvature decay β2 = 0.99
    Chosen to control the EMA of preconditioners; not swept in the paper.
  • Newton-Schulz iterations K = 8
    Number of Gram Newton-Schulz iterations; chosen as an accuracy/overhead tradeoff.
  • relative damping δ = 1e-4
    Damping for inverse square roots; prevents blow-up in low-curvature directions.
  • LoRA scale α = r
    Set equal to rank following common practice; not swept.
axioms (6)
  • standard math Per-sample gradients of a linear layer satisfy Gi = bi xi^T (rank-one)
    Lemma 1 property 2; exact for a linear layer by the chain rule.
  • standard math Leverage bound ⟨Σ†gi, gi⟩ ≤ n
    Lemma 1 property 1; follows from the projection inequality.
  • domain assumption Second-moment covariance is approximated as Σ ≈ Q⊗P (Kronecker)
    Section 2.3 eq (11); a standard approximation used to make the update tractable, but not exact.
  • domain assumption P and Q are diagonal
    Algorithm 1, line 2; reduces overhead but ignores cross-feature correlations in the curvature estimate.
  • domain assumption Local linearization of the loss (eq. 7)
    Section 2.3; assumes the step size is small enough that first-order approximation is valid.
  • domain assumption Linearized product update (eq. 3) with the quadratic term dropped
    Section 2.2; Remark 1 bounds the error as O(η) and shows it is small in the standard initialization.

pith-pipeline@v1.3.0-alltime-deepseek · 20928 in / 8156 out tokens · 77230 ms · 2026-08-01T17:27:09.848588+00:00 · methodology

0 comments
read the original abstract

Low-rank adaptation (LoRA) makes finetuning large language models cheaper by adding to each weight matrix a trainable low-rank update parameterized as the product of two matrices. These matrices are usually trained with Adam, which treats them as a single flat vector of parameters and ignores both the matrix and product structure of LoRA. Applying a matrix-aware optimizer such as Muon to each factor does not consistently improve over Adam, and neither do the product-aware Muon variants proposed in concurrent works. To realize consistent gains, we introduce PoLoRA, a Preconditioned Orthogonalized LoRA optimizer built from three ingredients: a product-aware spectral update direction, curvature preconditioning derived from controlling the per-sample loss change, and a magnitude rule that controls the sizes of both the factor and merged updates. We evaluate PoLoRA on instruction-tuning datasets for code and math across models from 1B to 8B parameters, and find that it reaches the final held-out loss achieved by tuned Adam in 1.2-1.7 times fewer steps, while adding at most 3% per-step overhead. Compared to Adam, PoLoRA is also less sensitive to the learning rate, and its optimal learning rate is stable across ranks.

Figures

Figures reproduced from arXiv: 2607.17620 by Nikhil Ghosh, Robert M. Gower, Tetiana Parshakova.

Figure 1
Figure 1. Figure 1: PoLoRA outperforms baselines during training. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Curvature and magnitude both improve PoLoRA. [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: PoLoRA exhibits learning-rate transfer. Final loss versus learning rate of Llama-3.2-1B finetuned on the math dataset at each rank r ∈ {32, 64, 128, 256}. The optimal learning rate for PoLoRA stays at 10−2 across ranks, while Adam’s drifts. work [20, 33] extends this to other initializations. Rank-stabilized LoRA [27] replaces the α/r adapter scale with α/√ r to avoid the rank-dependent shrinkage that make… view at source ↗
Figure 4
Figure 4. Figure 4: The speedup over Adam is larger on the low-resource language. [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Factor norm self-balancing in PoLoRA. Spectral-norm ratio ∥B∥2/∥A∥2 over training steps for Llama￾3.2-1B finetuned on the math dataset at each rank. The ratio settles near 1. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: PoLoRA speedup over Adam across settings. [PITH_FULL_IMAGE:figures/full_fig_p027_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

64 extracted references · 28 linked inside Pith

  1. [1]

    Amsel et al

    N. Amsel et al. The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm. 2025. arXiv: 2505.16932 [cs.LG]

    Pith/arXiv arXiv 2025

  2. [2]

    Anson and L

    B. Anson and L. Aitchison.Controlling changes to attention logits. 2025. arXiv: 2511.21377 [cs.LG]

    arXiv 2025

  3. [3]

    Modular Duality in Deep Learning

    J. Bernstein and L. Newhouse. “Modular Duality in Deep Learning”. In: Proceedings of the 42nd International Conference on Machine Learning. Vol. 267. Proceedings of Machine Learning Research. PMLR, 2025, pp. 3920–3930

    2025

  4. [4]

    Bertsimas and J

    D. Bertsimas and J. N. Tsitsiklis. Introduction to Linear Optimization. Athena Scientific, 1997

    1997

  5. [5]

    LoRA Learns Less and Forgets Less

    D. Biderman et al. “LoRA Learns Less and Forgets Less”. In:Transactions on Machine Learning Research (TMLR) (2024). arXiv: 2405.09673 [cs.LG]

    Pith/arXiv arXiv 2024

  6. [6]

    Bogachev et al.LoRA meets Riemannion: Muon Optimizer for Parametrization-independent Low-Rank Adapters

    V. Bogachev et al.LoRA meets Riemannion: Muon Optimizer for Parametrization-independent Low-Rank Adapters. 2025. arXiv: 2507.12142 [cs.LG]

    arXiv 2025

  7. [7]

    Bommasani et al

    R. Bommasani et al. On the Opportunities and Risks of Foundation Models. 2021. arXiv: 2108.07258 [cs.LG]

    Pith/arXiv arXiv 2021

  8. [8]

    Stochastic Spectral Descent for Restricted Boltzmann Machines

    D. Carlson, V. Cevher, and L. Carin. “Stochastic Spectral Descent for Restricted Boltzmann Machines”. In: Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics. Ed. by G. Lebanon and S. V. N. Vishwanathan. Vol. 38. Proceedings of Machine Learning Research. San Diego, California, USA: PMLR, 2015, pp. 111–119

    2015

  9. [9]

    Preconditioned Spectral Descent for Deep Learning

    D. E. Carlson et al. “Preconditioned Spectral Descent for Deep Learning”. In: Advances in Neural Information Processing Systems. Vol. 28. 2015, pp. 2971–2979

    2015

  10. [10]

    F. L. Cesista et al. LoRA-Muon: Spectral Steepest Descent on the Low-Rank Manifold . 2026. arXiv: 2606.12921 [cs.LG]

    Pith/arXiv arXiv 2026

  11. [11]

    Chen et al

    L. Chen et al. Punica: Multi-Tenant LoRA Serving. 2023. arXiv: 2310.18547 [cs.DC]

    Pith/arXiv arXiv 2023

  12. [12]

    N. Chen, S. Villar, and S. Hayou. Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning. 2026. arXiv: 2602.06204 [cs.LG]

    arXiv 2026

  13. [13]

    An Exploration of Non-Euclidean Gradient Descent: Muon and its Many Variants

    M. Crawshaw et al. “An Exploration of Non-Euclidean Gradient Descent: Muon and its Many Variants”. In: International Conference on Machine Learning (ICML). 2026

    2026

  14. [14]

    DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

    DeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. 2026. arXiv: 2606.19348 [cs.CL]

    arXiv 2026

  15. [15]

    Du and W

    Z. Du and W. Su. The Newton-Muon Optimizer. 2026. arXiv: 2604.01472 [math.OC]

    arXiv 2026

  16. [16]

    GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models

    GLM-4.5 Team. GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models . 2025. arXiv: 2508.06471 [cs.CL]

    Pith/arXiv arXiv 2025

  17. [17]

    Grattafiori et al

    A. Grattafiori et al. The Llama 3 Herd of Models. 2024. arXiv: 2407.21783 [cs.AI]

    Pith/arXiv arXiv 2024

  18. [18]

    Shampoo: Preconditioned Stochastic Tensor Optimization

    V. Gupta, T. Koren, and Y. Singer. “Shampoo: Preconditioned Stochastic Tensor Optimization”. In: Proceedings of the 35th International Conference on Machine Learning. Vol. 80. Proceedings of Machine Learning Research. PMLR, 2018, pp. 1842–1850. arXiv: 1802.09568 [cs.LG]

    Pith/arXiv arXiv 2018

  19. [19]

    LoRA+: Efficient Low Rank Adaptation of Large Models

    S. Hayou, N. Ghosh, and B. Yu. “LoRA+: Efficient Low Rank Adaptation of Large Models”. In: Proceedings of the 41st International Conference on Machine Learning. Vol. 235. Proceedings of Machine Learning Research. PMLR, 2024, pp. 17783–17806. arXiv: 2402.12354 [cs.LG]

    Pith/arXiv arXiv 2024

  20. [20]

    The Impact of Initialization on LoRA Finetuning Dynamics

    S. Hayou, N. Ghosh, and B. Yu. “The Impact of Initialization on LoRA Finetuning Dynamics”. In: Ad- vances in Neural Information Processing Systems. Ed. by A. Globerson et al. Vol. 37. Curran Associates, Inc., 2024, pp. 117015–117040. 12

    2024

  21. [21]

    Hayou, N

    S. Hayou, N. Ghosh, and B. Yu. PLoP: Precise LoRA Placement for Efficient Finetuning of Large Models

  22. [22]

    LoRA: Low-Rank Adaptation of Large Language Models

    E. J. Hu et al. “LoRA: Low-Rank Adaptation of Large Language Models”. In: International Conference on Learning Representations (ICLR). 2022. arXiv: 2106.09685 [cs.CL]

    Pith/arXiv arXiv 2022

  23. [23]

    Huang, T

    S. Huang, T. Cheng, et al. OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models

  24. [24]

    Janson, E

    P. Janson, E. Oyallon, and E. Belilovsky. Stabilizing Native Low-Rank LLM Pretraining. 2026. arXiv: 2602.12429 [cs.LG]

    Pith/arXiv arXiv 2026

  25. [25]

    Jordan et al

    K. Jordan et al. modded-nanogpt: Speedrunning the NanoGPT Baseline. 2024

    2024

  26. [26]

    Jordan et al

    K. Jordan et al. Muon: An optimizer for hidden layers in neural networks. https://kellerjordan. github.io/posts/muon/. 2024

    2024

  27. [27]

    Kalajdzievski

    D. Kalajdzievski. A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA. 2023. arXiv: 2312. 03732 [cs.LG]

    2023

  28. [28]

    Kaplan et al

    J. Kaplan et al. Scaling Laws for Neural Language Models. 2020. arXiv: 2001.08361 [cs.LG]

    Pith/arXiv arXiv 2020

  29. [29]

    Karpathy

    A. Karpathy. nanochat: The Best ChatGPT that $100 Can Buy. 2025

    2025

  30. [30]

    Keigwin et al

    B. Keigwin et al. Towards Compositional Steepest Descent. https://blog.tilderesearch.com/ blog/compositional-muon. Tilde Research. 2026

    2026

  31. [31]

    Adam: A Method for Stochastic Optimization

    D. P. Kingma and J. Ba. “Adam: A Method for Stochastic Optimization”. In: International Conference on Learning Representations (ICLR). 2015. arXiv: 1412.6980 [cs.LG]

    Pith/arXiv arXiv 2015

  32. [32]

    Low-Rank Kernel Learning with Bregman Matrix Diver- gences

    B. Kulis, M. A. Sustik, and I. S. Dhillon. “Low-Rank Kernel Learning with Bregman Matrix Diver- gences”. In: Journal of Machine Learning Research 10 (2009), pp. 341–376

    2009

  33. [33]

    Beyond zero initialization: Investigating the impact of non-zero initialization on lora fine-tuning dynamics

    S. Li et al. “Beyond zero initialization: Investigating the impact of non-zero initialization on lora fine-tuning dynamics”. In: arXiv preprint arXiv:2505.23194 (2025)

    Pith/arXiv arXiv 2025

  34. [34]

    Li et al

    Y. Li et al. Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifolds. 2026. arXiv: 2605. 09238 [cs.LG]

    2026

  35. [35]

    Li et al

    Z. Li et al. NorMuon: Making Muon More Efficient and Scalable. 2025. arXiv: 2510.05491 [cs.LG]

    arXiv 2025

  36. [36]

    Understanding and Improving Shampoo and SOAP via Kullback–Leibler Minimiza- tion

    W. Lin et al. “Understanding and Improving Shampoo and SOAP via Kullback–Leibler Minimiza- tion”. In: International Conference on Learning Representations (ICLR) . 2026. arXiv: 2509 . 03378 [stat.ML]

    2026

  37. [37]

    Liu et al

    J. Liu et al. BitDelta: Your Fine-Tune May Only Be Worth One Bit. 2024. arXiv: 2402.10193 [cs.LG]

    Pith/arXiv arXiv 2024

  38. [38]

    Muon is scalable for llm training

    J. Liu et al. “Muon is scalable for llm training”. In: arXiv preprint arXiv:2502.16982 (2025)

    Pith/arXiv arXiv 2025

  39. [39]

    J. R. Magnus and H. Neudecker. Matrix Differential Calculus with Applications in Statistics and Econometrics. third. John Wiley & Sons, 2019

    2019

  40. [40]

    Martens and R

    J. Martens and R. Grosse.Optimizing Neural Networks with Kronecker-factored Approximate Curvature

  41. [41]

    2 OLMo 2 Furious

    OLMo Team. 2 OLMo 2 Furious. 2025. arXiv: 2501.00656 [cs.CL]

    Pith/arXiv arXiv 2025

  42. [42]

    Muon Does Not Converge on Convex Lipschitz Functions

    T. Parshakova et al. “Muon Does Not Converge on Convex Lipschitz Functions”. In:arXiv preprint arXiv:2605.08980 (2026)

    Pith/arXiv arXiv 2026

  43. [43]

    Training Deep Learning Models with Norm-Constrained LMOs

    T. Pethick et al. “Training Deep Learning Models with Norm-Constrained LMOs”. In: Proceedings of the 42nd International Conference on Machine Learning. Ed. by A. Singh et al. Vol. 267. Proceedings of Machine Learning Research. PMLR, 2025, pp. 49069–49104. 13

    2025

  44. [44]

    Qiu et al

    S. Qiu et al. Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scales. 2025. arXiv: 2512.05620 [cs.LG]

    arXiv 2025

  45. [45]

    S. Raschka. Practical Tips for Finetuning LLMs Using LoRA (Low-Rank Adaptation) . https : / / magazine.sebastianraschka.com/p/practical-tips-for-finetuning-llms . Ahead of AI. 2023

    2023

  46. [46]

    LoRA Without Regret

    J. Schulman and T. M. Lab. “LoRA Without Regret”. In: Thinking Machines Lab: Connectionism (2025). https://thinkingmachines.ai/blog/lora/

    2025

  47. [47]

    Schulman et al

    J. Schulman et al. LoRA Without Regret. Thinking Machines Lab, Connectionism blog. 2025

    2025

  48. [48]

    Semenov, M

    A. Semenov, M. Pagliardini, and M. Jaggi. Benchmarking Optimizers for Large Language Model Pretraining. 2025. arXiv: 2509.01440 [cs.LG]

    Pith/arXiv arXiv 2025

  49. [49]

    Shazeer and M

    N. Shazeer and M. Stern. Adafactor: Adaptive Learning Rates with Sublinear Memory Cost. 2018. arXiv: 1804.04235 [cs.LG]

    Pith/arXiv arXiv 2018

  50. [50]

    A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks at-Scale

    H.-J. M. Shi et al. “A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks at-Scale”. In: arXiv preprint arXiv:2309.06497 (2023)

    Pith/arXiv arXiv 2023

  51. [51]

    Singh, F

    S. Singh, F. Vargus, D. Dsouza, et al.Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning. 2024. arXiv: 2402.06619 [cs.CL]

    Pith/arXiv arXiv 2024

  52. [52]

    Toshniwal et al

    S. Toshniwal et al. OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data. 2024. arXiv: 2410.01560 [cs.CL]

    Pith/arXiv arXiv 2024

  53. [53]

    Vyas et al

    N. Vyas et al. SOAP: Improving and Stabilizing Shampoo using Adam . 2024. arXiv: 2409 . 11321 [cs.LG]

    2024

  54. [54]

    Wen et al

    K. Wen et al. Fantastic Pretraining Optimizers and Where to Find Them. 2025. arXiv: 2509.02046 [cs.LG]

    Pith/arXiv arXiv 2025

  55. [55]

    Yang et al

    A. Yang et al. Qwen2.5 Technical Report. 2024. arXiv: 2412.15115 [cs.CL]

    Pith/arXiv arXiv 2024

  56. [56]

    G. Yang, J. B. Simon, and J. Bernstein. A Spectral Condition for Feature Learning. 2023. arXiv: 2310. 17813 [cs.LG]

    2023

  57. [57]

    LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Op- timization

    J.-N. Yen et al. “LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Op- timization”. In: International Conference on Learning Representations . 2025. arXiv: 2410 . 20625 [cs.LG]

    2025

  58. [58]

    Zhang et al

    J. Zhang et al. Gram Newton–Schulz: A Fast, Hardware-Aware Newton–Schulz Algorithm for Muon. https://tridao.me/blog/2026/gram-newton-schulz/. 2026

    2026

  59. [59]

    Zhang et al

    Y. Zhang et al. Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioning. 2026. arXiv: 2603.09697 [cs.LG]. 14 Appendix Contents A Experimental Details 16 B Proofs 17 C Momentum-Free Reductions to Product Muon 19 D Estimating the Preconditioners 21 E Implementation Details 23 E.1 Spectral-Norm Estimation . . . . . . . . . . . . . . . . ...

    arXiv 2026

  60. [63]

    Leverage bound: ⟨Σ†gi, gi⟩ ≤n,

  61. [64]

    escaped-mass

    Rank one: Gi = bix⊤ i for some xi ∈ Rdin and bi ∈ Rdout. Proof of Lemma 1. For the Leverage bound, let gi = vec( Gi) and M = [ g1, . . . , gn], so gi = M ei for every i = 1, . . . , n. Then ⟨Σ†gi, gi⟩ = ng⊤ i (M M⊤)†gi = ne⊤ i M ⊤(M M⊤)†M ei ≤ n, since M ⊤(M M⊤)†M is the orthogonal projector onto the range of M ⊤. The Rank one property is the chain rule: ...

    2000

  62. [2015]

    arXiv: 1503.05671 [cs.LG]

    Pith/arXiv arXiv

  63. [2024]

    arXiv: 2411.04905 [cs.CL]

    Pith/arXiv arXiv

  64. [2025]

    arXiv: 2506.20629 [cs.LG]

    Pith/arXiv arXiv