Stabilizing Native Low-Rank LLM Pretraining

Edouard Oyallon; Eugene Belilovsky; Paul Janson

REVIEW 4 major objections 4 minor 4 cited by

This paper claims that full-rank weights are unnecessary for LLM pretraining: a 454M low-rank transformer matches a 780M dense transformer at equal compute once factor updates are adaptively scaled by the factors' spectral norms.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 23:47 UTC pith:ZHU2NH44

load-bearing objection A real, simple to state method for native low-rank pretraining, with a spectral-norm instability story that mostly holds up, but the equal-FLOPs comparisons and the strictness of the bound need work. the 4 major comments →

arxiv 2602.12429 v2 pith:ZHU2NH44 submitted 2026-02-12 cs.LG

Stabilizing Native Low-Rank LLM Pretraining

Paul Janson , Edouard Oyallon , Eugene Belilovsky This is my paper

classification cs.LG

keywords low-rank factorizationLLM pretrainingspectral norm controlgradient orthogonalizationtraining stabilityfactorized transformerscompute-optimal scaling lawsIsoFLOP analysis

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Training a transformer from scratch in which every non-embedding weight is a low-rank product W = AB^T has reliably failed before: updates to A and B independently leave the spectral norm (largest singular value) of the composite update ΔW free to explode, producing loss spikes and divergence. The paper's central claim is that this instability has one dominant, fixable cause, and that a cheap update rule removes it: orthogonalize each factor's momentum update, estimate the spectral norms σ_A and σ_B by a single power iteration, and scale the shared update radius by 1/(σ_A + σ_B + 1). With this rule the paper reports stable end-to-end factored training with sub-1% overhead, a 454M factorized transformer that matches a 780M dense transformer at equal training FLOPs, and compute-optimal scaling laws (N_opt ∝ C^0.479, D_opt ∝ C^0.521) that mildly favor smaller, more token-hungry models than the dense Chinchilla allocation. If correct, full-rank weights are an optimization convenience rather than a capacity requirement, and pretrained models become about 40% cheaper to serve at matched quality.

Core claim

The discovery is that native low-rank pretraining fails for a controllable reason: the factorization W = AB^T has a scaling invariance, (λA)(λ^{-1}B) = W, so independent gradient updates of A and B let the largest singular values of the factors — and hence of the composite update ΔW = ΔA B^T + A ΔB^T + ΔA ΔB^T — grow without bound, amplifying the RMS change in activations. Spectron stops exactly this. It orthogonalizes the momentum updates of both factors (Newton–Schulz iterations, so ∥ΔA∥_2 and ∥ΔB∥_2 are at most ρ), estimates σ_A and σ_B with one power iteration, and sets ρ = η/(σ_A + σ_B + 1); submultiplicativity then yields ∥ΔW∥_2 ≤ η, the promised 'provable' bound. Empirically the metho

What carries the argument

Spectron — spectral renormalization with orthogonalization. The load-bearing bound is derived from the update identity ΔW = ΔA B^T + A ΔB^T + ΔA ΔB^T: with orthogonalized factor updates satisfying ∥ΔA∥_2, ∥ΔB∥_2 ≤ ρ, the triangle inequality and submultiplicativity give ∥ΔW∥_2 ≤ ρ(σ_A + σ_B + 1), so choosing ρ = η/(σ_A + σ_B + 1) enforces the target constraint ∥ΔW∥_2 ≤ η with η the learning rate. The two operational pieces are Newton–Schulz orthogonalization (about five matrix-multiply iterations that map a momentum matrix toward an orthogonal one) and a one-step power iteration whose Rayleigh quotient estimates σ_A and σ_B; the paper reports their combined cost as under 1% of training FLOPs,

Load-bearing premise

The guarantee that ∥ΔW∥_2 ≤ η holds only if the single power iteration returns spectral norms at least as large as the true σ_max(A) and σ_max(B); a one-step Rayleigh quotient is a lower bound, so an underestimate would inflate the radius ρ = η/(σ_A + σ_B + 1) and silently void the promised bound — a gap the paper does not analyze when it calls the update 'provably limiting.'

What would settle it

At a checkpoint of a Spectron run, compute the exact largest singular values of A and B by full SVD and reconstruct the applied update: if the true ∥ΔW∥_2 systematically exceeds η (the learning rate) or fluctuates an order of magnitude above it, the central bound is violated in practice. A cleaner causal test: run the identical algorithm but replace the one-step power iteration with several iterations or exact SVDs, which can only shrink ρ; if training then destabilizes or fails to match dense performance, the claimed spectral-norm bound is not the mechanism actually keeping native low-rank tr

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Full-rank guidance is unnecessary: the previously state-of-the-art self-guided recipe, which pays roughly 25% extra FLOPs to keep a dense copy alongside the factors, is dominated by Spectron at every tested scale (94M, 297M, 454M factorized).
Equal-compute parity transfers to inference savings: a 454M factorized model matching a 780M dense model means roughly 42% lower inference FLOPs at the same quality.
Stable high-learning-rate training: at η = 0.01, where naive AdamW training of factorized models diverges, Spectron converges smoothly, because bounding ∥ΔW∥_2 bounds the RMS change in activations via |Δy|_rms ≤ sqrt(n/m) ∥ΔW∥_2.
Predictable scaling: compute-optimal factored models follow power laws (N_opt ∝ C^0.479, D_opt ∝ C^0.521 by IsoFLOP; 0.45/0.55 by the parametric fit), so compute can be allocated between factored parameters and tokens with the same quantitative machinery used for dense models.
Factored models beat parameter-matched dense baselines in perplexity and downstream accuracy at equal compute across three tested scales, indicating the trade-off is not a simple capacity loss.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the spectral-norm diagnosis is the true driver, Spectron is one member of a design family: any mechanism that caps the factor spectral norms — explicit singular-value clipping, shared-scale parameterizations that remove the λ-invariance, or a few extra power iterations — should stabilize native low-rank training; swapping the one-step estimator for exact SVDs would isolate whether the stated bo
The scaling exponents extrapolate beyond the tested range: at C ≈ 10^26 FLOPs the fitted curves imply the compute-optimal factored model is roughly half the size of the optimal dense model, suggesting that if inference cost matters, large-budget frontier training should migrate toward factored, token-heavy configurations.
The paper's own comparison (perplexity below parameter-matched dense baselines) suggests the isocost surface is not flat — factored models may extract more quality per FLOP during training than dense ones — which, if confirmed at larger scales, would revise the usual assumption that factorization is a capacity tax paid for memory savings.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A real, simple to state method for native low-rank pretraining, with a spectral-norm instability story that mostly holds up, but the equal-FLOPs comparisons and the strictness of the bound need work.

read the letter

The new thing here is genuine: training LLMs from scratch with every non-embedding matrix low-rank and no full-rank auxiliary weights is something prior methods (Wei, Huh, Wang) all dodged. Spectron is a simple intervention — orthogonalize the factor updates, estimate spectral norms of the factors, and shrink the update radius accordingly. The ablation in Table 2 shows both pieces matter, and the comparison against self-guided training is fair: Spectron beats it at all three scales without the 25% FLOP overhead. The stability story is plausible and the spectral norm plots in Figures 2–3 give it visual support. If the method is as stable and cheap as reported, it is a practical step forward for memory-limited training and inference-heavy deployment.

Now the soft spots, in rough order of seriousness.

First, the equal-FLOPs comparison is confounded. The dense 780M model is trained on about 9.1B tokens while the low-rank 454M model gets 15.6B tokens at the same compute. Under the paper's own Chinchilla reference, a dense ~0.5B model trained on ~13–14B tokens is the compute-optimal allocation for that budget; the dense 780M run sits clearly off the frontier. So the headline "low-rank matches dense at equal compute" is partly a small-model, longer-training effect, not necessarily a low-rank advantage. The same issue runs through Figure 6. This doesn't kill the method, but it means the inference-efficiency claim is overstated without comparing against a properly tuned dense baseline.

Second, the spectral norm guarantee in Section 4 is not strict. Algorithm 3 uses one power iteration, which returns a lower bound on the true spectral norm. If σ_A and σ_B are underestimated, ρ in Eq. 16 is too large, and the claimed ∥ΔW∥₂ ≤ η does not actually follow. The paper calls the method "provably limiting" without addressing this gap. It is probably fixable with more power iterations or a careful error analysis, but as written the proof has a hole.

Third, the scaling-law contributions are weaker than the abstract suggests. The exponents N∝C^0.479 and D∝C^0.521 are fitted to the same IsoFLOP curves they are then used to "predict" from; that is not circular in the ordinary sense — it is an empirical summary — but the extrapolation to C≈10^26 for the claimed 50% inference saving is a 0.011 exponent difference carried over seven orders of magnitude with no uncertainty. That is fragile and should be flagged.

Finally, no code, no seeds, no error bars. For a paper whose claims are empirical, that is a real reproducibility gap, not a nitpick.

Overall: the central stability mechanism and the method itself deserve attention. The paper is a serious empirical contribution in need of revision, not a desk reject. I would send it to peer review, with the request that the authors provide code and seeds, rerun the dense comparisons at compute-optimal allocations, and correct or qualify the spectral-norm bound.

Referee Report

4 major / 4 minor

Summary. The paper proposes Spectron, an optimizer-level modification for training transformers in which all non-embedding weight matrices are parameterized as low-rank products W=AB^T from random initialization. The method combines Muon-style gradient orthogonalization with an adaptive update radius rho = eta/(sigma_A+sigma_B+1), where sigma_A, sigma_B are power-iteration estimates of the factor spectral norms, intended to enforce the bound ||Delta W||_2 <= eta. Empirically, the authors report that Spectron-trained factorized models (94M, 297M, 454M) outperform naive AdamW and self-guided baselines, match or beat dense models at matched FLOPs, and exhibit IsoFLOP-optimal scaling laws N_opt ~ C^0.479, D_opt ~ C^0.521, from which they extrapolate large inference savings at scale.

Significance. If the central claims held, the paper would make a meaningful contribution: a simple, stable recipe for native low-rank pretraining that removes the auxiliary full-rank weights used by prior methods, with potential memory and inference savings. The paper is also honest in reporting training curves, hyperparameters, and ablations, and it provides 39 IsoFLOP runs. However, the headline equal-FLOPs parity is confounded by an off-optimal dense baseline, the stability guarantee is not strict because the spectral estimates are lower bounds, and the scaling-law inference-savings conclusion rests on an unquantified extrapolation. The core idea is plausible and worth pursuing, but the current evidence does not support the strong claims as stated.

major comments (4)

[Section 4, Eq. (16), Algorithm 3] The claimed bound ||Delta W||_2 <= eta is not strict. With kpower=1, Algorithm 3 returns the Rayleigh quotient sigma = u^T W v for a fixed vector u, which is a lower bound on sigma_max(W). Substituting these lower bounds into the denominator of rho = eta/(sigma_A+sigma_B+1) makes rho larger than the value required by Eq. (15) using the true spectral norms, so the guarantee in Eq. (11) does not follow. The text calls the method 'provably limiting' but provides no error analysis or upper-bound estimator. The authors should either use provable upper-bound estimates for sigma_max (e.g., a few power iterations with a certificate, or a randomized norm estimator with an explicit upper bound) or explicitly soften the claim to an empirical one.
[Section 5.2, Figures 1 and 5, Appendix E.3] The central equal-FLOPs comparison is confounded by model-size versus token-allocation. Dense-L (780M) trains for 8,657 steps (~9.1B tokens) and low-rank-L (454M) for 14,878 steps (~15.6B tokens), both at 1.048M tokens/step, so C ~ 4.2e19 FLOPs. Under the paper's own Chinchilla reference, the compute-optimal dense model for this C has roughly 0.5-0.6B parameters and ~12-14B tokens. The dense 780M/9.1B run is far from that frontier, while the low-rank 454M/15.6B run is close to it. The parity in Figures 1 and 5 may therefore simply reflect the known small-model/longer-training tradeoff, not a low-rank advantage. The same issue affects Figure 6's comparisons of dense 134M/500M/780M with low-rank 94M/297M/454M. A dense compute-optimal baseline (~0.5B, ~13B tokens) trained under identical conditions is needed to support the inference-efficiency claim.
[Section 6, Figure 8, Appendix D] The scaling-law inference-savings claim is an extrapolation without uncertainty. The exponents N_opt ~ C^0.479 and D_opt ~ C^0.521 are fitted to four IsoFLOP budgets spanning only 2.2e18-3.57e19 FLOPs, yet Figure 8 (right) predicts ~50% inference savings at C~1e26, seven orders of magnitude beyond the data. The parametric fit in Appendix D yields exponents 0.45 and 0.55, showing substantial sensitivity to the fitting approach, and no confidence intervals are reported for A, alpha, B, beta, E, or the exponents. Moreover, the comparison is against Chinchilla coefficients fitted to dense models on different data, tokenizers, and optimizers; a 0.011 exponent difference is within plausible fit error. The authors should report uncertainties and provide same-setting dense IsoFLOP fits before making quantitative claims.
[Section 5, Tables 1 and 3, Figures 1,4-7] No seeds, error bars, or significance tests are reported. Headline numbers such as the 6-12% perplexity improvement over self-guided training, the parity with dense models in Figure 1, and the rank-ratio sensitivity in Table 3 all come from single runs with hyperparameter sweeps. Since the paper's central claim is about training stability and reproducible gains, at least three seeds (or a bootstrap over the validation set) are needed to distinguish systematic improvement from run-to-run variance.

minor comments (4)

[Section 4, Eq. (15)] The step in Eq. (15) uses the bound rho^2 <= rho when rho < 1, but no justification is given for rho < 1. In practice rho can exceed 1 for small spectral norms (e.g., early training or warmup). The condition should be stated explicitly, or the exact quadratic bound should be used.
[Appendix A, Algorithm 2] The pseudocode for the transpose branch of Newton-Schulz is confusing: after setting X_1 <- X_1^T for m > n, the loop computes Gram matrices on the transposed matrix and then restores orientation. This is likely correct, but the notation 'X_{kns+1} <- X_{kns+1}^T' mixes iteration indices and may lead to off-by-one confusion. Please clean up the variable indexing.
[Figure 8 (right)] The estimated inference savings formula (1 - 1/C^0.011)*100% compares the two scaling exponents while silently dropping the proportionality constants of the N_opt laws. If the constants differ between low-rank and dense, the asymptotic comparison is incomplete; please include the fitted constants or state why they cancel.
[Appendix E.3] Hyperparameters are reported, but no code or data-release statement is given. Providing the code (or at least a detailed configuration file) would materially improve reproducibility, especially given the unusual optimizer and the lack of public implementation for some baselines.

Circularity Check

1 steps flagged

The inference-efficiency advantage in Section 6 is an algebraic restatement of exponents fitted on the paper's own IsoFLOP curves; the central Spectron stability claim is empirically tested and not circular.

specific steps

fitted input called prediction [Section 6, Figure 8 (right panel caption)]
"Estimated inference cost savings, computed as (1−N opt/NChinchilla)×100 = (1−1/C 0.011)×100%, assuming inference cost scales as 2Nopt ·D inf with identical proportionality constants for both low-rank and dense models. Under contemporary FLOP budgets for training (∼10 26 FLOPs), low-rank models achieve up to 50% inference cost reduction compared to Chinchilla-optimal dense transformers."

The exponent 0.479 in Nopt ∝ C^0.479 is obtained by quadratic fits to the paper's own four IsoFLOP curves (Section 6, Figure 9). The claimed savings (1 − 1/C^0.011) is the algebraic transform of the difference between this fitted exponent and the Chinchilla 0.49 exponent; no held-out data or independent measurement is involved. Thus the 'estimated inference saving' is not a prediction derived from the method but a restatement of the fitted coefficient, making the extrapolated inference-efficiency conclusion reduce by construction to the fit. The same issue extends to the D∝C^0.521 token exponent, which is also a fitted value rather than an independently validated scaling law.

full rationale

The central contribution, Spectron, is not circular: the paper identifies spectral-norm growth as a hypothesized instability mechanism, introduces a constraint that bounds ∥ΔW∥₂ by design, and then tests the hypothesis through loss curves, ablations, and comparisons against dense and self-guided baselines. The spectral-norm bound (Eq. 16) is a construction whose guarantee would depend on exact norms; the single power-iteration estimate creates a correctness gap, but that is a numerical issue, not a circular derivation. Self-citations (Nabli et al. 2025, Rivaud et al.) appear only as background on parallelization and are not load-bearing. The one circular-adjacent element is the compute-optimal scaling-law section: the inference-efficiency advantage is a direct algebraic consequence of exponents fitted on the same IsoFLOP data, so it should be read as a fit/extrapolation rather than an independent prediction. This does not undermine the empirical demonstration that Spectron stabilizes native low-rank pretraining, hence a moderate score of 4 rather than a higher one.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central claim rests on standard matrix norm inequalities plus two domain assumptions (power iteration and Newton-Schulz accuracy) and one paper-specific stability hypothesis. The scaling-law exponents are fitted quantities, not first-principles derivations. No new physical or architectural entities are introduced.

free parameters (5)

Rank ratio r = 0.25n = 0.25
Chosen by hand from the ablation in Appendix B.2 as a trade-off between parameter reduction and performance; used in all main experiments.
Parametric scaling-law coefficients (A, alpha, B, beta, E) = A=1000, alpha=0.398, B=1000, beta=0.332, E=1.777
Fitted to all 39 IsoFLOP runs via Huber loss in Appendix D; used to derive N_opt ∝ C^0.45 and D_opt ∝ C^0.55.
IsoFLOP exponents = 0.479 (N) and 0.521 (D)
Obtained from quadratic fits to the four IsoFLOP curves in Section 6, then used to claim inference savings.
Newton-Schulz coefficients (a,b,c) = 3.4445, -4.7750, 2.0315
Taken from prior Muon work; used as fixed constants in Algorithm 2.
Algorithm hyperparameters = k_power=1, k_ns=5, momentum beta=0.95
Choices stated in Algorithm 1 and used for all experiments; not fitted to target results but affect the method's behavior.

axioms (5)

standard math Spectral norm submultiplicativity: ∥XY∥₂ ≤ ∥X∥₂∥Y∥₂
Used in Eq. 13–14 to bound the composite update.
domain assumption One-step power iteration yields a usable estimate of the spectral norm of A and B
Algorithm 3 returns a Rayleigh quotient, which is a lower bound on σ_max; the paper treats it as if it were the true norm in Eq. 16.
domain assumption Newton-Schulz with 5 iterations produces approximately orthogonal updates
Algorithm 2 is claimed to give O_A ≈ U V^⊤; residual errors are not quantified.
domain assumption The scaling-law functional form L(N,D) = E + A/N^α + B/D^β
Adopted from Hoffmann et al. and used in Appendix D to fit the data.
ad hoc to paper Bounding ∥ΔW∥₂ ≤ η is sufficient to prevent training instability
This is the central hypothesis of the paper; motivating the method. It is validated empirically, not proven.

pith-pipeline@v1.3.0-alltime-deepseek · 19090 in / 13819 out tokens · 118007 ms · 2026-08-02T23:47:45.474721+00:00 · methodology

0 comments

read the original abstract

Foundation models have achieved remarkable success, yet their growing parameter counts pose significant computational and memory challenges. Low-rank factorization offers a promising route to reduce training and inference costs, but the community lacks a stable recipe for training models from scratch using exclusively low-rank weights while matching the performance of the dense model. We demonstrate that Large Language Models (LLMs) can be trained from scratch using exclusively low-rank factorized weights for all non-embedding matrices without auxiliary "full-rank" guidance required by prior methods. While native low-rank training often suffers from instability and loss spikes, we identify uncontrolled growth in the spectral norm (largest singular value) of the weight matrix update as the dominant factor. To address this, we introduce Spectron: Spectral renormalization with orthogonalization, which dynamically bounds the resultant weight updates based on the current spectral norms of the factors. Our method enables stable, end-to-end factorized training with negligible overhead. Finally, we establish compute-optimal scaling laws for natively low-rank transformers, demonstrating predictable power-law behavior and improved inference efficiency relative to dense models.

Figures

Figures reproduced from arXiv: 2602.12429 by Edouard Oyallon, Eugene Belilovsky, Paul Janson.

**Figure 1.** Figure 1: Natively Low-Rank Training Achieves Dense-Level Performance. Validation loss curves comparing a 780M dense Transformer (Vaswani et al., 2017)(red) against our 454M low-rank factorized Transformer (blue) across 3.5 × 106 training TFLOPs on FineWeb (Penedo et al., 2024). Our method Spectron enables stable end-to-end factorized training that matches dense performance at equal compute, yielding an inference-o… view at source ↗

**Figure 2.** Figure 2: Low-Rank Parameterization Destabilizes Spectral Norm Dynamics. Weight update spectral norm (∥∆W∥2 ) comparison between low-rank (green) and dense (gray) AdamW (Kingma & Ba, 2015) training on layer 4 attention output projection of a Transformer (Vaswani et al., 2017). Dense training maintains stable, bounded spectral norms, while low-rank factorization exhibits 10-30× higher spectral norm magnitudes, reve… view at source ↗

**Figure 3.** Figure 3: Spectral Norm Constraints Stabilize Low-Rank Training. Comparison of (a) weight update spectral norm ∥∆W∥2 , (b) activation RMS change |∆y| rms , and (c) weight spectral norm ∥W∥2 across 8000 training steps for layer 4 attention output projection of a 94M parameter Factorized Transformer (Vaswani et al., 2017). AdamW (Kingma & Ba, 2015) (green, left axis) exhibits explosive growth in all metrics with uncon… view at source ↗

**Figure 4.** Figure 4: Spectrally Normalized Low-Rank Training Outperforms Baselines. Validation loss on FineWeb (Penedo et al., 2024) held-out set during Factorized Transformer-M (297M) pretraining comparing Spectron (blue), self-guided training (red), and naive AdamW (green). Our approach achieves both faster initial convergence and superior final performance ( [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗

**Figure 5.** Figure 5: Low-Rank Factorization Matches Dense Performance with Longer Training. Validation loss comparison between Dense Transformer-L (780M parameters) and our Low-Rank Factorized Transformer-L (454M parameters) trained for equal FLOPs by matching training steps. Despite a ∼ 42% parameter reduction, our factorized model (blue) converges to the same final validation loss as the dense baseline (red), demonstrating t… view at source ↗

**Figure 6.** Figure 6: reveals that factorized models achieve consistently lower perplexity than parameter-matched dense baselines. This yields a more compact model for a given perplexity threshold, thereby substantially reducing inference costs [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗

**Figure 7.** Figure 7: extends these findings to downstream evaluation, where factorized transformers match or exceed dense performance across HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), and ARC Easy (Clark et al., 2018) benchmarks. 6. Towards compute optimal Low-Rank Pretraining Having established performance parity with dense models under equal compute (Section 5), we investigate the fundamental scaling prope… view at source ↗

**Figure 8.** Figure 8: Factorized Models Scale More Conservatively Than Dense Transformers, Yielding Substantial Inference Efficiency Gains. (Left) Optimal model size versus compute budget for low-rank architectures follows Nopt ∝ C 0.479 (blue points), compared to Chinchilla’s N ∝ C 0.49 (green dashed reference). (Center) Optimal training tokens scale as Dopt ∝ C 0.521 (red points), versus Chinchilla’s D ∝ C 0.51. The reduced p… view at source ↗

**Figure 9.** Figure 9: Low-Rank Architectures Exhibit Clear ComputeOptimal Model Sizes. Validation loss versus parameter count for factorized transformers trained at four compute budgets (2.20e+18 to 3.57e+19 FLOPs). Each IsoFLOP curve displays a distinct minimum (vertical dashed lines), with optimal model size increasing monotonically with compute budget. This replicates the fundamental structure of Chinchilla scaling laws (… view at source ↗

**Figure 10.** Figure 10: Ablation Study of Orthogonalization and Spectral Renormalization Components. Validation loss curves over 1,800 training steps for a Factorized Transformer-S (94M) model trained with different combinations of our method’s core components: combined orthogonalization and spectral renormalization (blue), orthogonalization only (red), spectral renormalization only (green), and naive SGD baseline (orange). Whil… view at source ↗

**Figure 11.** Figure 11: Effect of Rank Ratio in Low-Rank Factorized Training. Validation loss curves across 1,800 optimization steps for Factorized Transformer-S(94M) trained with different rank ratios under our method. Models with rank ratios 0.4 (blue) and 0.25 (red ) exhibit nearly identical convergence dynamics with 0.4 being slightly better. In contrast, aggressive compression to rank ratio 0.125 (green ) results in fundame… view at source ↗

**Figure 12.** Figure 12: Higher Learning Rates Destabilize Factorized Training in Naive training. Validation loss curves comparing different optimization methods for low-rank Factorized Transformer-S across learning rates η ∈ {0.001, 0.01}. Naive AdamW (orange ) diverges catastrophically at η = 0.01 and converges slowly at η = 0.001, demonstrating that standard optimizers cannot support aggressive learning rates for factorized mo… view at source ↗

**Figure 13.** Figure 13: Spectron Outperforms Baselines when only fully connected layers factorized to low rank Validation loss curves comparing Spectron (blue) against self-guided learning (red ) and naive AdamW training (green ) when applying low-rank factorization exclusively to feedforward layers. Spectron achieves lower validation loss throughout training and converges faster, surpassing self-guided learning even within the … view at source ↗

**Figure 14.** Figure 14: Training curves across different experimental settings 20 [PITH_FULL_IMAGE:figures/full_fig_p020_14.png] view at source ↗

discussion (0)

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifolds
cs.LG 2026-05 unverdicted novelty 7.0

Intrinsic Muon provides closed-form linear maximization oracles on multiple Riemannian matrix manifolds for unitarily invariant norms, with convergence rates depending only on manifold dimension or rank.
PoLoRA: A Preconditioned Orthogonalized LoRA Optimizer
cs.LG 2026-07 conditional novelty 6.0

A preconditioned, product-aware spectral optimizer for LoRA reaches Adam's final loss in 1.2–1.7x fewer steps on instruction-tuning, with ~3% per-step overhead.
SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales
cs.LG 2026-07 conditional novelty 6.0

SOAP and Muon, stabilized by per-step QR eigenbasis updates and KL-Shampoo covariance accumulation, beat AdamW on large-batch LLM pretraining up to 100M-token batches.
LoRA-Muon: Spectral Steepest Descent on the Low-Rank Manifold
cs.LG 2026-06 unverdicted novelty 6.0

LoRA-Muon applies Muon's spectral steepest descent to low-rank factors with split weight decay, acting as a transferable proxy for full-rank Muon and Shampoo optimizers.

Reference graph

Works this paper leans on

5 extracted references · cited by 4 Pith papers

[4]

optimizer. B.2. Effect of Rank Ratio Figure 11 and Table 3 examine sensitivity to the rank ratio. We defined rank ratio as the multiple used to set the low rank r using the input dimension n of a matrix sized m×n . Rank ratios of 0.25×n and 0.4×n achieve comparable performance 14 Stabilizing Native Low-Rank LLM Pretraining 0 200 400 600 800 1000 1200 1400...

2024
[2015]

For Muon (Jordan et al., 2024) experiments we use 0.95 for momentum

is used we useβ 1 andβ 2 as 0.9 and 0.95. For Muon (Jordan et al., 2024) experiments we use 0.95 for momentum. Following Wei et al. (2024a), we initialize low-rank models using spectral initialization (Khodak et al., 2021) while applying standard decoupled weight decay regularization (Loshchilov & Hutter, 2019). All models are trained on the Fineweb (Pene...

2024
[2017]

Si, C., Zhang, D., and Shen, W

URL https://openreview.net/forum? id=B1ckMDqlg. Si, C., Zhang, D., and Shen, W. Adamuon: Adaptive muon optimizer.arXiv preprint arXiv:2507.11005, 2025. Su, J., Ahmed, M., Lu, Y ., Pan, S., Bo, W., and Liu, Y . Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024. Team, K., Bai, Y ., Bao, Y ., Chen, G., Chen, J., C...

arXiv 2025
[2024]

10 Stabilizing Native Low-Rank LLM Pretraining Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J

URL https://openreview.net/forum? id=ozX92bu8VA. 10 Stabilizing Native Low-Rank LLM Pretraining Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. InInternational Conference on Learning Representations,
[2025]

Nocedal, J

URL https://openreview.net/forum? id=1qKUVyymXs. Nocedal, J. Updating quasi-newton matrices with limited storage.Mathematics of computation, 35(151):773–782, 1980. Penedo, G., Kydl´ıˇcek, H., Lozhkov, A., Mitchell, M., Raffel, C. A., V on Werra, L., Wolf, T., et al. The fineweb datasets: Decanting the web for the finest text data at scale.Ad- vances in Ne...

1980

[1] [4]

optimizer. B.2. Effect of Rank Ratio Figure 11 and Table 3 examine sensitivity to the rank ratio. We defined rank ratio as the multiple used to set the low rank r using the input dimension n of a matrix sized m×n . Rank ratios of 0.25×n and 0.4×n achieve comparable performance 14 Stabilizing Native Low-Rank LLM Pretraining 0 200 400 600 800 1000 1200 1400...

2024

[2] [2015]

For Muon (Jordan et al., 2024) experiments we use 0.95 for momentum

is used we useβ 1 andβ 2 as 0.9 and 0.95. For Muon (Jordan et al., 2024) experiments we use 0.95 for momentum. Following Wei et al. (2024a), we initialize low-rank models using spectral initialization (Khodak et al., 2021) while applying standard decoupled weight decay regularization (Loshchilov & Hutter, 2019). All models are trained on the Fineweb (Pene...

2024

[3] [2017]

Si, C., Zhang, D., and Shen, W

URL https://openreview.net/forum? id=B1ckMDqlg. Si, C., Zhang, D., and Shen, W. Adamuon: Adaptive muon optimizer.arXiv preprint arXiv:2507.11005, 2025. Su, J., Ahmed, M., Lu, Y ., Pan, S., Bo, W., and Liu, Y . Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024. Team, K., Bai, Y ., Bao, Y ., Chen, G., Chen, J., C...

arXiv 2025

[4] [2024]

10 Stabilizing Native Low-Rank LLM Pretraining Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J

URL https://openreview.net/forum? id=ozX92bu8VA. 10 Stabilizing Native Low-Rank LLM Pretraining Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. InInternational Conference on Learning Representations,

[5] [2025]

Nocedal, J

URL https://openreview.net/forum? id=1qKUVyymXs. Nocedal, J. Updating quasi-newton matrices with limited storage.Mathematics of computation, 35(151):773–782, 1980. Penedo, G., Kydl´ıˇcek, H., Lozhkov, A., Mitchell, M., Raffel, C. A., V on Werra, L., Wolf, T., et al. The fineweb datasets: Decanting the web for the finest text data at scale.Ad- vances in Ne...

1980