{"total":23,"items":[{"citing_arxiv_id":"2606.30509","ref_index":33,"ref_count":1,"confidence":0.9,"is_internal_anchor":false,"paper_title":"Muon learns balanced solutions in matrix factorization without slow saddle-to-saddle dynamics","primary_cat":"cs.LG","submitted_at":"2026-06-29T16:17:38+00:00","verdict":"UNVERDICTED","verdict_confidence":"LOW","novelty_score":6.0,"formal_verification":"none","one_line_summary":"Muon in matrix factorization avoids saddle-to-saddle dynamics, learns top modes simultaneously, conserves sqrt(P^TP) - sqrt(Q^TQ), and reaches balanced solutions from small initialization with a two-step alignment schedule.","context_count":0,"top_context_role":null,"top_context_polarity":null,"context_text":null},{"citing_arxiv_id":"2606.09658","ref_index":134,"ref_count":1,"confidence":0.9,"is_internal_anchor":false,"paper_title":"Muon Learns More Robust and Transferable Features than Adam","primary_cat":"cs.LG","submitted_at":"2026-06-08T15:42:54+00:00","verdict":"UNVERDICTED","verdict_confidence":"LOW","novelty_score":5.0,"formal_verification":"none","one_line_summary":"Muon learns more robust and transferable features than Adam and SGD, shown via corruption robustness tests, transfer experiments, layer-wise probes, effective rank measurements, and a theoretical proof on margins in a multi-component classification problem.","context_count":0,"top_context_role":null,"top_context_polarity":null,"context_text":null},{"citing_arxiv_id":"2606.06470","ref_index":109,"ref_count":1,"confidence":0.9,"is_internal_anchor":false,"paper_title":"PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training","primary_cat":"cs.LG","submitted_at":"2026-06-04T17:55:11+00:00","verdict":"UNVERDICTED","verdict_confidence":"LOW","novelty_score":5.0,"formal_verification":"none","one_line_summary":"A polynomial preconditioning layer controls singular value spectra of transformer weights to stabilize pre-training, shown effective on Llama-1B and supported by convergence theory for deep linear networks.","context_count":0,"top_context_role":null,"top_context_polarity":null,"context_text":null},{"citing_arxiv_id":"2606.06418","ref_index":107,"ref_count":1,"confidence":0.9,"is_internal_anchor":false,"paper_title":"Double Preconditioning (DoPr): Optimization for Test-Time Performance, not Validation Loss","primary_cat":"cs.LG","submitted_at":"2026-06-04T17:22:58+00:00","verdict":"UNVERDICTED","verdict_confidence":"LOW","novelty_score":6.0,"formal_verification":"none","one_line_summary":"Double preconditioning (DoPr) improves downstream task performance in test-time feedback settings without consistent gains in validation loss.","context_count":0,"top_context_role":null,"top_context_polarity":null,"context_text":null},{"citing_arxiv_id":"2606.05484","ref_index":62,"ref_count":1,"confidence":0.9,"is_internal_anchor":false,"paper_title":"Learned Subspace Compression for Communication-Efficient Pipeline Parallelism","primary_cat":"cs.LG","submitted_at":"2026-06-03T22:10:16+00:00","verdict":"UNVERDICTED","verdict_confidence":"LOW","novelty_score":6.0,"formal_verification":"none","one_line_summary":"MAPL learns task-specific orthogonal compression subspaces per pipeline stage via manifold-constrained optimization and recovers signals with low-overhead anchors, yielding better compression-performance tradeoffs than fixed projections on LLaMA models up to 1B parameters.","context_count":0,"top_context_role":null,"top_context_polarity":null,"context_text":null},{"citing_arxiv_id":"2606.04662","ref_index":56,"ref_count":1,"confidence":0.9,"is_internal_anchor":false,"paper_title":"Why Muon Outperforms Adam: A Curvature Perspective","primary_cat":"cs.LG","submitted_at":"2026-06-03T09:40:30+00:00","verdict":"CONDITIONAL","verdict_confidence":"LOW","novelty_score":7.0,"formal_verification":"none","one_line_summary":"Muon outperforms Adam by reducing curvature penalty via lower Normalized Directional Sharpness, as shown via Taylor approximation on LLM training and proven on stylized quadratic problems with heterogeneous curvature.","context_count":0,"top_context_role":null,"top_context_polarity":null,"context_text":null},{"citing_arxiv_id":"2606.04048","ref_index":15,"ref_count":1,"confidence":0.9,"is_internal_anchor":false,"paper_title":"Unlocking Feature Learning in Gated Delta Networks at Scale","primary_cat":"cs.LG","submitted_at":"2026-06-02T08:45:24+00:00","verdict":"UNVERDICTED","verdict_confidence":"LOW","novelty_score":7.0,"formal_verification":"none","one_line_summary":"Derives μP-style scaling rules for Gated Delta Networks and validates stable learning-rate transfer in language model pre-training experiments.","context_count":0,"top_context_role":null,"top_context_polarity":null,"context_text":null},{"citing_arxiv_id":"2605.26459","ref_index":5,"ref_count":1,"confidence":0.9,"is_internal_anchor":false,"paper_title":"MuCon: Clipped Muon Updates for LLM Training","primary_cat":"cs.LG","submitted_at":"2026-05-26T02:16:17+00:00","verdict":"UNVERDICTED","verdict_confidence":"LOW","novelty_score":5.0,"formal_verification":"none","one_line_summary":"MuCon defines a clipped-Muon update via singular-value clipping and derives two exact identities for approximating the clip without dense SVD, while noting numerical instability near the threshold.","context_count":0,"top_context_role":null,"top_context_polarity":null,"context_text":null},{"citing_arxiv_id":"2605.21803","ref_index":48,"ref_count":1,"confidence":0.9,"is_internal_anchor":false,"paper_title":"Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Laws","primary_cat":"cs.LG","submitted_at":"2026-05-20T23:00:34+00:00","verdict":"UNVERDICTED","verdict_confidence":"LOW","novelty_score":6.0,"formal_verification":"none","one_line_summary":"The same Transformer architecture follows different spectral scaling laws under different optimizers, with Muon achieving linear hard-rank scaling on tail representations while AdamW shows weak scaling, even when perplexity is matched.","context_count":0,"top_context_role":null,"top_context_polarity":null,"context_text":null},{"citing_arxiv_id":"2605.18528","ref_index":30,"ref_count":2,"confidence":0.9,"is_internal_anchor":false,"paper_title":"Scale-Invariant Neural Network Optimization: Norm Geometry and Heavy-Tailed Noise","primary_cat":"math.OC","submitted_at":"2026-05-18T15:13:18+00:00","verdict":"UNVERDICTED","verdict_confidence":"LOW","novelty_score":6.0,"formal_verification":"none","one_line_summary":"Establishes matching Ω and O(min{m,n} ε^-(3p-2)/(p-1)) bounds for scale-invariant spectral-norm methods under heavy-tailed noise, plus an improved O(min{m,n} ε^-(5p-3)/(2p-2)) rate via transported Scion under Hessian Lipschitz continuity.","context_count":1,"top_context_role":"background","top_context_polarity":"background","context_text":"Applying this inequality successively fork=t−1, t−2, . . . ,0 gives E   t−1X s=0 BX i=1 ap s∥ζ i s∥p ⋆ ! 1 p   ≤B 1 p σ0 \u0010 t−1X s=0 ap s \u0011 1 p +B 1 p σ1 t−1X s=0 as EΓs. Substituting the above into Eq. (B.29) yields the desired result.□ Lemma B.14Suppose that Assumptions 2.1 and 2.2 hold, and letτ ⋆ :=τ(∥ · ∥ ⋆, m, n, p). In Algo- rithm 1, ifβ t ≡β∈[0,1)andη t ≡η≤ 1 L1 , then for everyt= 0, . . . , T−1, we have E∥mt+1 − ∇F(X t)∥⋆ ≤ τ⋆ B p−1 p h βt σ0 +σ 1∥∇F(X 0)∥⋆ \u0001 + (1−β) p−1 p σ0 i + βηL0(1−β t) 1−β +ηL 1 t−1X s=0 β t−s E∥∇F(X s)∥⋆ + τ⋆σ1 B p−1 p tX s=1 (1−β)β t−s E∥∇F(X s)∥⋆. Proof.Letζ i s :=G(X s, ξi s)− ∇F(X s) and ¯ζs := ¯Gs − ∇F(X s). By Assumption 2.2 and the fresh i.i.d. mini-batches at each query pointX s, Lemma B.13 applies with Γs =∥∇F(X s)∥⋆. Also,m 1 −∇F(X 0) = ¯ζ0. Fort≥1, mt+1 − ∇F(X t) =β(m t − ∇F(X t−1)) +β ∇F(X t−1)− ∇F(X t) \u0001 + (1−β) ¯ζt. Iterating, we get mt+1 − ∇F(X t) =β t¯ζ0 + t−1X s=0 β t−s ∇F(X s)− ∇F(X s+1) \u0001 + tX s=1 (1−β)β t−s¯ζs, where the sums are empty whent= 0. Using Assumption 2.1, we have∥∇F(X s)− ∇F(X s+1)∥⋆ ≤ η(L0 +L 1∥∇F(X s)∥⋆). Therefore, E∥mt+1 − ∇F(X t)∥⋆ ≤β t E∥¯ζ0∥⋆ + t−1X s=0 β t−sη L0 +L 1E∥∇F(X s)∥⋆ \u0001 +E tX s=1 (1−β)β t−s¯ζs ⋆ .(B.30) It remains to control the batched noise terms. Takingt= 1 anda 0 = 1 in Lemma B.13 gives E∥¯ζ0∥⋆ ≤ τ⋆ B p−1 p σ0 +σ 1∥∇F(X 0)∥⋆ \u0001 . 37 Fort≥1, applying Lemma B.13 after shifting the index set to the batches 1, . . . , twith weights as−1 = (1−β)β t−s gives E tX s=1 (1−β)β t−s¯ζs ⋆ ≤ τ⋆ B p−1 p  σ0(1−β) tX s=1 βp(t−s) ! 1 p +σ 1 tX s=1 (1−β)β t−sE∥∇F(X s)∥⋆   ≤ τ⋆ B p−1 p \" σ0(1−β) p−1 p +σ 1 tX s=1 (1−β)β t−sE∥∇F(X s)∥⋆ # . Combining the above two inequalities with Eq. (B.30) yields the desired result.□ Proof of Theorem 3.2.LetA T :=E∥∇F( eXT )∥⋆,r= p−1 p ,S 0 :=σ 0 +σ 1∥∇F(X 0)∥⋆. Since ⟨mt+1,lmo(m t+1)⟩=−∥m t+1∥⋆ and∥lmo(m t+1)∥ ≤1, we have ⟨∇F(X t),lmo(m t+1)⟩ ≤ −∥∇F(X t)∥⋆ + 2∥mt+1 − ∇F(X t)∥⋆. Using Proposition B.12 andη≤1/L 1, we obtain F(X t+1)≤F(X t)−η∥∇F(X t)∥⋆"},{"citing_arxiv_id":"2605.15290","ref_index":20,"ref_count":1,"confidence":0.9,"is_internal_anchor":false,"paper_title":"GQA-{\\mu}P: The maximal parameterization update for grouped query attention","primary_cat":"cs.LG","submitted_at":"2026-05-14T18:03:16+00:00","verdict":"UNVERDICTED","verdict_confidence":"LOW","novelty_score":7.0,"formal_verification":"none","one_line_summary":"Derives μP scalings for GQA via promoted spectral-norm definition of feature learning and a modified norm preserving scaling laws for non-full-rank matrices, with experiments showing learning-rate transfer.","context_count":0,"top_context_role":null,"top_context_polarity":null,"context_text":null},{"citing_arxiv_id":"2605.12492","ref_index":86,"ref_count":1,"confidence":0.9,"is_internal_anchor":false,"paper_title":"Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation","primary_cat":"cs.LG","submitted_at":"2026-05-12T17:59:34+00:00","verdict":"UNVERDICTED","verdict_confidence":"LOW","novelty_score":5.0,"formal_verification":"none","one_line_summary":"Pion is an optimizer that preserves the singular values of weight matrices in LLM training by applying orthogonal equivalence transformations.","context_count":1,"top_context_role":"background","top_context_polarity":"background","context_text":"itive alternative to standard optimizers for both LLM pretraining and finetuning. 1 Introduction As large language models (LLMs) continue to scale, the difficulty of training them also increases significantly. One of the most critical challenges today is designing optimizers that are both efficient and stable. Training stability can be partially characterized by the Maximal Update Parameterization (µP) [86], where spectral norms of weights and updates are constrained such that width-invariant activations are of constant scale and hence prevent explosions. By performing the steepest descent under the spectral norm through update orthogonalization, Muon [36] has emerged as a competitive alternative to AdamW [37, 49]. Although Muon's orthogonalization ensures that each update is easily"},{"citing_arxiv_id":"2605.11872","ref_index":42,"ref_count":1,"confidence":0.9,"is_internal_anchor":false,"paper_title":"LOFT: Low-Rank Orthogonal Fine-Tuning via Task-Aware Support Selection","primary_cat":"cs.LG","submitted_at":"2026-05-12T09:51:55+00:00","verdict":"UNVERDICTED","verdict_confidence":"LOW","novelty_score":7.0,"formal_verification":"none","one_line_summary":"LOFT unifies orthogonal PEFT by treating adaptation as low-rank subspace rotation and adds task-aware support selection that improves efficiency under fixed budgets.","context_count":0,"top_context_role":null,"top_context_polarity":null,"context_text":null},{"citing_arxiv_id":"2605.10164","ref_index":4,"ref_count":1,"confidence":0.9,"is_internal_anchor":false,"paper_title":"Hyperparameter Transfer for Dense Associative Memories","primary_cat":"cs.LG","submitted_at":"2026-05-11T08:15:13+00:00","verdict":"UNVERDICTED","verdict_confidence":"LOW","novelty_score":7.0,"formal_verification":"none","one_line_summary":"Explicit scaling prescriptions for hyperparameters in DenseAMs are derived from model dynamics and shown to match empirical results across scales.","context_count":0,"top_context_role":null,"top_context_polarity":null,"context_text":null},{"citing_arxiv_id":"2605.09238","ref_index":68,"ref_count":1,"confidence":0.9,"is_internal_anchor":false,"paper_title":"Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifolds","primary_cat":"cs.LG","submitted_at":"2026-05-10T00:39:13+00:00","verdict":"UNVERDICTED","verdict_confidence":"LOW","novelty_score":7.0,"formal_verification":"none","one_line_summary":"Intrinsic Muon provides closed-form linear maximization oracles on multiple Riemannian matrix manifolds for unitarily invariant norms, with convergence rates depending only on manifold dimension or rank.","context_count":1,"top_context_role":"baseline","top_context_polarity":"baseline","context_text":"polar-factor solution and admits no closed form on most manifolds of interest. Existing literature addresses these issues only partially. Spectron [26] bounds the ambient update by an iterate-dependent factor rescaling computed dynamically. Riemannion [10] relies on an approximate tangent space orthogonalization. Manifold constrained steepest descent (MCSD) [68] drops the tangent constraint and projects back to the manifold instead. However, none of the existing approaches resolves (O1) and (O2) jointly. In this paper, we propose intrinsic Muon (iMuon), a principled and unified framework for spectral, and, more generally, unitarily invariant norm-constrained optimization on Riemannian matrix manifolds (whose points can be represented as matrices) that resolves both (O1) and (O2) simultaneously."},{"citing_arxiv_id":"2605.06654","ref_index":38,"ref_count":1,"confidence":0.9,"is_internal_anchor":false,"paper_title":"Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Less","primary_cat":"cs.LG","submitted_at":"2026-05-07T17:57:02+00:00","verdict":"UNVERDICTED","verdict_confidence":"LOW","novelty_score":6.0,"formal_verification":"none","one_line_summary":"Full finetuning with the pretraining optimizer reduces forgetting compared to other optimizers or LoRA while achieving comparable new-task performance.","context_count":0,"top_context_role":null,"top_context_polarity":null,"context_text":null},{"citing_arxiv_id":"2605.04418","ref_index":21,"ref_count":1,"confidence":0.9,"is_internal_anchor":false,"paper_title":"Demystifying Manifold Constraints in LLM Pre-training","primary_cat":"cs.LG","submitted_at":"2026-05-06T02:22:06+00:00","verdict":"UNVERDICTED","verdict_confidence":"LOW","novelty_score":6.0,"formal_verification":"none","one_line_summary":"Manifold constraints via the new MACRO optimizer independently bound activation scales and enforce rotational equilibrium in LLM pre-training, subsuming RMS normalization and decoupled weight decay while delivering competitive performance with convergence guarantees.","context_count":1,"top_context_role":"background","top_context_polarity":"background","context_text":"like model. In this subsection, we analyze and compare Frobenius and spectral spheres, and output/input Oblique manifolds. The major benefit of weight constraints is controlling the scale of activations. Consider a single linear layer Y=XW ⊤, where X∈ R T×D in ,Y∈ R T×D out and W∈ R Dout×Din for a sequence of length T . Following Yang et al. [20] and Su [21], our goal is for the forward pass to maintain a constant root-mean-square (RMS) norm: ∥vec(Y)∥ RMS = Θ(1) . We can express this sequence-level operation using the Kro- necker product: vec(Y) = (W⊗I T ) vec(X). Importantly, the spectral norm remains invariant under this expansion (∥W⊗I T ∥2 =∥W∥ 2), thus explicitly constraining ∥W∥2 directly bounds the activation amplification."},{"citing_arxiv_id":"2604.02505","ref_index":11,"ref_count":1,"confidence":0.9,"is_internal_anchor":false,"paper_title":"Optimal Projection-Free Adaptive SGD for Matrix Optimization","primary_cat":"math.OC","submitted_at":"2026-04-02T21:02:10+00:00","verdict":"UNVERDICTED","verdict_confidence":"LOW","novelty_score":6.0,"formal_verification":"none","one_line_summary":"Proving stability of Leon's preconditioner enables the first tuning-free Nesterov-accelerated projection-free adaptive SGD variant with improved non-smooth non-convex rates.","context_count":1,"top_context_role":"background","top_context_polarity":"background","context_text":"k(mk−1; mk), we are ready to establish the main upper bound on the regret-like quantity Reg+ K in the following Theorem 2. Theorem 2 (↓). Let η = R and δ > 0. Let Sk+1 ⪰ Sk for all k ∈ N0. Let g0, . . . , gK ∈ X and x0 = 0. Let x1, . . . , xK+1 be generated according to eqs. (2) and (3). Then, it holds that Reg+ K ≤ Ψ∗ 0(0) + 3 2 η∥ p SK∥tr − KX k=0 1 4η ∥xk+1 − xk∥2 S1/2 k . (11) In the following section, we show how to use the upper-bound in Theorem 2 for obtaining improved guarantees in online convex and non-smooth non-convex optimization. 3.3 Application to Online and Non-smooth Non-convex Optimization Online optimization. We start with the general improved result for Algorithm 1 for solving online convex optimization, as stated in the following Theorem 3."},{"citing_arxiv_id":"2603.00541","ref_index":46,"ref_count":1,"confidence":0.9,"is_internal_anchor":false,"paper_title":"Spectral Condition for $\\mu$P under Width-Depth Scaling","primary_cat":"cs.LG","submitted_at":"2026-02-28T08:38:50+00:00","verdict":"UNVERDICTED","verdict_confidence":"LOW","novelty_score":6.0,"formal_verification":"none","one_line_summary":"A unified spectral condition for μP under width-depth scaling reveals a transition at k=1 vs k≥2 transformations per residual block and enables stable feature learning for practical architectures like Transformers.","context_count":0,"top_context_role":null,"top_context_polarity":null,"context_text":null},{"citing_arxiv_id":"2510.04212","ref_index":33,"ref_count":1,"confidence":0.9,"is_internal_anchor":false,"paper_title":"Why Low-Precision Transformer Training Fails: An Analysis on Flash Attention","primary_cat":"cs.LG","submitted_at":"2025-10-05T14:01:24+00:00","verdict":"UNVERDICTED","verdict_confidence":"LOW","novelty_score":7.0,"formal_verification":"none","one_line_summary":"Low-precision Flash Attention fails due to similar low-rank attention representations combined with biased rounding errors that accumulate and corrupt weight updates; a minimal fix to reduce rounding bias stabilizes training.","context_count":0,"top_context_role":null,"top_context_polarity":null,"context_text":null},{"citing_arxiv_id":"2502.07529","ref_index":214,"ref_count":1,"confidence":0.9,"is_internal_anchor":false,"paper_title":"Training Deep Learning Models with Norm-Constrained LMOs","primary_cat":"cs.LG","submitted_at":"2025-02-11T13:10:34+00:00","verdict":"UNVERDICTED","verdict_confidence":"LOW","novelty_score":7.0,"formal_verification":"none","one_line_summary":"Scion is a new stochastic LMO-based optimizer family that unifies existing methods, supports unconstrained problems, and delivers hyperparameter transferability plus speedups on nanoGPT training.","context_count":0,"top_context_role":null,"top_context_polarity":null,"context_text":null},{"citing_arxiv_id":"2409.20325","ref_index":37,"ref_count":1,"confidence":0.9,"is_internal_anchor":false,"paper_title":"Old Optimizer, New Norm: An Anthology","primary_cat":"cs.LG","submitted_at":"2024-09-30T14:26:12+00:00","verdict":"UNVERDICTED","verdict_confidence":"LOW","novelty_score":7.0,"formal_verification":"none","one_line_summary":"Optimizers like Adam reduce to steepest descent under particular norms, opening a design space of norm assignments tailored to layer roles.","context_count":0,"top_context_role":null,"top_context_polarity":null,"context_text":null},{"citing_arxiv_id":"2403.03507","ref_index":50,"ref_count":1,"confidence":0.9,"is_internal_anchor":false,"paper_title":"GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection","primary_cat":"cs.LG","submitted_at":"2024-03-06T07:29:57+00:00","verdict":"CONDITIONAL","verdict_confidence":"LOW","novelty_score":7.0,"formal_verification":"none","one_line_summary":"GaLore performs full-parameter LLM training with up to 65.5% less optimizer memory by projecting gradients onto a low-rank subspace at each step, matching full-rank performance on LLaMA pre-training and RoBERTa fine-tuning.","context_count":0,"top_context_role":null,"top_context_polarity":null,"context_text":null}],"limit":50,"offset":0}