The reviewed record of science sign in
Pith

arxiv: 2606.25975 · v1 · pith:XVW26VDW · submitted 2026-06-24 · cs.LG · cs.CV· cs.NA· math.NA

Tensorion: A Tensor-Aware Generalization of the Muon Optimizer

Vladimir Bogachev , Vladimir Aletov , Alexander Molozhavenko , Sergei Kudriashov , Maxim Rakhuba This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-25 19:59 UTCgrok-4.3pith:XVW26VDWrecord.jsonopen to challenge →

classification cs.LG cs.CVcs.NAmath.NA
keywords TensorionMuon optimizertensor-aware optimizationlinear minimization oraclespectral normunfolding matricesmachine learning optimizerstensor parameters
0
0 comments X

The pith

Tensorion generalizes Muon from matrices to higher-order tensors via a linear minimization oracle over a tensor norm ball.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces Tensorion to handle the multilinear structure of tensor parameters in machine learning models instead of treating them as flat vectors. It extends Muon's spectral-norm steepest descent to tensors by defining a norm whose linear minimization oracle reduces to operations on selected unfolding matrices. The design keeps the oracle tractable while aiming to bound the tensor spectral norm. A reader would care because many models use tensor weights, and respecting that structure could change how gradients are updated during training. When limited to matrices the method matches Muon exactly.

Core claim

Tensorion performs constrained optimization over a tensor norm ball through its linear minimization oracle. The norm is selected so the oracle remains computable by reducing to operations on adaptively chosen unfolding matrices while still bounding the tensor spectral norm. The method recovers Muon exactly on order-2 tensors. On tensor-based computer vision tasks it produces more stable gradient updates and improved convergence relative to Adam and existing tensor-aware methods.

What carries the argument

The linear minimization oracle over a tensor norm ball chosen to bound the spectral norm while reducing to unfolding-matrix operations.

If this is right

  • Tensorion recovers Muon exactly when restricted to matrices.
  • It applies to higher-order tensor weights in computer vision models.
  • It yields more stable gradient updates than Adam-based methods in the tested settings.
  • The oracle stays tractable for tensors beyond order 2 through unfolding reductions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same unfolding reduction might extend to other structured parameter spaces such as those appearing in physics or recommendation models.
  • Different norm choices could trade bound tightness against computation speed in future variants.
  • Testing the method on non-vision tensor problems would show whether the gains depend on the specific task structure.

Load-bearing premise

A tensor norm exists that bounds the spectral norm tightly enough for the linear minimization oracle to stay tractable through operations on unfolding matrices.

What would settle it

An experiment in which Tensorion updates on tensor vision tasks show no gain in convergence speed or stability over Adam baselines, or in which the oracle steps violate the intended spectral norm bound.

Figures

Figures reproduced from arXiv: 2606.25975 by Alexander Molozhavenko, Maxim Rakhuba, Sergei Kudriashov, Vladimir Aletov, Vladimir Bogachev.

Figure 1
Figure 1. Figure 1: Ablation on the unfolding set τ for ResNet-18 on CIFAR￾10 (25 epochs; constant LR, no scheduler) using exact SVD for orthogonalization. Curves labeled “τ = {·}” use a fixed unfolding index set τ ⊂ {1, 2, 3, 4}.“τ opt” selects τ once per layer via the offline heuristic in Eq. (24). “τ per iteration” performs online se￾lection by explicitly maximizing ∥X(τ)∥∗ over τ at each iteration as in Eq. (21). SGD-M: S… view at source ↗
Figure 2
Figure 2. Figure 2: Tiny ImageNet results for ResNet-50 trained for 200 epochs. We report mean ± std over multiple seeds for test accuracy and test loss Tensorion is used only for tensor-valued blocks with order d ≥ 3 (e.g., convolutional kernels), while all remaining parameters (d ≤ 2) are optimized with a first-order method (SGD) [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: CIFAR-10 results for ResNet-18 trained for 200 epochs under the standard CIFAR pipeline of (Contributors, 2023). We report mean ± std over multiple seeds for test accuracy (Figure 4a), test loss (Figure 4b), and train loss (Figure 4c). Tensorion is used only for tensor-valued blocks with order d ≥ 3 (e.g., convolutional kernels), while all remaining parameters (d ≤ 2) are optimized with a first-order metho… view at source ↗
Figure 5
Figure 5. Figure 5: CIFAR-100 results for ResNet-34 trained for 250 epochs under the standard CIFAR pipeline of (Contributors, 2023). We report mean ± std over multiple seeds for test accuracy (Figure 5a), test loss (Figure 5b), and train loss (Figure 5c). Tensorion is used only for tensor-valued blocks with order d ≥ 3 (e.g., convolutional kernels), while all remaining parameters (d ≤ 2) are optimized with a first-order meth… view at source ↗
Figure 6
Figure 6. Figure 6: Test accuracy and training loss across all Swin models. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Test accuracy and training loss across ViT models. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Minimization of the Rayleigh quotient (96) for the smallest eigenpair of A = −∆d on a d-dimensional tensor-product grid, with the eigenvector x parameterized in TT format with bounded TT-ranks. Left: final functional-residual as a function of the learning rate. Right: functional-residual versus optimization step at the best-tuned learning rate for each method. 0 20 40 60 80 100 Step 0.989 0.990 0.991 0.992… view at source ↗
Figure 9
Figure 9. Figure 9: Actual tensor spectral norm of the LMO solutions with relaxed tensor spectral norm on optimization trajectories of ResNet-18, CIFAR-10 E. Experimental Details E.1. Tau ablation We enumerate all proper non-empty subsets τ ⊊ {1, 2, 3, 4}. Since unfolding with τ and with its complement τ c produces matrices that are transposes of each other, and the nuclear norm is invariant to this operation, ∥X(τ)∥∗ = ∥X(τ … view at source ↗
read the original abstract

Common first-order optimizers, such as Adam, implicitly treat each parameter block as an unstructured vector, which disregards the multilinear weight structure present in many modern machine learning models. Recent work has shown that exploiting matrix structure can improve optimization dynamics. A notable example is Muon, which performs steepest descent under the spectral norm constraint. We take the next step and introduce Tensorion, a tensor-aware optimizer that extends Muon's constrained optimization perspective from matrices to higher-order tensors. Tensorion is built around a linear minimization oracle (LMO) over a tensor norm ball. The norm is carefully chosen to balance two objectives: tightly bounding the tensor spectral norm, while still keeping the LMO tractable. This LMO becomes computable because it reduces to operations on adaptively selected unfolding matrices. Notably, when restricted to order-2 tensors (i.e., matrices), Tensorion recovers Muon exactly. Experiments on tensor-based computer vision problems suggest that Tensorion can offer improved convergence behavior and more stable gradient updates compared with Adam-based and existing tensor-aware baselines in the evaluated settings.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 0 minor

Summary. The manuscript introduces Tensorion as a generalization of the Muon optimizer from matrices to higher-order tensors. It constructs a tensor norm ball whose linear minimization oracle (LMO) is claimed to be tractable by reduction to operations on adaptively selected unfolding matrices while still tightly bounding the tensor spectral norm; the matrix case recovers Muon exactly, and experiments on tensor-based computer vision tasks are said to show improved convergence and more stable updates versus Adam and existing tensor-aware baselines.

Significance. If the norm construction, LMO reduction, and empirical gains hold, the work would extend principled spectral-norm constrained optimization to the multilinear setting common in modern models, addressing a clear limitation of vector-based methods like Adam.

major comments (2)
  1. [Abstract] Abstract (and throughout): no derivation or explicit definition of the tensor norm is supplied, nor is there a proof that the chosen norm both bounds the multilinear spectral norm and yields a tractable LMO via adaptive unfoldings; without these steps the central technical claim cannot be verified.
  2. [Abstract] Abstract: the experimental claim of improved convergence and stability is stated without any quantitative results, protocol details, dataset descriptions, or baseline implementations, rendering the performance assertion impossible to assess.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their comments on our manuscript. We address each major comment point by point below.

read point-by-point responses

  1. Referee: [Abstract] Abstract (and throughout): no derivation or explicit definition of the tensor norm is supplied, nor is there a proof that the chosen norm both bounds the multilinear spectral norm and yields a tractable LMO via adaptive unfoldings; without these steps the central technical claim cannot be verified.

    Authors: The abstract serves as a high-level summary and therefore omits the full technical details. The manuscript body supplies the explicit definition of the tensor norm, the derivation showing that it tightly bounds the multilinear spectral norm, and the proof of the tractable LMO reduction to adaptive unfolding matrices. We will revise the abstract to include a concise reference to these elements and their guarantees. revision: yes

  2. Referee: [Abstract] Abstract: the experimental claim of improved convergence and stability is stated without any quantitative results, protocol details, dataset descriptions, or baseline implementations, rendering the performance assertion impossible to assess.

    Authors: Space limitations in the abstract preclude inclusion of quantitative results or full protocol details. The manuscript's Experiments section provides the complete experimental protocol, dataset descriptions for the evaluated computer vision tasks, baseline implementations, and quantitative results showing the reported improvements. We will revise the abstract to add a brief statement of key quantitative findings. revision: yes

Circularity Check

0 steps flagged

No significant circularity; derivation is self-contained

full rationale

The central construction defines a tensor norm whose ball yields a tractable LMO via adaptive unfoldings while bounding the multilinear spectral norm, with explicit verification that the order-2 restriction recovers Muon exactly. This is a consistency property of the generalization rather than a definitional reduction. No equations reduce a claimed prediction to a fitted input by construction, no load-bearing self-citations appear, and the experimental claims rest on external benchmarks rather than internal renaming or ansatz smuggling. The provided material shows an independent design choice whose tractability and bounding properties are asserted as engineering outcomes, not tautologies.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract supplies no explicit free parameters, axioms, or invented entities; the norm selection is described only qualitatively as balancing two objectives.

pith-pipeline@v0.9.1-grok · 5743 in / 999 out tokens · 19530 ms · 2026-06-25T19:59:52.535953+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

57 extracted references · 8 canonical work pages · 1 internal anchor

  1. [1]

    An image is worth 16x16 words: Transformers for image recognition at scale , author =

  2. [2]

    2509.03439 , archiveprefix =

    Concentration Inequalities for Sub-Weibull Random Tensors , author =. 2509.03439 , archiveprefix =

    work page arXiv

  3. [3]

    Foundations and Trends in Machine Learning , volume = 4, number = 1, pages =

    Optimization with Sparsity-Inducing Penalties , author =. Foundations and Trends in Machine Learning , volume = 4, number = 1, pages =

  4. [4]

    Lectures on mathematical theory of extremum problems , author =

  5. [5]

    IEEE Transactions on Information Theory , volume = 60, number = 2, pages =

    Blind Multilinear Identification , author =. IEEE Transactions on Information Theory , volume = 60, number = 2, pages =. doi:10.1109/TIT.2013.2291876 , keywords =

    work page doi:10.1109/tit.2013.2291876 2013

  6. [6]

    Linear and Multilinear Algebra , publisher =

    Variation of tensor powers and spectrat , author =. Linear and Multilinear Algebra , publisher =. doi:10.1080/03081088208817475 , url =

    work page doi:10.1080/03081088208817475

  7. [7]

    Gluon: Making muon & scion great again!(bridging theory and practice of lmo-based optimizers for llms) , author =

  8. [8]

    Proceedings of the IEEE conference on computer vision and pattern recognition , pages =

    Learning separable filters , author =. Proceedings of the IEEE conference on computer vision and pattern recognition , pages =

  9. [9]

    Dion: Distributed orthonormalized updates , author =

  10. [10]

    Muon optimizes under spectral norm constraints , author =

  11. [11]

    European Conference on Computer Vision , pages =

    Stable low-rank tensor decomposition for compression of convolutional neural network , author =. European Conference on Computer Vision , pages =

  12. [12]

    Thinking Machines Lab: Connectionism , doi =

    Modular Manifolds , author =. Thinking Machines Lab: Connectionism , doi =

  13. [13]

    International conference on machine learning , pages =

    signSGD: Compressed optimisation for non-convex problems , author =. International conference on machine learning , pages =

  14. [14]

    Advances in neural information processing systems , volume = 28, doi =

    Tensorizing neural networks , author =. Advances in neural information processing systems , volume = 28, doi =

  15. [15]

    Linear algebra and its applications , publisher =

    Operator norm inequalities between tensor unfoldings on the partition lattice , author =. Linear algebra and its applications , publisher =

  16. [16]

    Fantastic four: Differentiable bounds on singular values of convolution layers , author =

  17. [17]

    2019 IEEE international conference on image processing (ICIP) , pages =

    Improving robustness to adversarial examples by encouraging discriminative features , author =. 2019 IEEE international conference on image processing (ICIP) , pages =. doi:10.1109/ICIP.2019.8803601 , organization =

    work page doi:10.1109/icip.2019.8803601 2019

  18. [18]

    Computer Vision -- ECCV 2024 , publisher =

    Tight and Efficient Upper Bound on Spectral Norm of Convolutional Layers , author =. Computer Vision -- ECCV 2024 , publisher =. doi:10.1007/978-3-031-73024-5_2 , isbn =

    work page doi:10.1007/978-3-031-73024-5_2 2024

  19. [19]

    Muon is scalable for llm training , author =

  20. [20]

    Mathematics of Computation , volume = 87, number = 311, pages =

    Nuclear norm of higher-order tensors , author =. Mathematics of Computation , volume = 87, number = 311, pages =

  21. [21]

    3rd International Conference on Learning Representations,

    Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition , author =. 3rd International Conference on Learning Representations,

  22. [22]

    Convex analysis , author =

  23. [23]

    SIAM Journal on Numerical Analysis , volume = 8, number = 2, pages =

    An Iterative Algorithm for Computing the Best Estimate of an Orthogonal Matrix , author =. SIAM Journal on Numerical Analysis , volume = 8, number = 2, pages =. doi:10.1137/0708036 , url =

    work page doi:10.1137/0708036

  24. [24]

    Kimi K2: Open Agentic Intelligence

    Kimi K2: Open Agentic Intelligence , author =. 2507.20534 , archiveprefix =

    work page internal anchor Pith review Pith/arXiv arXiv

  25. [25]

    2601.23261 , archiveprefix =

    TEON: Tensorized Orthonormalization Beyond Layer-Wise Muon for Large Language Model Pre-Training , author =. 2601.23261 , archiveprefix =

    work page arXiv

  26. [26]

    CoRR , volume =

    Adam: A Method for Stochastic Optimization , author =. CoRR , volume =

  27. [27]

    ArXiv , volume =

    Most Tensor Problems Are NP-Hard , author =. ArXiv , volume =

  28. [28]

    Muon: An optimizer for hidden layers in neural networks , author =

  29. [29]

    Deriving Muon , author =

  30. [30]

    OpenMMLab's Pre-training Toolbox and Benchmark , author =

  31. [31]

    Proceedings of the 38th International Conference on Neural Information Processing Systems , location =

    Multilinear mixture of experts: scalable expert specialization through factorization , author =. Proceedings of the 38th International Conference on Neural Information Processing Systems , location =

  32. [32]

    Learning multiple layers of features from tiny images , author =

  33. [33]

    Decoupled weight decay regularization , author =

  34. [34]

    Proceedings of the IEEE conference on computer vision and pattern recognition , pages =

    Deep residual learning for image recognition , author =. Proceedings of the IEEE conference on computer vision and pattern recognition , pages =

  35. [35]

    Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) , year=

    Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective , author=. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) , year=

  36. [36]

    2019 , month=

    Imagenette: A smaller subset of 10 easily classified classes from Imagenet , author=. 2019 , month=

    2019

  37. [37]

    Proceedings of the IEEE/CVF international conference on computer vision , pages =

    Swin transformer: Hierarchical vision transformer using shifted windows , author =. Proceedings of the IEEE/CVF international conference on computer vision , pages =

  38. [38]

    Bert: Pre-training of deep bidirectional transformers for language understanding , author =. Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) , pages =

    2019

  39. [39]

    International conference on machine learning , pages =

    Optimizing neural networks with kronecker-factored approximate curvature , author =. International conference on machine learning , pages =

  40. [40]

    International Conference on Machine Learning , pages =

    A kronecker-factored approximate fisher matrix for convolution layers , author =. International Conference on Machine Learning , pages =

  41. [41]

    Advances in Neural Information Processing Systems , volume = 36, pages =

    Kronecker-factored approximate curvature for modern neural network architectures , author =. Advances in Neural Information Processing Systems , volume = 36, pages =

  42. [42]

    International Conference on Machine Learning , pages =

    Shampoo: Preconditioned stochastic tensor optimization , author =. International Conference on Machine Learning , pages =

  43. [43]

    Soap: Improving and stabilizing shampoo using adam , author =

  44. [44]

    A New Perspective on Shampoo's Preconditioner , author =

  45. [45]

    Old optimizer, new norm: An anthology , author =

  46. [46]

    International Conference on Machine Learning , pages =

    Monarch: Expressive structured matrices for efficient and accurate training , author =. International Conference on Machine Learning , pages =

  47. [47]

    LeSTD: LLM Compression via Learning-based Sparse Tensor Decomposition , author =

  48. [48]

    2025 International Joint Conference on Neural Networks (IJCNN) , pages =

    TensorLLM: Tensorising Multi-Head Attention for Enhanced Reasoning and Compression in LLMs , author =. 2025 International Joint Conference on Neural Networks (IJCNN) , pages =

    2025

  49. [49]

    Yuebin XU and YANHONG WANG and Xuemei Peng and Hui Zang and Chen Minghao and Pengfei Xia and Zeyi Wen , year = 2026, booktitle =

    2026

  50. [50]

    International Conference on Artificial Intelligence and Statistics , pages =

    Training a Tucker model with shared factors: a Riemannian optimization approach , author =. International Conference on Artificial Intelligence and Statistics , pages =

  51. [51]

    Vladimir Bogachev and Vladimir Aletov and Alexander Molozhavenko and Denis Bobkov and Vera Soboleva and Aibek Alanov and Maxim Rakhuba , year = 2026, booktitle =. Lo

    2026

  52. [52]

    Parameter and memory efficient pretraining via low-rank riemannian optimization , author =

  53. [53]

    International Conference on Machine Learning , doi =

    Training Deep Learning Models with Norm-Constrained LMOs , author =. International Conference on Machine Learning , doi =

  54. [54]

    Thirty-seventh Conference on Neural Information Processing Systems , year =

    Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and width , author =. Thirty-seventh Conference on Neural Information Processing Systems , year =

  55. [55]

    Early convolutions help transformers see better , year =

    Xiao, Tete and Singh, Mannat and Mintun, Eric and Darrell, Trevor and Doll\'. Early convolutions help transformers see better , year =. Proceedings of the 35th International Conference on Neural Information Processing Systems , articleno =

  56. [56]

    Journal of Physics: Conference Series , volume=

    HPC resources of the higher school of economics , author=. Journal of Physics: Conference Series , volume=. 2021 , organization=

    2021

  57. [57]

    Computational and Applied Mathematics , volume = 45, url =

    Optimization on the extended tensor-train manifold with shared factors , author =. Computational and Applied Mathematics , volume = 45, url =