Pith. sign in

REVIEW 3 major objections 5 minor 56 references

ManifoldFlow: SPD-Relaxed Stiefel Layers with Learnable Singular Spectrum

T0 review · 3 major / 5 minor · reviewed 2026-07-11 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read When a neural layer benefits from an orthonormal basis, its singular values need not stay frozen at one.

desk verdict Clean, scoped Stiefel relaxation with solid factorization math and honest boundaries; headline gains are real in direction but under-quantified for seed variance. read the letter →

arxiv 2607.04535 v1 pith:PAB5S5EL submitted 2026-07-05 cs.LG stat.ML

Haiwen Yi , Xinyuan Song This is my paper
classification cs.LGstat.ML
keywords StiefelmanifoldSPDspectrumorthogonalneurallayerssingularvaluecontrolFlowrecurrentlanguagemodelsRiemannianoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Fixed Stiefel and orthogonal layers give neural weights exact spectral control, but they force every represented singular value to equal one. ManifoldFlow is a minimal relaxation that keeps the basis on the Stiefel manifold while learning a bounded positive spectrum through the factorization W = Q S^{1/2}, where S is symmetric positive definite. Because W transpose W equals S, the eigenvalues of S are exactly the squared singular values of the realized weight, so clipping those eigenvalues is direct singular-value control. In paired experiments that match architecture, optimizer family, data split, and Stiefel update, enabling the learnable spectrum improves the fixed-spectrum Stiefel counterpart in settings where the Stiefel prior is useful, with the largest gains in recurrent language-model projections. Boundary cases such as convolutional classifier heads and stronger dense baselines mark the intended scope: the method is a spectrum-learnable Stiefel relaxation, not a universal dense-layer replacement.

What carries the argument

The parametrization W = Q S^{1/2} with Q on the Stiefel manifold and S symmetric positive definite. The exact identity W transpose W = S turns eigenvalue clipping of S into direct singular-value control on the realized weight, recovering fixed Stiefel when S is held at the identity.

What would settle it

On a recurrent language-model projection task where the Stiefel prior is known to help, if enabling the bounded SPD spectrum under the paper’s matched 1000-epoch protocol fails to reduce validation perplexity relative to fixed Stiefel, the main empirical claim is false.

Watch

Extended reading notes

Core claim

When the architecture, optimizer, data, and Stiefel update are held fixed, replacing a unit-spectrum Stiefel layer by ManifoldFlow’s factorization W = Q S^{1/2} improves the paired layer in the reported sequence, tabular, and image settings where an orthonormal basis is a useful prior. The identity W transpose W = S makes the eigenvalues of the SPD factor exactly the squared singular values of the forward weight, so a bounded spectrum can be learned without abandoning orthonormal directions.

Load-bearing premise

The claim only holds when an orthonormal basis is already a useful prior for the layer; if that prior is wrong, freeing its spectrum does not fix the mismatch.

Editorial extensions

If this is right

  • Recurrent language-model projections that currently pin singular values at one can free a bounded spectrum without abandoning orthonormality and expect lower perplexity, especially under SGD.
  • Layer-internal spectrum learning can supply structured preconditioning that external adaptive optimizers only partially replace.
  • Network spectral-complexity and Jacobian bounds can be written directly in terms of the learned eigenvalues of S rather than external spectral normalization.
  • Fixed-Stiefel training is recovered exactly by zeroing the geometry step, so the method is a controlled relaxation of an existing layer family rather than a separate architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • On tasks where dense or diagonal-spectrum layers already dominate the Stiefel family, first imposing orthonormality and then relaxing its spectrum is likely the wrong design path; spectrum learning without the Stiefel constraint may be preferable.
  • Very large wrapped dimensions will need structured, diagonal-plus-low-rank, or blockwise SPD updates before the method is practical inside large Transformer feed-forward blocks.
  • Because equal-norm symmetric directions induce equal local SPD motion near isotropic S, many geometry directions beyond rejected-gradient pressure could realize the same spectrum-relaxation benefit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ManifoldFlow, a product-manifold relaxation of fixed-spectrum Stiefel layers that keeps an orthonormal factor Q on St(p,r) while learning a clipped SPD spectrum S through W=Q S^{1/2}. The central identity W^T W=S makes eigenvalue clipping direct singular-value control; Fixed-Stiefel is recovered by freezing S=I. Theory covers rejected-gradient pressure as a Gram signal, feasible affine-invariant SPD updates, product-manifold stationarity (including biased geometry estimators and bounded preconditioners), spectral complexity, and local preconditioning interpretations. Empirically, the paper runs matched FS/MF pairs across recurrent language-model projections, tabular and image MLPs, a Mini-Transformer FFN, and convolutional classifier heads, reporting the largest gains on recurrent projections (especially under SGD) and treating CNN heads and dense/diagonal alternatives as scope boundaries rather than universal wins.

Significance. If the paired gains hold under proper multi-seed quantification, the contribution is a clean, scoped modeling tool rather than a broad architecture replacement: when an orthonormal basis is already a useful prior, freezing its spectrum at one is an unnecessary restriction. Strengths include the exact Gram/singular-value identity, nested FS baseline (geometry LR set to zero), explicit clipping and product-manifold analysis, pressure ablations that correctly de-emphasize the heuristic, and released code. The work sits usefully between pure Stiefel layers and unconstrained dense maps, and the boundary audit (dense/Qdiag/Intrinsic Muon) is a responsible scope statement rather than overclaim.

major comments (3)
  1. Tables 1 and 5 and the summary heatmap (Fig. 11) report the load-bearing 1000-epoch FS/MF deltas as single-run endpoints under plateau early-stopping, with multi-seed means/± only in the short-budget exploratory cells (Table 3; Appendix Table 9). The largest claimed effects (e.g., LSTM/WikiText-2 SGD ΔPPL=+184.51; GRU/WikiText-2 SGD +218.61) are therefore unquantified for seed variance. For a methods claim whose empirical support is the paired improvement under matched architecture/optimizer/Stiefel update, the main convergence batch needs multi-seed means and standard errors (or at least 3–5 seeds on the headline sequence and Adult/Covertype cells) before the signed gains can be treated as stable.
  2. Section 6.6 and Appendix Table 10 use archived short-protocol dense, Qdiag(s), and Intrinsic Muon baselines to delimit scope, while the converged Covertype FS/MF gains in Table 6 are much larger than the earlier exploratory numbers. The paper correctly refuses to claim dominance over dense layers, but the protocol split makes the “Stiefel prior is useful here” premise hard to audit on the same footing as the main FS/MF cells. Either rerun the key boundary baselines under the 1000-epoch protocol for Covertype and the LSTM projection, or move those archived numbers fully out of the main interpretive narrative and state that scope is conditioned only on FS/MF pairs.
  3. Free parameters controlling the SPD path (clip bounds α,β; ρ_geo / γ_t; pressure EMA β_P; gate thresholds in Eq. (15); λ_S in Eq. (16)) are not systematically reported for the main cells. Table 4 shows that EMA/gate/random-pressure ablations change Adult/Covertype accuracy by ≤0.1%, which supports spectrum relaxation over pressure uniqueness, but does not show robustness of the large sequence Δs to α,β and ρ_geo. A short sensitivity table on LSTM/WikiText-2 (or Adult) for clip bounds and geometry step size would make the empirical claim less dependent on unreported defaults.
minor comments (5)
  1. Figure 2 and Figure 6 are informative, but axis labels and panel captions sometimes mix “spectral signal,” eigenvalue interval, and κ(S) without restating the clip box S_{α,β}; a one-line reminder of (α,β) in each spectral figure would help.
  2. Mini-Transformer results (Table 3, Fig. 5) are correctly labeled exploratory/short-budget, yet they still appear in the main experimental narrative; a clearer visual separation (e.g., appendix-only or a dedicated “exploratory” subsection header) would reduce protocol confusion.
  3. Proposition 8’s local inverse-Hessian alignment condition is a useful interpretation of the SGD amplification, but the manuscript could note more explicitly that it is a sufficient local model rather than a verified property of the trained recurrent projections.
  4. Code link is given; adding the exact clip bounds, ρ_geo, and gate hyperparameters used for each main table in Appendix B would improve reproducibility without lengthening the main text.
  5. Minor notation consistency: both S≻0 and S∈SPD(r), and both St(p,r) and Stiefel, appear; pick one primary convention in the preliminaries and stick to it.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: parametrization identities, manifold updates, and paired FS/MF experiments are self-contained and not forced by construction or self-citation.

full rationale

ManifoldFlow is an empirical methods paper whose central claim is a nested comparison (enable vs. freeze the SPD factor under matched architecture/optimizer/Stiefel update). Proposition 2’s identity WᵀW = S and σᵢ²(W) = λᵢ(S) follows by direct algebra from the definition W = QS^{1/2} with Q on Stiefel; it is the intended spectral meaning of the parametrization, not a prediction derived from data or a prior uniqueness theorem. Stationarity (Theorems 1–2), feasible SPD retraction (Prop. 3), and complexity bounds (Thm. 3) are standard product-manifold arguments under stated smoothness/clipping assumptions; they do not import load-bearing uniqueness results from the authors’ prior work. Empirical gains are measured against the exact nested baseline obtained by setting ρ_geo = 0 (recovering Fixed-Stiefel), not against quantities fitted to the same metric being reported. Boundary audits and ablations further delimit scope rather than close a definitional loop. No fitted-input-as-prediction, self-definitional, or ansatz-via-self-citation steps appear.

Assumptions & free parameters 4 free parameters · 4 assumptions · 2 invented entities

The method rests on standard Stiefel/SPD Riemannian calculus plus several design knobs (clip bounds, geometry rate, pressure EMA/gate) and the modeling premise that an orthonormal basis is worth keeping. No new physical entities; the invented objects are algorithmic (pressure state, geometry gate, ManifoldFlow factorization).

free parameters (4)
  • SPD eigenvalue clip bounds (α, β)
    Hand-chosen spectral box that enforces S∈S_{α,β}; β=4 appears in LSTM spectral plots. Directly controls allowed singular values and thus capacity.
  • geometry learning rate ρ_geo and schedule γ_t
    Scales SPD steps; ρ_geo=0 recovers Fixed-Stiefel. Chosen by authors and central to whether spectrum learning is on.
  • pressure EMA β_P and geometry gate thresholds (τ_c, τ_r, α_c, α_r)
    Control which rejected-gradient signal updates S. Ablations show small effect, but they are free design parameters of the default method.
  • log-barrier coefficient λ_S
    Regularizes S toward I in the theoretical objective J; strength is a free hyperparameter of the geometry objective.
assumptions (4)
  • standard math Stiefel tangent projection, retractions, and standard Riemannian SGD stationarity on compact product manifolds
    Used throughout §3–5 and Appendix A; taken from Absil/Boumal-style manifold optimization.
  • standard math Affine-invariant SPD metric, exponential map, and log-Euclidean barrier geometry
    Defines Exp_S and d_AI for the S update (Eqs. 5, 18); standard SPD geometry citations.
  • domain assumption An orthonormal basis is a useful inductive prior in the positive experimental cells
    Explicit scope condition in abstract/§6.6/§7; without it the paired FS/MF comparison is the wrong contest (dense can dominate).
  • ad hoc to paper Rejected normal component (pressure) is a practical default SPD direction near isotropic S
    Prop. 1 motivates it at S=I; Prop. 5 and random-pressure ablations admit non-uniqueness. Core claim is spectrum relaxation, not uniqueness of pressure.
invented entities (2)
  • ManifoldFlow layer W=Q S^{1/2}
    purpose: Decouple orthonormal basis from learnable bounded singular spectrum with exact Gram identity W^T W=S.
    Primary proposed object; independent evidence is empirical FS/MF gains and spectral traces, not external measurement.
  • Rejected-gradient pressure P_t and geometry gate a_t
    purpose: Turn Stiefel-rejected normal component into a gated SPD update direction.
    Algorithmic construction in §4; ablations show it is replaceable by random symmetric directions within ~0.1%.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ManifoldFlow: SPD-Relaxed Stiefel Layers with Learnable Singular Spectrum." pith.science (2026). https://pith.science/paper/PAB5S5EL

@misc{pith2026260704535,
  author       = {Pith},
  title        = {Pith review of: ManifoldFlow: SPD-Relaxed Stiefel Layers with Learnable Singular Spectrum},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PAB5S5EL}},
  note         = {Machine review of arXiv:2607.04535}
}
read the original abstract

Orthogonal and Stiefel layers give neural weights exact spectral control, but they also impose a strong modeling constraint: all represented singular values are fixed at one. Many settings that benefit from an orthonormal basis still need direction-dependent attenuation or amplification. We introduce ManifoldFlow, a minimal relaxation of a fixed-spectrum Stiefel layer that keeps the basis on the Stiefel manifold while learning a bounded positive spectrum through W = Q S^{1/2}, with Q^T Q = I and S positive definite. Since W^T W = S, the eigenvalues of S are exactly the squared singular values of the realized weight, making eigenvalue clipping a direct singular-value control mechanism. Across paired sequence, tabular, and image experiments, the learnable SPD spectrum improves the fixed-spectrum Stiefel counterpart in the reported settings where the Stiefel prior is useful, with the largest gains in recurrent language-model projections. Boundary cases in convolutional classifier heads clarify the intended scope: ManifoldFlow is not a universal dense-layer replacement, but a spectrum-learnable Stiefel relaxation for settings where an orthonormal basis is a useful prior. When the basis should be orthonormal, its spectrum need not be frozen. Code available at https://github.com/Hik289/manifold_flow

Figures

Figures reproduced from arXiv: 2607.04535 by the authors.

Figure 1
Figure 1. ManifoldFlow overview. Fixed Stiefel fixes the represented spectrum at one; ManifoldFlow keeps the basis and learns a bounded SPD spectrum. question in this paper is therefore deliberately focused: when the Stiefel basis is the right inductive bias, should its spectrum remain pinned at one, or should the spectrum be learned under explicit control? We present ManifoldFlow, an SPD-relaxed Stiefel layer that separates … view at source ↗
Figure 2
Figure 2. Motivation diagnostics on Adult MLP. (a) Pressure coherence by layer. (b) Learned eigenvalue interval over training. (c) Accuracy versus λmax(St). (d) Paired FS-Adam and MF-Adam accuracy trajectories. 2 Related work Stiefel and orthogonal optimization. Optimization on matrix manifolds provides the tangent projections, retractions, vector transports, and convergence tools used by Stiefel neural layers [1, 10]. Feasib… view at source ↗
Figure 3
Figure 3. One ManifoldFlow update. The schematic shows the forward factorization, the projected￾gradient split, the tangent update for Qt, the SPD update for St, and the final composed weight. Rejected-gradient pressure. Let GW,t = ∇WtLt and Rt = S 1/2 t . The ambient gradient with respect to Qt is G¯ t = GW,tRt. (8) The Stiefel tangent projection decomposes this matrix as G tan Q,t = G¯ t − Qtsym(Q ⊤ t G¯ t), G¯ t = G tan Q,… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Main convergence comparisons. Paired FS and MF trajectories for the headline sequence and tabular settings. on UCI benchmarks, image-derived MLPs on Fashion-MNIST and CIFAR, a Mini-Transformer feed-forward block, and convolutional classifier heads on CIFAR [36, 30, 8, …
Figure 5
Figure 5. Figure 5: Mini-Transformer FFN trajectories. Validation perplexity curves when only feed-forward linear layers are wrapped. 0 200 400 600 800 1000 Epoch 1.0 1.5 2.0 2.5 3.0 3.5 4.0 Eigen v alue of St (a) LSTM λmax(St) λmin(St) S = I initial β = 4 clip 0 200 400 600 800 1000 Epoc…
Figure 6
Figure 6. Figure 6: Spectral adaptation. (a,b) LSTM/WikiText-2 MF-Adam eigenvalue traces for St. (c) CIFAR-100 ResNet-18 per-layer eigenvalue-ratio traces. 6.6 Boundary of the paired claim The paired result should not be read as broad dominance over dense or diagonal-spectrum alternatives…
Figure 7
Figure 7. Figure 7: Pressure persistence on Adult MLP. Cosine similarity between consecutive pressure matrices Pt and Pt−1 by layer [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Additional sequence-model trajectories. Validation perplexity curves for the recurrent language-model settings in [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Tabular and image-MLP trajectories. Test accuracy curves for the settings in [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Convolutional classifier-head trajectories. Test accuracy curves for the settings in [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Summary heatmap of converged paired cells. Each cell reports the signed paired gain: perplexity reduction for language modeling and accuracy change in % for classification. convolutional classifier-head cells expose boundary cases where the Stiefel prior is less relia…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

56 extracted references · 1 canonical work pages

  1. [1]

    Absil, R

    P.-A. Absil, R. Mahony, and R. Sepulchre.Optimization Algorithms on Matrix Manifolds. Princeton University Press, Dec 2008. ISBN 9781400830244. doi: 10.1515/9781400830244. URLhttp://dx.doi.org/10.1515/9781400830244

    doi:10.1515/9781400830244 2008

  2. [2]

    Geometric means in a novel vector space structure on symmetric positive-definite matrices.SIAM Journal on Matrix Analysis and Applications, 29(1):328–347, Jan 2007

    Vincent Arsigny, Pierre Fillard, Xavier Pennec, and Nicholas Ayache. Geometric means in a novel vector space structure on symmetric positive-definite matrices.SIAM Journal on Matrix Analysis and Applications, 29(1):328–347, Jan 2007. ISSN 1095-7162. doi: 10.1137/050637996. URLhttp://dx.doi.org/10.1137/050637996

    doi:10.1137/050637996 2007

  3. [3]

    Bartlett and Shahar Mendelson

    Peter L. Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. InProceedings of the 15th Annual Conference on Computational Learning Theory, pages 224–240. Springer, 2002. doi: 10.1007/3-540-45435-7_16

    work page doi:10.1007/3-540-45435-7_16 2002

  4. [4]

    Bartlett, Dylan J

    Peter L. Bartlett, Dylan J. Foster, and Matus J. Telgarsky. Spectrally-normalized margin bounds for neural networks. InAdvances in Neural Information Processing Systems, vol- ume 30, 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/ file/b22b257ad0519d4500539da3c8bcf4dd-Paper.pdf

    2017

  5. [5]

    Riemannian adaptive optimization methods

    Gary Bécigneul and Octavian-Eugen Ganea. Riemannian adaptive optimization methods. In International Conference on Learning Representations, 2019. URL https://openreview. net/forum?id=r1eiqi09K7

    2019

  6. [6]

    Springer, 1997

    Rajendra Bhatia.Matrix Analysis. Springer, 1997. ISBN 9780387948461

    1997

  7. [7]

    Princeton University Press, 2007

    Rajendra Bhatia.Positive Definite Matrices. Princeton University Press, 2007. ISBN 9780691129181

    2007

  8. [8]

    Blackard and Denis J

    Jock A. Blackard and Denis J. Dean. Comparative accuracies of artificial neural networks and discriminant analysis in predicting forest cover types from cartographic variables.Computers and Electronics in Agriculture, 24(3):131–151, 1999. doi: 10.1016/S0168-1699(99)00046-0

    doi:10.1016/s0168-1699(99)00046-0 1999

Show all 56 references
  1. [9]

    Stochastic gradient descent on riemannian manifolds.IEEE Transactions on Automatic Control, 58(9):2217–2229, 2013

    Silvère Bonnabel. Stochastic gradient descent on riemannian manifolds.IEEE Transactions on Automatic Control, 58(9):2217–2229, 2013. doi: 10.1109/TAC.2013.2254619. 17

    2013 doi

  2. [10]

    Cambridge University Press, Mar 2023

    Nicolas Boumal.An Introduction to Optimization on Smooth Manifolds. Cambridge University Press, Mar 2023. ISBN 9781009166157. doi: 10.1017/9781009166164. URL http://dx. doi.org/10.1017/9781009166164

    2023 doi

  3. [11]

    Learning phrase representations using rnn encoder– decoder for statistical machine translation

    Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder– decoder for statistical machine translation. InProceedings of the 2014 Conference on Empirical Methods ...

    2014 doi

  4. [12]

    Modeling wine preferences by data mining from physicochemical properties.Decision Support Systems, 47(4):547–553, 2009

    Paulo Cortez, António Cerdeira, Fernando Almeida, Telmo Matos, and José Reis. Modeling wine preferences by data mining from physicochemical properties.Decision Support Systems, 47(4):547–553, 2009

    2009

  5. [13]

    Uci machine learning repository, 2017

    Dheeru Dua and Casey Graff. Uci machine learning repository, 2017. URLhttps://archive. ics.uci.edu/

    2017

  6. [14]

    Adaptive subgradient methods for online learning and stochastic optimization.Journal of Machine Learning Research, 12:2121–2159, 2011

    John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization.Journal of Machine Learning Research, 12:2121–2159, 2011. URL https://jmlr.org/papers/v12/duchi11a.html

    2011

  7. [15]

    Understanding the difficulty of training deep feedforward neural networks

    Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. InProceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 249–256. PMLR, 2010. URL https://proceedings.mlr. press/v9/glor...

    2010

  8. [16]

    Golub and Charles F

    Gene H. Golub and Charles F. Van Loan.Matrix Computations. Johns Hopkins University Press, 4 edition, 2013. ISBN 9781421407944. doi: 10.56021/9781421407944. URL http: //dx.doi.org/10.56021/9781421407944

    2013 doi

  9. [17]

    Shampoo: Preconditioned stochastic tensor optimization

    Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. InProceedings of the 35th International Conference on Machine Learning, pages 1842–1850. PMLR, 2018. URL https://proceedings.mlr.press/v80/gupta18a.html

    2018

  10. [18]

    Train faster, generalize better: Stability of stochastic gradient descent

    Moritz Hardt, Benjamin Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. InProceedings of the 33rd International Conference on Machine Learning, pages 1225–1234. PMLR, 2016. URL https://proceedings.mlr.press/v48/ hardt16.html

    2016

  11. [19]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In2015 IEEE International Conference on Computer Vision, pages 1026–1034, 2015. doi: 10.1109/ICCV .2015.123

    2015 doi

  12. [20]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778. IEEE, June 2016. doi: 10.1109/CVPR.2016.90. URLhttp://dx.doi.org/ 10.1109/CVPR.2016.90

    2016 doi

  13. [21]

    Orthogonal recurrent neural networks with scaled cayley transform

    Kyle Helfrich, Devin Willmott, and Qiang Ye. Orthogonal recurrent neural networks with scaled cayley transform. InProceedings of the 35th International Conference on Machine Learning, pages 1969–1978. PMLR, 2018. URL https://proceedings.mlr.press/v80/ helfrich18a.html

    1969

  14. [22]

    Higham.Functions of Matrices: Theory and Computation

    Nicholas J. Higham.Functions of Matrices: Theory and Computation. Society for Industrial and Applied Mathematics, January 2008. ISBN 9780898717778. doi: 10.1137/1.9780898717778. URLhttp://dx.doi.org/10.1137/1.9780898717778

    2008 doi

  15. [23]

    Long short-term memory.Neural Computation, 9 (8):1735–1780, 1997

    Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory.Neural Computation, 9 (8):1735–1780, 1997. doi: 10.1162/neco.1997.9.8.1735

    1997 doi

  16. [24]

    Orthogonal weight normalization: Solution to optimization over multiple dependent stiefel manifolds in deep neural networks, 2017

    Lei Huang, Xianglong Liu, Bo Lang, Adams Wei Yu, Yongliang Wang, and Bo Li. Orthogonal weight normalization: Solution to optimization over multiple dependent stiefel manifolds in deep neural networks, 2017. URLhttps://arxiv.org/abs/1709.06079. 18

    2017 arXiv

  17. [25]

    Batch normalization: Accelerating deep network training by reducing internal covariate shift

    Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. InProceedings of the 32nd International Conference on Machine Learning, pages 448–456. PMLR, 2015. URL https://proceedings.mlr.press/ v37/ioffe15.html

    2015

  18. [26]

    Gated orthogonal recurrent units: On learning to forget, 2017

    Li Jing, Caglar Gulcehre, John Peurifoy, Yichen Shen, Max Tegmark, Marin Soljacic, and Yoshua Bengio. Gated orthogonal recurrent units: On learning to forget, 2017. URL https: //arxiv.org/abs/1706.02761

    2017 arXiv

  19. [27]

    Muon: An optimizer for hidden layers in neural networks

    Keller Jordan. Muon: An optimizer for hidden layers in neural networks. GitHub repository,

  20. [28]

    URLhttps://github.com/KellerJordan/Muon

  21. [29]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014. URL https://arxiv.org/abs/1412.6980

    2014 arXiv

  22. [30]

    Geoopt: Riemannian optimization in pytorch.arXiv preprint arXiv:2005.02819, 2020

    Maxim Kochurov, Rasul Karimov, and Serge Kozlukov. Geoopt: Riemannian optimization in pytorch.arXiv preprint arXiv:2005.02819, 2020. URL https://arxiv.org/abs/2005. 02819

    2005 arXiv

  23. [31]

    Adult data set, 1996

    Ron Kohavi and Barry Becker. Adult data set, 1996. URL https://archive.ics.uci.edu/ ml/datasets/adult

    1996

  24. [32]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. URLhttps://www.cs.toronto.edu/~kriz/ learning-features-2009-TR.pdf

    2009

  25. [33]

    Gradient-based learning applied to document recognition.Proceedings of the IEEE, 86(11):2278–2324, 1998

    Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition.Proceedings of the IEEE, 86(11):2278–2324, 1998. doi: 10.1109/5.726791

    1998 doi

  26. [34]

    Trivializations for gradient-based optimization on manifolds, 2019

    Mario Lezcano-Casado. Trivializations for gradient-based optimization on manifolds, 2019. URLhttps://arxiv.org/abs/1909.09501

    2019 arXiv

  27. [35]

    Intrinsic muon: Spectral optimization on riemannian matrix manifolds, 2026

    Yibang Li, Bihari Lal Pandey, Ravi Sah, Andi Han, Cyrus Mostajeran, Pratik Jawanpuria, and Bamdev Mishra. Intrinsic muon: Spectral optimization on riemannian matrix manifolds, 2026. URLhttps://arxiv.org/abs/2605.09238

    2026 arXiv

  28. [36]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InInternational Conference on Learning Representations, 2019. URL https://openreview.net/forum? id=Bkg6RiCqY7

    2019

  29. [37]

    Pointer sentinel mixture models, 2016

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016. URLhttps://arxiv.org/abs/1609.07843

    2016 arXiv

  30. [38]

    Efficient orthogonal parametrisation of recurrent neural networks using householder reflections

    Zakaria Mhammedi, Andrew Hellicar, Ashfaqur Rahman, and James Bailey. Efficient orthogonal parametrisation of recurrent neural networks using householder reflections. InProceedings of the 34th International Conference on Machine Learning, pages 2401–2409. PMLR, 2017. URL https...

    2017

  31. [39]

    All you need is a good init

    Dmytro Mishkin and Jiri Matas. All you need is a good init. InInternational Conference on Learning Representations, 2016. URLhttps://arxiv.org/abs/1511.06422

    2016 arXiv

  32. [40]

    Spectral normalization for generative adversarial networks, 2018

    Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks, 2018. URLhttps://arxiv.org/abs/1802.05957

    2018 arXiv

  33. [41]

    A differential geometric approach to the geometric mean of symmetric positive- definite matrices.SIAM Journal on Matrix Analysis and Applications, 26(3):735–747, 2005

    Maher Moakher. A differential geometric approach to the geometric mean of symmetric positive- definite matrices.SIAM Journal on Matrix Analysis and Applications, 26(3):735–747, 2005. doi: 10.1137/S0895479803436937

    2005 doi

  34. [42]

    Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. InProceedings of the 27th International Conference on Machine Learning, pages 807–814, 2010. URLhttps://icml.cc/Conferences/2010/papers/432.pdf. 19

    2010

  35. [43]

    A riemannian framework for tensor com- puting.International Journal of Computer Vision, 66(1):41–66, Jan 2006

    Xavier Pennec, Pierre Fillard, and Nicholas Ayache. A riemannian framework for tensor com- puting.International Journal of Computer Vision, 66(1):41–66, Jan 2006. ISSN 1573-1405. doi: 10.1007/s11263-005-3222-z. URLhttp://dx.doi.org/10.1007/s11263-005-3222-z

    2006 doi

  36. [44]

    Schoenholz, and Surya Ganguli

    Jeffrey Pennington, Samuel S. Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: Theory and practice. InAdvances in Neural Information Processing Systems, volume 30, 2017. URL https://proceedings.neurips.cc/paper_ files/paper/2...

    2017

  37. [45]

    Reddi, Satyen Kale, and Sanjiv Kumar

    Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In International Conference on Learning Representations, 2018. URL https://openreview. net/forum?id=ryQu7f-RZ

    2018

  38. [46]

    A stochastic approximation method.The Annals of Mathematical Statistics, 22(3):400–407, 1951

    Herbert Robbins and Sutton Monro. A stochastic approximation method.The Annals of Mathematical Statistics, 22(3):400–407, 1951. doi: 10.1214/aoms/1177729586

    1951 doi

  39. [47]

    The perceptron: A probabilistic model for information storage and organiza- tion in the brain.Psychological Review, 65(6):386–408, 1958

    Frank Rosenblatt. The perceptron: A probabilistic model for information storage and organiza- tion in the brain.Psychological Review, 65(6):386–408, 1958. doi: 10.1037/h0042519

    1958 doi

  40. [48]

    Rumelhart, Geoffrey E

    David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by back-propagating errors.Nature, 323(6088):533–536, 1986. doi: 10.1038/323533a0

    1986 doi

  41. [49]

    Tim Salimans and Diederik P. Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks, 2016. URL https://arxiv.org/abs/1602. 07868

    2016

  42. [50]

    Saxe, James L

    Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. InInternational Conference on Learning Representations, 2014. URLhttps://openreview.net/forum?id=_wzZwKpTDF_9C

    2014

  43. [51]

    Dropout: A simple way to prevent neural networks from overfitting.Journal of Machine Learning Research, 15(56):1929–1958, 2014

    Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhut- dinov. Dropout: A simple way to prevent neural networks from overfitting.Journal of Machine Learning Research, 15(56):1929–1958, 2014. URL https://jmlr.org/papers/ v15/srivastava14a.html

    1929

  44. [52]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2017. URL https://arxiv. org/abs/1706.03762

    2017 arXiv

  45. [53]

    On orthogonality and learning recurrent networks with long term dependencies

    Eugene V orontsov, Chiheb Trabelsi, Samuel Kadoury, and Chris Pal. On orthogonality and learning recurrent networks with long term dependencies. InProceedings of the 34th International Conference on Machine Learning, pages 3570–3578. PMLR, 2017. URL https://proceedings.mlr.pre...

    2017

  46. [54]

    A feasible method for optimization with orthogonality constraints

    Zaiwen Wen and Wotao Yin. A feasible method for optimization with orthogonality constraints. Mathematical Programming, 142(1–2):397–434, Aug 2012. ISSN 1436-4646. doi: 10.1007/ s10107-012-0584-1. URLhttp://dx.doi.org/10.1007/s10107-012-0584-1

    2012 doi

  47. [55]

    Full-capacity unitary recurrent neural networks

    Scott Wisdom, Thomas Powers, John Hershey, Jonathan Le Roux, and Les Atlas. Full-capacity unitary recurrent neural networks. InAdvances in Neural Information Processing Systems, volume 29, 2016. URL https://proceedings.neurips.cc/paper_files/paper/2016/ file/44ec5716c9b803c27a...

    2016

  48. [56]

    LnetY l=1 λmax(Sl) #

    Han Xiao, Kashif Rasul, and Roland V ollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.arXiv preprint arXiv:1708.07747, 2017. 20 A Theory and Proofs This appendix proves the statements in Section 5. All wrapped weights are written as W(...

    2017 arXiv

Pith tools

Reviewed July 11, 2026 · model on record in the stance chip above.