Pith. sign in

REVIEW 2 major objections 4 minor 1 cited by

Efficient Optimization with Orthogonality Constraint: a Randomized Riemannian Submanifold Method

T0 review · 2 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Low-dimensional random submanifold updates cut the retraction cost of Riemannian optimization under orthogonality constraints from $O(np^2)$ to $O(r^3)$ while preserving convergence.

desk verdict A solid and useful randomized submanifold method for orthogonality-constrained problems whose expectation guarantees hold up, but whose high-probability orthogonal guarantee is overstated for small r due to a reversed inequality on 1−τ. read the letter →

arxiv 2505.12378 v1 pith:HUE2GVVT submitted 2025-05-18 math.OC cs.LGstat.ML

Andi Han , Pierre-Louis Poirion , Akiko Takeda This is my paper · ORCID
classification math.OCcs.LGstat.ML MSC 90C3065K0590C26
keywords randomizedRiemanniansubmanifolddescentStiefelmanifoldorthogonalityconstraintretractioncomplexitypermutationsamplingorthogonalPolyak-Lojasiewiczconditionstochasticoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Riemannian optimization on the Stiefel manifold — the set of $n\times p$ column-orthonormal matrices — routinely pays for a retraction that costs at least $O(np^2)$ per iteration, which becomes prohibitive as $n$ and $p$ grow. This paper proposes instead to move along a low-dimensional random submanifold of the orthogonal group, using an update of the form $X_{k+1}=U_k(Y_k)X_k$ with $U_k(Y)=P_k^\top\mathrm{diag}(Y,I_{n-r})P_k$; the retraction is then performed on $O(r)$ instead of $O(n)$, at cost $O(r^3)$. The authors prove that the expected squared norm of the submanifold gradient is exactly $\frac{r(r-1)}{n(n-1)}$ times the squared norm of the full Riemannian gradient, and from this identity they derive convergence to an $\epsilon$-stationary point in $O(n^2 r^{-2}\epsilon^{-2})$ iterations, with per-iteration cost $O(nr^2)$ for permutation sampling and $O(npr)$ for orthogonal sampling. If the analysis is right, RSDM is a scalable alternative to retraction-based Riemannian gradient descent whose total complexity matches RGD when $p$ is on the order of $n$.

What carries the argument

The load-bearing device is the conjugation parameterization $U_k(Y)=P_k^\top\mathrm{diag}(Y,I_{n-r})P_k$ together with the gradient projection identity $\mathrm{grad}\tilde F_k(I_r)=P_k(r)\,\mathrm{grad}F_k(I_n)\,P_k(r)^\top$, where $P_k(r)$ is the first $r$ rows of the sampled matrix. This identity turns a random step on $O(r)$ into a two-sided random projection of the full skew-symmetric gradient; the expectation of its squared norm is exactly the ratio $r(r-1)/(n(n-1))$, the factor that appears in every iteration bound. Lemma 5.2 transfers Assumption 5.1, bounded Euclidean gradient and Hessian in the ambient space, into $L$-smoothness of the pulled-back function on $O(r)$, and Lemma 5.3 controls the full gradient by the pulled-back gradient, so the descent inequality is the same as for Riemannian gradient descent with an extra dimension ratio.

What would settle it

Sample $P$ uniformly from the Haar distribution on $O(n)$ and, for a fixed skew-symmetric matrix $M$, estimate $\mathbb{E}\|P(r)MP(r)^\top\|^2/\|M\|^2$ by Monte Carlo; Proposition 5.5 predicts exactly $r(r-1)/(n(n-1))$, and the same value for uniform permutation sampling. A stable discrepancy beyond sampling error would falsify the identity on which every convergence theorem in the paper depends.

Watch

Extended reading notes

Core claim

The paper's central claim is that the retraction bottleneck in optimization over the Stiefel manifold $\mathrm{St}(n,p)=\{X\in\mathbb{R}^{n\times p}:X^\top X=I_p\}$ can be bypassed by randomizing the search direction. Each iteration samples an orthogonal or permutation matrix $P_k$ and restricts the update to the random submanifold $U_k(Y)=P_k^\top\mathrm{diag}(Y,I_{n-r})P_k$ with $Y\in O(r)$; the next iterate is $X_{k+1}=U_k(Y_k)X_k$. The key quantitative fact is Proposition 5.5: for either sampling distribution, $\mathbb{E}\|\mathrm{grad}\tilde F_k(I_r)\|^2 = \frac{r(r-1)}{n(n-1)}\|\mathrm{grad}F_k(I_n)\|^2$, and Lemma 5.3 gives $\|\mathrm{grad}F_k(I_n)\|^2\ge \tfrac{1}{2}\|\mathrm{grad}F(X_k)\|^2$. Theorem 5.7 then yields the expected sublinear rate $\min_{i<k}\mathbb{E}\|\mathrm{grad}F(X_i)\|^2\le \frac{4L\Delta_0}{k}\frac{n(n-1)}{r(r-1)}$, together with a linear rate under the Riemannian PL condition, a stochastic counterpart, and an exact-convergence variant when the sampled submanifolds satisfy a non-degeneracy condition. The same scheme extends to quotient manifolds such as Grassmann and flag manifolds.

Load-bearing premise

Everything rests on Assumption 5.1, which requires the objective to have bounded Euclidean gradient and Hessian in the ambient matrix space; if the objective is only smooth on the manifold, or is nonsmooth, the pulled-back function need not be $L$-smooth and the descent inequalities used throughout fail.

Editorial extensions

If this is right

  • Both permutation and orthogonal sampling reach an $\epsilon$-stationary point in expectation in $O(n^2 r^{-2}\epsilon^{-2})$ iterations (Theorem 5.7).
  • Per-iteration cost falls to $O(nr^2)$ for permutation sampling and $O(npr)$ for orthogonal sampling, so total complexity can match the $O(np^2\epsilon^{-2})$ of Riemannian gradient descent when $p=\Omega(n)$.
  • Under the Riemannian PL condition the method converges linearly in expectation, with per-iteration contraction factor $1-\frac{\mu}{2L}\frac{r(r-1)}{n(n-1)}$.
  • A stochastic version with unbiased bounded-variance gradients converges at rate $O(n^2 r^{-2}K^{-1/2})$ up to a variance floor.
  • The construction transfers to Grassmann and flag manifolds through the quotient representation $O(n)/K$, because the pulled-back functions live on $O(n)$.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the gradient-ratio identity is dimension-exact rather than asymptotic, RSDM could be combined with variance reduction or momentum to obtain estimators whose bias is a known constant factor; the paper's own momentum experiment is a first step in that direction.
  • For thin problems with $p\ll n$, the total complexity loses to RGD's $O(np^2)$; a hybrid that switches to full-rank updates when $p$ is small would preserve both regimes, which the paper leaves as future work.
  • Assumption 5.1 is stated in ambient Euclidean terms; if smoothness could be relaxed to Riemannian-only smoothness, the method would apply to a broader class of objective functions without changing the sampling scheme.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes RSDM, a randomized Riemannian submanifold method for minimizing a smooth function F over the Stiefel manifold St(n,p). At each iteration the algorithm samples an orthogonal or permutation matrix P_k, restricts the update to a random O(r) submanifold through the parameterization U_k(Y)=P_k^T diag(Y,I_{n-r})P_k, performs a retraction step on O(r) using the projected Riemannian gradient, and sets X_{k+1}=U_k(Y_k)X_k. The advertised contributions are a reduction in per-iteration cost from O(np^2) to O(nr^2) (permutation sampling) or O(npr) (orthogonal sampling), convergence guarantees in expectation and high probability for general nonconvex objectives, linear convergence under a Riemannian PL condition, stochastic and finite-sum extensions, a deterministic variant with exact convergence, and an extension to quotient manifolds such as Grassmann and flag manifolds. The theoretical core is Proposition 5.5, which identifies the expected squared norm of the projected gradient as r(r-1)/(n(n-1)) times the full-space quantity, and the subsequent descent and telescoping arguments. The paper also reports experiments on Procrustes problems, PCA, quadratic assignment, and orthogonal neural networks, with code made available.

Significance. If the results stand, RSDM is a genuinely useful approach for large-scale optimization with orthogonality constraints: it has a simple implementation, a clean and parameter-free gradient-scaling identity (Proposition 5.5), and a total complexity that matches Riemannian gradient descent in the regime p=Omega(n), while avoiding expensive retractions on the full Stiefel manifold. The paper is also valuable for clearly isolating the trade-off between cheap permutation sampling and more expensive orthogonal sampling. The expectation bounds, PL analysis, stochastic analysis, and quotient-manifold extension are well structured. The main weakness is in the high-probability statements: the factor 1-tau in Theorem 5.9 is not uniformly Theta(1), so the advertised high-probability rate and the claimed O(1) iteration overhead in Remark 5.10 and Table 1 are overstated. The expectation-based central claims are not affected by this issue.

major comments (2)
  1. [Theorem 5.9, Remark 5.10, Table 1] The orthogonal high-probability bound is presented as O(n^2/(r^2 k)) with success probability at least 1-exp(-(1-tau)k/8), where tau = exp(-r^2(r-1)^2/(2048 n^2(n-1)^2)). Remark 5.10 claims that tau in (0,0.9995) implies 1-tau = Theta(1), but the inequality runs in the opposite direction: for all admissible r one has 0 < 1-tau <= 1-exp(-1/2048) approx 4.88e-4, and for r << n one has 1-tau = O(r^4/n^4). Consequently the high-probability gradient bound in Theorem 5.9 contains an omitted factor 1/(1-tau), the stated success probability is non-negligible only once k = Omega(n^4/r^4), and the Table 1 high-probability entry O(n^2/(r^2 k)) together with the orthogonal/permutation trade-off in Remark 5.10 is not supported as written. The statement of Theorem 5.9 with the explicit 1/(1-tau) is formally valid, and the expectation guarantee in Theorem 5.7 is unaffected; the issue is in the advertised simplification and in Table 1.
  2. [Appendix D.4, proof of Proposition D.1] The displayed concentration inequality P(h(X) <= r(r-1)/(n(n-1))||M||^2 - t) <= exp(-(n-1)t^2/(512||M||^4)) is not algebraically consistent with the stated conclusion exp(-r^2(r-1)^2/(2048 n^2(n-1)^2)). Substituting t = (1/2) r(r-1)/(n(n-1)) ||M||^2 gives exponent -r^2(r-1)^2/(2048 n^2(n-1)), which differs from the stated exponent by a factor (n-1). Since this tau is exactly the quantity used in Theorem 5.9, the proof needs to be corrected: either the concentration inequality as quoted from Gotze and Sambale has a different coefficient, or the exponent in Proposition D.1 must be revised.
minor comments (4)
  1. [Appendix D.3, Eq. (9)] In the orthogonal-sampling proof of Proposition 5.5, the displayed line after Eq. (8) uses a plus sign before E[P_{i1}P_{j1}P_{i2}P_{j2}]; from Eq. (8) this term is negative, and the final value E||grad eF_X(I_r)||^2 = (r^2-r)/(n^2-n) ||grad F_X(I_n)||^2 requires a minus sign. The final result is correct, but the displayed equality as written is not.
  2. [Table 1] The permutation-sampling entry in the EXPECTATION columns appears to include the binomial factor binom(n,r), which would contradict Theorem 5.7 and Remark 5.8 where the expectation rate is O(n^2/(r^2 k)) for both sampling strategies. Please align the table with the theorem statements.
  3. [Assumption 5.1] The sentence 'Assumption 5.1 is naturally satisfied given X in St(n,p), which is a compact submanifold' is only valid if F is defined and smooth on a neighborhood of the Stiefel manifold. If F is only Riemannian smooth on St(n,p), the assumed bound on the ambient Euclidean Hessian is an additional regularity condition and should be stated as such.
  4. [Remark 5.10] The expression 'k >= 4000 log(1/delta') delta'^{-2} = eOmega(1)' appears to contain a typo: the right-hand side should be a constant depending on delta', not a quantity that grows with n and r. This is related to the incorrect Theta(1) claim for 1-tau in the same remark.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: RSDM rates are derived from smoothness assumptions, exact projected-gradient moment computations, and external concentration/retraction results.

full rationale

I walked the derivation chain of the paper. The main convergence result (Theorem 5.7) is assembled from: (i) Lemma 5.2, which derives smoothness of the pulled-back function from Assumption 5.1; (ii) Lemma 5.3, an inequality relating the gradient of F_X at the identity to the Riemannian gradient of F; (iii) Proposition 5.5, which computes the expectation of the squared projected-gradient norm exactly for both permutation and orthogonal sampling; and (iv) a standard descent lemma. None of these steps defines the target quantity in terms of itself, and none involves fitting a parameter to data and then renaming it a prediction. The external ingredients — the Götze–Sambale concentration inequality, Vershynin's Chernoff bound, the retraction bound from Chen et al., and the Rebjock–Boumal quadratic-growth lemma — are independent of the authors and are not replaced by self-citation. The self-citations present (Han et al. for coordinate descent, the encyclopedia entry, and Fuji et al. for a heuristic explanation of observed speedup) are contextual or explicitly speculative, and they do not carry the proof. No ansatz is smuggled in via citation: the submanifold parameterization is defined in the paper and its gradient formula is derived directly. The claim that Riemannian coordinate descent is a special case is a remark, not a load-bearing premise. The high-probability statement in Theorem 5.9 follows the same descent lemma with a concentration argument; its proof is self-contained relative to the stated lemmas. I found no circular step. (Remark 5.10's assertion that 1−τ = Θ(1) appears arithmetically questionable for small r, but this is a correctness/quantification concern, not circularity.)

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The central claims rely only on standard smoothness and variance assumptions, external concentration and retraction bounds, and an optional local PL condition. No free parameters are fitted to data; the step size is set to 1/L and r is a user-chosen submanifold dimension, not a fitted constant. No new entities are postulated.

assumptions (6)
  • domain assumption Assumption 5.1: F has bounded Euclidean gradient and Hessian on the ambient space (C0, C1).
    Used to prove L = C0 + C1 smoothness of the pulled-back function on O(r); all descent lemmas depend on it.
  • domain assumption Assumption 5.12: stochastic gradients are unbiased and have bounded variance sigma^2.
    Standard for stochastic convergence analysis; used in Theorem 5.13.
  • standard math Concentration inequality for Lipschitz functions on St(n,r) from Gotze & Sambale (2023).
    Provides the high-probability lower bound on the projected gradient norm for orthogonal sampling in Proposition D.1.
  • standard math Retraction norm bound ||Retr_X(U) - X|| <= M||U|| from Chen et al. (2020), Lemma F.2.
    Used in the exact convergence proof (Theorem 5.11) to bound the deviation of inner iterates.
  • domain assumption Local Riemannian PL condition and the assumption that iterates enter a neighborhood U of an isolated local minimizer.
    Required for the linear convergence results in Theorems 5.7 and 5.9; the general nonconvex result does not need it.
  • standard math QR decomposition of a random Gaussian r x n matrix yields the first r rows of a Haar-uniform orthogonal matrix.
    Justifies the sampling procedure for orthogonal sampling; standard result from Meckes (2019).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Optimization with Orthogonality Constraint: a Randomized Riemannian Submanifold Method." pith.science (2026). https://pith.science/paper/HUE2GVVT

@misc{pith2026250512378,
  author       = {Pith},
  title        = {Pith review of: Efficient Optimization with Orthogonality Constraint: a Randomized Riemannian Submanifold Method},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HUE2GVVT}},
  note         = {Machine review of arXiv:2505.12378}
}
read the original abstract

Optimization with orthogonality constraints frequently arises in various fields such as machine learning. Riemannian optimization offers a powerful framework for solving these problems by equipping the constraint set with a Riemannian manifold structure and performing optimization intrinsically on the manifold. This approach typically involves computing a search direction in the tangent space and updating variables via a retraction operation. However, as the size of the variables increases, the computational cost of the retraction can become prohibitively high, limiting the applicability of Riemannian optimization to large-scale problems. To address this challenge and enhance scalability, we propose a novel approach that restricts each update on a random submanifold, thereby significantly reducing the per-iteration complexity. We introduce two sampling strategies for selecting the random submanifolds and theoretically analyze the convergence of the proposed methods. We provide convergence results for general nonconvex functions and functions that satisfy Riemannian Polyak-Lojasiewicz condition as well as for stochastic optimization settings. Additionally, we demonstrate how our approach can be generalized to quotient manifolds derived from the orthogonal manifold. Extensive experiments verify the benefits of the proposed method, across a wide variety of problems.

Figures

Figures reproduced from arXiv: 2505.12378 by the authors.

Figure 1
Figure 1. Proposed random sub￾manifold method on 2-sphere. Each iteration restricts the update to a 1-dimensional randomly se￾lected submanifold, i.e., a circle. where the matrix variable X ∈ R n×p with n ≥ p is column orthonormal and F : R n×p → R. Optimization with orthog￾onality constraint arises naturally in various domains of ap￾plications because it is crucial for achieving certain desired properties, such as linear ind… view at source ↗
Figure 2
Figure 2. Experiments on Procrustes problem and PCA problem under various settings. The numbers in brackets represent [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Experiment results on PCA (n = 2000, p = 1500) by (a) varying low-dimension r and (b) random seed with r = 700. The results suggest the outperformance of proposed RSDM over RGD is robust to changes in r as well as random seed. 0 5 10 15 20 25 30 35 Time 10 2 10 1 10 0 Optimality Gap RGD Landing PCAL RSDM-P RSDM-O (a) QUAD [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: Test accuracy for training orthogonal neural network (O-FNN) and orthogonal vision transformer (O-ViT) on [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Convergence in terms of iteration on Procrustes problem and PCA problem and quadratic assignment problem [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Convergence in loss plot for on image classification. [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Experiments on PCA problem with various settings of [PITH_FULL_IMAGE:figures/full_fig_p027_8.png]
Figure 9
Figure 9. Figure 9: Comparison of proposed RSDM with RSSM (Cheung et al., 2024) on the PCA problem. We observe RSDM converges significantly faster than RSSM. 3. Update the columns of Xk in C by projected gradient descent while keeping other columns the same: X k+1 C = ProjSt(n,|C|) (Xk C …
Figure 10
Figure 10. Figure 10: Comparison of proposed RSDM with OBCD (Yuan, 2023) on the PCA problem. We observe RSDM converges significantly faster than OBCD. The key difference is that we have a skew-symmetric operation for ∇F(Xk)X⊤ k , which renders the update direction properly defined as Riema…
Figure 11
Figure 11. Figure 11: Comparison of RSDM with momentum with RGD with mo￾mentum on the PCA problem. where Zk = Uk(Y )Xk = Xk + Pk(r) ⊤(Y − Ir)Pk(r)Xk. To test the feasibility of the proposed algorithm, we evaluate RSDM with mo￾mentum (RSDM-M) with permutation sampling and compare against RG…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. An Embarrassingly Simple Way to Optimize Orthogonal Matrices at Scale

    cs.LG 2026-02 conditional novelty 6.0 of 10

    POGO uses a two-step tangent-plus-normal update with lambda = 1/2 to keep iterates near the Stiefel manifold at the cost of five matrix multiplications, making large-scale orthogonality constraints practical.

Reference graph

Works this paper leans on

80 extracted references · 62 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  3. [3]

    and Peyr \'e , G

    Ablin, P. and Peyr \'e , G. Fast and accurate optimization on the orthogonal manifold without retraction. In International Conference on Artificial Intelligence and Statistics, pp.\ 5636--5657. PMLR, 2022

    work page 2022

  4. [5]

    and Malick, J

    Absil, P.-A. and Malick, J. Projection-like retractions on matrix manifolds. SIAM Journal on Optimization, 22 0 (1): 0 135--158, 2012

    work page 2012

  5. [6]

    G., and Gallivan, K

    Absil, P.-A., Baker, C. G., and Gallivan, K. A. Trust-region methods on R iemannian manifolds. Foundations of Computational Mathematics, 7: 0 303--330, 2007

    work page 2007

  6. [7]

    Optimization algorithms on matrix manifolds

    Absil, P.-A., Mahony, R., and Sepulchre, R. Optimization algorithms on matrix manifolds. Princeton University Press, 2008

    work page 2008

  7. [8]

    Adaptive regularization with cubics on manifolds

    Agarwal, N., Boumal, N., Bullins, B., and Cartis, C. Adaptive regularization with cubics on manifolds. Mathematical Programming, 188: 0 85--134, 2021

    work page 2021

  8. [9]

    and Sra, S

    Ahn, K. and Sra, S. From N esterov’s estimate sequence to R iemannian acceleration. In Conference on Learning Theory, pp.\ 84--118. PMLR, 2020

    work page 2020

Show all 80 references
  1. [10]

    Momentum improves optimization on R iemannian manifolds

    Alimisis, F., Orvieto, A., Becigneul, G., and Lucchi, A. Momentum improves optimization on R iemannian manifolds. In International Conference on Artificial Intelligence and Statistics, pp.\ 1351--1359. PMLR, 2021

    2021

  2. [11]

    Stochastic gradient descent on R iemannian manifolds

    Bonnabel, S. Stochastic gradient descent on R iemannian manifolds. IEEE Transactions on Automatic Control, 58 0 (9): 0 2217--2229, 2013

    2013

  3. [12]

    An introduction to optimization on smooth manifolds

    Boumal, N. An introduction to optimization on smooth manifolds. Cambridge University Press, 2023

    2023

  4. [13]

    and Cartis, C

    Boumal, N. and Cartis, C. Global rates of convergence for nonconvex optimization on manifolds. IMA Journal of Numerical Analysis, 39 0 (1): 0 1--33, 2019

    2019

  5. [14]

    E., Karisch, S

    Burkard, R. E., Karisch, S. E., and Rendl, F. QAPLIB --a quadratic assignment problem library. Journal of Global optimization, 10: 0 391--403, 1997

    1997

  6. [15]

    Continual learning in low-rank orthogonal subspaces

    Chaudhry, A., Khan, N., Dokania, P., and Torr, P. Continual learning in low-rank orthogonal subspaces. Advances in Neural Information Processing Systems, 33: 0 9900--9911, 2020

    2020

  7. [16]

    Chen, M.-S., Wang, C.-D., Huang, D., Lai, J.-H., and Yu, P. S. Efficient orthogonal multi-view subspace clustering. In ACM SIGKDD conference on knowledge discovery and data mining, pp.\ 127--135, 2022

    2022

  8. [17]

    Proximal gradient method for nonsmooth optimization over the S tiefel manifold

    Chen, S., Ma, S., Man-Cho So, A., and Zhang, T. Proximal gradient method for nonsmooth optimization over the S tiefel manifold. SIAM Journal on Optimization, 30 0 (1): 0 210--239, 2020

    2020

  9. [18]

    Y.-M., Wang, J., Yue, M.-C., and So, A

    Cheung, A. Y.-M., Wang, J., Yue, M.-C., and So, A. M.-C. Randomized submanifold subgradient method for optimization over S tiefel manifolds. arXiv:2409.01770, 2024

    2024 arXiv

  10. [19]

    and Rajawat, K

    Darmwal, Y. and Rajawat, K. Low-complexity subspace-descent over symmetric positive definite manifold. arXiv:2305.02041, 2023

    2023 arXiv

  11. [20]

    A., and Smith, S

    Edelman, A., Arias, T. A., and Smith, S. T. The geometry of algorithms with orthogonality constraints. SIAM journal on Matrix Analysis and Applications, 20 0 (2): 0 303--353, 1998

    1998

  12. [21]

    O-vit: Orthogonal vision transformer

    Fei, Y., Liu, Y., Wei, X., and Chen, M. O-vit: Orthogonal vision transformer. arXiv:2201.12133, 2022

    2022 arXiv

  13. [22]

    and Oliveira, P

    Ferreira, O. and Oliveira, P. Subgradient algorithm on R iemannian manifolds. Journal of Optimization Theory and Applications, 97: 0 93--104, 1998

    1998

  14. [23]

    Convexification with bounded gap for randomly projected quadratic optimization

    Fuji, T., Poirion, P.-L., and Takeda, A. Convexification with bounded gap for randomly projected quadratic optimization. SIAM Journal on Optimization, 32 0 (2): 0 874--899, 2022. doi:10.1137/21M1433678

    2022 doi

  15. [24]

    Parallelizable algorithms for optimization problems with orthogonality constraints

    Gao, B., Liu, X., and Yuan, Y.-x. Parallelizable algorithms for optimization problems with orthogonality constraints. SIAM Journal on Scientific Computing, 41 0 (3): 0 A1949--A1983, 2019

    2019

  16. [25]

    and Gower, R

    Garrigos, G. and Gower, R. M. Handbook of convergence theorems for (stochastic) gradient methods. arXiv:2301.11235, 2023

    2023 arXiv

  17. [26]

    and Lan, G

    Ghadimi, S. and Lan, G. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23 0 (4): 0 2341--2368, 2013

    2013

  18. [27]

    and Sambale, H

    G \"o tze, F. and Sambale, H. Higher order concentration on stiefel and grassmann manifolds. Electronic Journal of Probability, 28: 0 1--30, 2023

    2023

  19. [28]

    Gutman, D. H. and Ho-Nguyen, N. Coordinate descent without coordinates: Tangent subspace descent on R iemannian manifolds. Mathematics of Operations Research, 48 0 (1): 0 127--159, 2023

    2023

  20. [29]

    and Gao, J

    Han, A. and Gao, J. Escape saddle points faster on manifolds via perturbed Riemannian stochastic recursive gradient . arXiv:2010.12191, 2020

    2010 arXiv

  21. [30]

    and Gao, J

    Han, A. and Gao, J. Improved variance reduction methods for Riemannian non-convex optimization . IEEE Transactions on Pattern Analysis and Machine Intelligence, 44 0 (11): 0 7610--7623, 2021 a

    2021

  22. [31]

    and Gao, J

    Han, A. and Gao, J. Riemannian stochastic recursive momentum method for non-convex optimization. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, pp.\ 2505--2511, 8 2021 b . URL https://doi.org/10.24963/ijcai.2021/345

    2021 doi

  23. [32]

    Nonconvex-nonconcave min-max optimization on R iemannian manifolds

    Han, A., Mishra, B., Jawanpuria, P., and Gao, J. Nonconvex-nonconcave min-max optimization on R iemannian manifolds. Transactions on Machine Learning Research, 2023 a

    2023

  24. [33]

    Riemannian accelerated gradient methods via extrapolation

    Han, A., Mishra, B., Jawanpuria, P., and Gao, J. Riemannian accelerated gradient methods via extrapolation. In International Conference on Artificial Intelligence and Statistics, pp.\ 1554--1585. PMLR, 2023 b

    2023

  25. [34]

    Riemannian Hamiltonian methods for min-max optimization on manifolds

    Han, A., Mishra, B., Jawanpuria, P., Kumar, P., and Gao, J. Riemannian Hamiltonian methods for min-max optimization on manifolds . SIAM Journal on Optimization, 33 0 (3): 0 1797--1827, 2023 c

    2023

  26. [35]

    Riemannian coordinate descent algorithms on matrix manifolds

    Han, A., Jawanpuria, P., and Mishra, B. Riemannian coordinate descent algorithms on matrix manifolds. International Conference on Machine Learning, 2024 a

    2024

  27. [36]

    Riemannian Optimization

    Han, A., Jawanpuria, P., and Mishra, B. Riemannian Optimization . In Encyclopedia of Optimization, pp.\ 1--6. Springer, 2024 b

    2024

  28. [37]

    K., and Takeda, A

    Han, A., Mishra, B., Jawanpuria, P. K., and Takeda, A. A framework for bilevel optimization on Riemannian manifolds . Advances in Neural Information Processing Systems, 37: 0 103829--103872, 2024 c

    2024

  29. [38]

    Orthogonal recurrent neural networks with scaled cayley transform

    Helfrich, K., Willmott, D., and Ye, Q. Orthogonal recurrent neural networks with scaled cayley transform. In International Conference on Machine Learning, pp.\ 1969--1978. PMLR, 2018

    1969

  30. [39]

    Analysis of a complex of statistical variables into principal components

    Hotelling, H. Analysis of a complex of statistical variables into principal components. Journal of Educational Psychology, 24 0 (6): 0 417, 1933

    1933

  31. [40]

    A R iemannian block coordinate descent method for computing the projection robust W asserstein distance

    Huang, M., Ma, S., and Lai, L. A R iemannian block coordinate descent method for computing the projection robust W asserstein distance. In International Conference on Machine Learning, pp.\ 4446--4455. PMLR, 2021

    2021

  32. [41]

    A., and Absil, P.-A

    Huang, W., Gallivan, K. A., and Absil, P.-A. A Broyden class of quasi-Newton methods for Riemannian optimization . SIAM Journal on Optimization, 25 0 (3): 0 1660--1685, 2015

    2015

  33. [42]

    Huang, W., Absil, P.-A., and Gallivan, K. A. A R iemannian bfgs method without differentiated retraction for nonconvex optimization problems. SIAM Journal on Optimization, 28 0 (1): 0 470--495, 2018

    2018

  34. [43]

    Riemannian stochastic recursive gradient algorithm

    Kasai, H., Sato, H., and Mishra, B. Riemannian stochastic recursive gradient algorithm. In International Conference on Machine Learning, pp.\ 2516--2524. PMLR, 2018

    2018

  35. [44]

    Riemannian adaptive stochastic gradient algorithms on matrix manifolds

    Kasai, H., Jawanpuria, P., and Mishra, B. Riemannian adaptive stochastic gradient algorithms on matrix manifolds. In International Conference on Machine Learning, pp.\ 3262--3271. PMLR, 2019

    2019

  36. [45]

    and Maji, P

    Khan, A. and Maji, P. Multi-manifold optimization for multi-view subspace clustering. IEEE Transactions on Neural Networks and Learning Systems, 33 0 (8): 0 3895--3907, 2021

    2021

  37. [46]

    Momentum stiefel optimizer, with applications to suitably-orthogonal attention, and optimal transport

    Kong, L., Wang, Y., and Tao, M. Momentum stiefel optimizer, with applications to suitably-orthogonal attention, and optimal transport. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=vCJ9-Ri-6xU

    2023

  38. [47]

    A stochastic subspace approach to gradient-free optimization in high dimensions

    Kozak, D., Becker, S., Doostan, A., and Tenorio, L. A stochastic subspace approach to gradient-free optimization in high dimensions. Computational Optimization and Applications, 79 0 (2): 0 339--368, 2021

    2021

  39. [48]

    Learning multiple layers of features from tiny images

    Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009

    2009

  40. [49]

    Gradient-based learning applied to document recognition

    LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86 0 (11): 0 2278--2324, 1998

    1998

  41. [50]

    and Mart nez-Rubio, D

    Lezcano-Casado, M. and Mart nez-Rubio, D. Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group. In International Conference on Machine Learning, pp.\ 3794--3803. PMLR, 2019

    2019

  42. [51]

    and Ma, S

    Li, J. and Ma, S. Riemannian bilevel optimization. Journal of Machine Learning Research, 26 0 (18): 0 1--44, 2025

    2025

  43. [52]

    Efficient riemannian optimization on the stiefel manifold via the cayley transform

    Li, J., Fuxin, L., and Todorovic, S. Efficient riemannian optimization on the stiefel manifold via the cayley transform. arXiv:2002.01113, 2020

    2002 arXiv

  44. [53]

    A R iemannian alternating direction method of multipliers

    Li, J., Ma, S., and Srivastava, T. A R iemannian alternating direction method of multipliers. Mathematics of Operations Research, 2024

    2024

  45. [54]

    Orthogonal deep neural networks

    Li, S., Jia, K., Wen, Y., Liu, T., and Tao, D. Orthogonal deep neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43 0 (4): 0 1352--1368, 2019

    2019

  46. [55]

    Projection robust W asserstein distance and R iemannian optimization

    Lin, T., Fan, C., Ho, N., Cuturi, M., and Jordan, M. Projection robust W asserstein distance and R iemannian optimization. Advances in Neural Information Processing Systems, 33: 0 9383--9397, 2020

    2020

  47. [56]

    Parameter-efficient orthogonal finetuning via butterfly factorization

    Liu, W., Qiu, Z., Feng, Y., Xiu, Y., Xue, Y., Yu, L., Feng, H., Liu, Z., Heo, J., Peng, S., et al. Parameter-efficient orthogonal finetuning via butterfly factorization. In The Twelfth International Conference on Learning Representations, 2024

    2024

  48. [57]

    One pass late fusion multi-view clustering

    Liu, X., Liu, L., Liao, Q., Wang, S., Zhang, Y., Tu, W., Tang, C., Liu, J., and Zhu, E. One pass late fusion multi-view clustering. In International conference on machine learning, pp.\ 6850--6859. PMLR, 2021

    2021

  49. [58]

    Meckes, E. S. The Random Matrix Theory of the Classical Compact Groups. Cambridge Tracts in Mathematics. Cambridge University Press, 2019

    2019

  50. [59]

    Fixed-rank matrix factorizations and R iemannian low-rank optimization

    Mishra, B., Meyer, G., Bonnabel, S., and Sepulchre, R. Fixed-rank matrix factorizations and R iemannian low-rank optimization. Computational Statistics, 29: 0 591--621, 2014

    2014

  51. [60]

    and Vidal, R

    Peng, L. and Vidal, R. Block coordinate descent on smooth manifolds: Convergence theory and twenty-one examples. arXiv:2305.14744, 2023

    2023 arXiv

  52. [61]

    A., and Absil, P.-A

    Qi, C., Gallivan, K. A., and Absil, P.-A. Riemannian BFGS algorithm with applications. In Recent Advances in Optimization and its Applications in Engineering, pp.\ 183--192. Springer, 2010

    2010

  53. [62]

    Controlling text-to-image diffusion by orthogonal finetuning

    Qiu, Z., Liu, W., Feng, H., Xue, Y., Feng, Y., Liu, Z., Zhang, D., Weller, A., and Sch \"o lkopf, B. Controlling text-to-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems, 36: 0 79320--79362, 2023

    2023

  54. [63]

    and Boumal, N

    Rebjock, Q. and Boumal, N. Fast convergence to non-isolated minima: four equivalent conditions for c2 functions. Mathematical Programming, pp.\ 1--49, 2024

    2024

  55. [64]

    and Chechik, G

    Shalit, U. and Chechik, G. Coordinate-descent for learning orthogonal matrices through G ivens rotations. In International Conference on Machine Learning, pp.\ 548--556. PMLR, 2014

    2014

  56. [65]

    and Avron, H

    Shustin, B. and Avron, H. Faster randomized methods for orthogonality constrained problems. Journal of Machine Learning Research, 25 0 (257): 0 1--59, 2024

    2024

  57. [66]

    J., Cason, T

    Theis, F. J., Cason, T. P., and Absil, P. A. Soft dimension reduction for ICA by joint diagonalization on the S tiefel manifold. In Independent Component Analysis and Signal Separation: 8th International Conference, ICA 2009, Paraty, Brazil, March 15-18, 2009. Proceedings 8, p...

    2009

  58. [67]

    Convex functions and optimization methods on R iemannian manifolds , volume 297

    Udriste, C. Convex functions and optimization methods on R iemannian manifolds , volume 297. Springer Science & Business Media, 2013

    2013

  59. [68]

    Improved Differentially Private Riemannian Optimization: Fast Sampling and Variance Reduction

    Utpala, S., Han, A., Jawanpuria, P., and Mishra, B. Improved Differentially Private Riemannian Optimization: Fast Sampling and Variance Reduction . Transactions on Machine Learning Research, 2023. ISSN 2835-8856

    2023

  60. [69]

    Low-rank matrix completion by R iemannian optimization

    Vandereycken, B. Low-rank matrix completion by R iemannian optimization. SIAM Journal on Optimization, 23 0 (2): 0 1214--1236, 2013

    2013

  61. [70]

    Optimization without retraction on the random generalized S tiefel manifold

    Vary, S., Ablin, P., Gao, B., and Absil, P.-A. Optimization without retraction on the random generalized S tiefel manifold. In International Conference on Machine Learning. PMLR, 2024

    2024

  62. [71]

    High-dimensional probability: An introduction with applications in data science, volume 47

    Vershynin, R. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018

    2018

  63. [72]

    Wang, J., Chen, Y., Chakraborty, R., and Yu, S. X. Orthogonal convolutional neural networks. In Conference on Computer Vision and Pattern Recognition, pp.\ 11505--11515, 2020

    2020

  64. [73]

    and Yin, W

    Wen, Z. and Yin, W. A feasible method for optimization with orthogonality constraints. Mathematical Programming, 142 0 (1): 0 397--434, 2013

    2013

  65. [74]

    A class of smooth exact penalty function methods for optimization problems with orthogonality constraints

    Xiao, N., Liu, X., and Yuan, Y.-x. A class of smooth exact penalty function methods for optimization problems with orthogonality constraints. Optimization Methods and Software, 37 0 (4): 0 1205--1241, 2022

    2022

  66. [75]

    A block coordinate descent method for nonsmooth composite optimization under orthogonality constraints

    Yuan, G. A block coordinate descent method for nonsmooth composite optimization under orthogonality constraints. arXiv:2304.03641, 2023

    2023 arXiv

  67. [76]

    Riemannian SVRG: Fast stochastic optimization on Riemannian manifolds

    Zhang, H., J Reddi, S., and Sra, S. Riemannian SVRG: Fast stochastic optimization on Riemannian manifolds . Advances in Neural Information Processing Systems, 29, 2016

    2016

  68. [77]

    Sion’s minimax theorem in geodesic metric spaces and a R iemannian extragradient algorithm

    Zhang, P., Zhang, J., and Sra, S. Sion’s minimax theorem in geodesic metric spaces and a R iemannian extragradient algorithm. SIAM Journal on Optimization, 33 0 (4): 0 2885--2908, 2023

    2023

  69. [78]

    and Shen, C

    Zhu, X. and Shen, C. Practical gradient and conjugate gradient methods on flag manifolds. Computational Optimization and Applications, 88 0 (2): 0 491--524, 2024

    2024

  70. [79]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  71. [80]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  72. [81]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2015 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.