Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Distributed Retraction-Free and Communication-Efficient Optimization on the Stiefel Manifold

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proposes EF-Landing, a distributed retraction-free algorithm for optimization on the Stiefel manifold that compresses communicated gradients while provably retaining the same O(1/√(NK)) linear speedup rate as uncompressed methods.

desk verdict EF-Landing is a genuine algorithmic contribution with a concrete, fixable proof gap in Lemma 5.8; it deserves review, but the theorem proofs need repair and the experiments need more rigor. read the letter →

arxiv 2506.02879 v2 pith:MFZIXBDL submitted 2025-06-03 math.OC

Yilong Song , Peijin Li , Bin Gao , Kun Yuan This is my paper
classification math.OC MSC 90C2690C3068W1565K10
keywords Stiefelmanifoldretraction-freeoptimizationLandingmethodcommunicationcompressionerrorfeedbackdistributedlinearspeedupstochastic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that distributed stochastic optimization on the Stiefel manifold, the set of matrices with orthonormal columns, can be made communication-efficient by compressing the Euclidean gradient rather than the Riemannian descent direction, provided compression is paired with momentum error feedback. This yields EF-Landing, presented as the first retraction-free and communication-efficient distributed method for such problems. The central result is that, with contractive compressors and error feedback, EF-Landing converges at the same asymptotic rate O(1/√(NK)) as the vanilla Landing method without compression, so compression does not degrade the convergence order. The analysis covers deterministic and stochastic settings, gradient descent and momentum variants, and block-wise Stiefel constraints, and the authors validate the rates numerically on distributed online PCA and orthogonally constrained neural networks.

What carries the argument

The load-bearing object is the Landing descent direction Λ(X) = grad f(X) + λ∇N(X), whose two components are Frobenius-orthogonal for every X; EF-Landing compresses the Euclidean gradient ∇f(X) and plugs the compressed g into grad(g) = skew(gX^T)X, preserving that orthogonality via Proposition 4.1. Convergence is carried by a merit function bound (Lemma 5.8) that lower-bounds ⟨Λ̃(X;g), ∇m(X)⟩, isolating the compression error ∥g − ∇f(X)∥²_F, together with a Lyapunov function that adds per-node compression error, momentum error, and master error to the merit decrease, and a uniform safe step size (Lemma 5.7) that keeps iterates inside the safe region St(p,n)ϵ.

What would settle it

For a smooth objective on a Stiefel manifold with a contractive compressor, construct an instance satisfying all assumptions except that the true gradient bound exceeds the asserted L′; if the step size chosen from the smaller L′ drives iterates out of the safe region or stops convergence, the uniform safe-step-size lemma fails. Alternatively, run the single-node toy example of Appendix A.2 with gradient ∇f = (2,1)^T, the Top-1 compressor, and starting point X = (1,0)^T: if vanilla gradient compression converges rather than stagnating, the claimed necessity of error feedback is refuted.

Watch

Extended reading notes

Core claim

The authors establish that for the Landing descent direction Λ(X) = grad f(X) + λ∇N(X), the orthogonality between the Riemannian component gradf(X) = skew(∇f(X)X^T)X and the penalty component ∇N(X) = X(X^T X − I_p) survives communication compression only if one compresses the Euclidean gradient g rather than the whole direction: ⟨skew(gX^T)X, X(X^T X − I_p)⟩ = 0 for every g. They further show that vanilla gradient compression can make iterates stagnate, because a greedy compressed gradient may lie entirely in the orthogonal complement of the tangent space, and prove that error feedback overcomes this. With momentum error feedback and gradient clipping, the merit-function analysis yields a uniform safe step size and a Lyapunov decrease giving O(1/K) in deterministic settings and O(1/√(NK)) linear speedup in stochastic settings, with the compressor's effect confined to higher-order terms.

Load-bearing premise

The entire guarantee rests on Assumption 5.1: every stochastic gradient inside the safe region is uniformly bounded by a known constant L′, which fixes the safe step size and the clipping threshold, and if no finite L′ exists or is unknown, the stated rates do not follow.

Editorial extensions

If this is right

  • EF-Landing achieves the same O(1/√(NK)) linear speedup rate as uncompressed stochastic Landing, so communication compression does not change the convergence order.
  • In deterministic settings EF-Landing converges at O(1/K), matching the vanilla Landing rate with exact constants when the compressor is the identity.
  • Error feedback is necessary for convergence under contractive compression on the Stiefel manifold, even in a deterministic single-node setting, unlike unconstrained optimization.
  • The same rates carry over to block-wise Stiefel manifolds, so orthogonally constrained neural-network layers and other structured constraints inherit the guarantees.
  • Aggressive compression (θ → 0) slows the constants but not the asymptotic rate, and the influence of the compressor appears only in higher-order terms of the stochastic bound.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The Euclidean-gradient compression insight likely transfers to other retraction-free penalty-based manifold algorithms whose descent directions share the same orthogonal decomposition into a skew-symmetric and a symmetric penalty term.
  • The stagnation phenomenon suggests a manifold-aware compressor that projects onto the current tangent space before applying Top-K or Rand-K could avoid the error-feedback overhead entirely, a testable algorithmic variant not explored in the paper.
  • Because the experiments set the clipping bound L′ to 10^8, effectively disabling clipping, the theoretical dependence on Assumption 5.1 is empirically untested; a tight L′ may change convergence behavior in ways the paper does not probe.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes EF-Landing, a distributed error-feedback version of the retraction-free Landing method for optimization on the Stiefel manifold. Workers compress their Euclidean gradient estimates with a contractive compressor, the master aggregates the compressed gradients and forms the Landing descent direction, and a clipping step keeps iterates in a safe region around the manifold. The main theoretical contributions are a deterministic O(1/K) rate and a stochastic O(1/sqrt(NK)) linear-speedup rate that match the uncompressed Landing algorithm, plus an extension to block-wise Stiefel constraints. The analysis is built on a merit function and a Lyapunov function that track momentum, stochastic, and compression errors.

Significance. If the rate statements are correct, the paper provides a clean answer to the two questions it poses: the Euclidean gradient (rather than the composed Landing direction) is the right object to compress, and error feedback is necessary. Matching the asymptotic rate of uncompressed Landing is a meaningful result, as it shows the Stiefel constraint need not degrade the linear-speedup behavior achieved by error-feedback methods in unconstrained problems. The proof is detailed and mostly self-contained, and the rates are derived rather than fitted; the reduction to vanilla Landing in Corollary 5.12 is a useful consistency check. The main caveats are a localized algebra error in the merit-function bound, a mismatch between the theoretical step-size condition and the experimental step sizes, and an unsupported claim in the block-wise extension.

major comments (3)
  1. [Appendix B.2, Eq. (41)] The displayed simplification in Eq. (41) is a false inequality. From the estimates above (40), with sigma_1 <= sqrt(1+epsilon) and L, L' <= Lhat, the negative term is bounded below by -Lhat^2 (1+epsilon) (1+sqrt(1+epsilon))^2 / (1-epsilon), not by -4 Lhat^2 (1+epsilon). The displayed denominator 2-(1-epsilon) equals 1+epsilon, whereas the valid lower bound on 2-sigma_1^2 is 1-epsilon; and (L'+L sigma_1)^2 is bounded by Lhat^2 (1+sqrt(1+epsilon))^2, not by 4(1+epsilon) Lhat^2. For epsilon=0.1, sigma_1=sqrt(1.1), L=L'=Lhat, the left side of the claimed inequality in (41) is about 5.13 Lhat^2 while the right side is 4.4 Lhat^2. Consequently the stated mu condition (42) does not follow from the displayed algebra. The gap is repairable by replacing (42) with mu >= [2L(1-epsilon) + 6 sqrt(1+epsilon) L' + (1+epsilon)(1+sqrt(1+epsilon))^2 Lhat^2/(lambda(1-epsilon))] / (3-4epsilon), which preserves the subsequent Lyapunov argument and the rates. As written, however, Lemma 5.8, and therefore Theorem 5.9 and all downstream rate theorems, rest on an unproven step.
  2. [Section 7 and Appendix D, Tables 2-8] The numerical experiments do not instantiate the step-size regime of the theorems. Lemma 5.7 requires gamma <= gamma_s, with gamma_s of order lambda(1-epsilon)epsilon / ((1+epsilon)^2 (L')^2). With the reported clipping bound L'=10^8 and lambda around 1, gamma_s is on the order of 10^{-16}, yet the experiments use gamma=1.0 or 0.1 (sometimes with decay). The statement in Section 7 that overestimating L' does not incur any loss is the opposite of the lemma's implication: overestimating L' shrinks the allowed uniform step size. The experiments may still be indicative of practical behavior, but they do not validate the theorem's step-size condition. Please either recompute gamma_s from an actual L' or state explicitly that the experiments operate outside the theoretical step-size range.
  3. [Appendix C.2, around Eq. (75)] The proof of Theorem C.1 relies on the assertion that partial h_j / partial X_k = 0 for k != j, where h_j(X_j) = (1/2) <sym(X_j^T partial f / partial X_j), X_j^T X_j - I>. This is false for a general objective f in problem (13), because partial f / partial X_j typically depends on all blocks, as it does in the convolutional networks of Section 7.2. The cross-block terms in the gradient of the block-wise merit function are therefore omitted from the lower bound (75), so the block-wise theorem is not proven as stated. Either add a block-separability assumption, for example f(X_1,...,X_J;x) = sum_j f_j(X_j) + f_0(x), or extend the merit-function calculation to include the cross terms; with the current text, Theorem C.1 is unsupported.
minor comments (5)
  1. [Section 2] There is a typo in the Related Work paragraph: 'Optimal compelxity with communicaiton compression' should be 'Optimal complexity with communication compression'.
  2. [Section 7.1 versus Appendix D.1.2] The main text says the penalty parameter lambda was set to 1 for EF-Landing, while Tables 2-5 in Appendix D.1.2 report lambda=0.5 for the same experiments; please reconcile this discrepancy.
  3. [Appendix B.2] There is a spelling error: 'sigular values' should be 'singular values' in the text around Eq. (40).
  4. [Appendix C.2, Eq. (80)] The summation over j appears to start at j=0 in the constraint-violation term, but the block-wise quantities N_j are defined only for j=1,...,J; please correct the index range.
  5. [Abstract and Section 2] The abstract describes EF-Landing as the 'first' retraction-free and communication-efficient algorithm for distributed stochastic optimization on the Stiefel manifold; in light of the cited works on compression and decentralized manifold methods, please either substantiate the firstness claim precisely or qualify it.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the EF-Landing convergence rates are derived from a Lyapunov argument and are not fitted or definitionally tied to their conclusions.

full rationale

The central convergence claims (Theorems 5.9, 5.11, 5.14, and 5.16) are obtained by telescoping a Lyapunov function built from the merit function, momentum and compression error terms, and explicit step-size conditions. The gradient-norm and feasibility rates follow from the coefficients in the Lyapunov descent inequality rather than from any fitted parameter or from a quantity defined in terms of the target rate. The paper does cite earlier Landing work, including Ablin et al. 2024, on which one current co-author (Bin Gao) also appears; that citation supplies the safe-region definition and parts of the merit-function calculus. However, this is a published, externally checkable result and does not itself assert the EF-Landing rates; the EF-Landing-specific compression and error-feedback analysis is carried out in the present appendices. No step was found in which Eq. X equals Eq. Y by construction or in which a fitted input is renamed as a prediction. A separate mathematical concern exists: the inequality in Eq. (41) of Appendix B.2 appears numerically false for some choices of epsilon and sigma_1, which would leave Lemma 5.8 unproved as written; that is a correctness or rigor issue, not a circularity issue, and it does not affect the circularity score. Similarly, the experimental use of L' = 1e8 with gamma = 1 outside the conditions of Lemma 5.7 is a validation/theory mismatch rather than circular reasoning.

Assumptions & free parameters 6 free parameters · 7 assumptions · 0 invented entities

The central claim relies on standard smoothness, bounded-variance, and contractive-compressor assumptions, plus two imported results from Ablin et al. (2024). No new physical entities are postulated; the merit function is an analytical device, not an entity with independent falsifiable content.

free parameters (6)
  • penalty coefficient lambda = 0.5 or 1.0 in experiments (8.0 for the penalty-method baseline)
    Controls feasibility penalty strength. Theory holds for any lambda > 0, but the step-size and merit-function constants depend on it.
  • merit-function coefficient mu = not instantiated; lower-bounded in Lemma 5.8
    Hand-chosen in the analysis as a function of L, L', lambda, epsilon; it does not appear in Algorithm 1 and is not tuned in experiments.
  • gradient clipping bound L' = 1e8 in experiments
    Assumed known in Assumption 5.1; experimental value is so large that clipping is effectively inactive.
  • momentum rate eta = 0.1 or 0.5 in experiments
    The theory prescribes a K-dependent schedule (Lemma B.6, Theorem 5.14); experiments grid-search it.
  • step size gamma = 0.1 or 1.0 in experiments
    Theory bounds gamma by constants depending on L, L', sigma, lambda; experiments grid-search it.
  • compressor retention ratio or quantization level = 0.1 or 0.2 for Top-K and Rand-K; 8 or 16 for QSGD
    Determines the contractive factor alpha, which enters the rate through theta = 1 - sqrt(1-alpha). Chosen per experiment, not fit to data.
assumptions (7)
  • domain assumption Assumption 5.1: uniform gradient bound ||grad F(X; xi_i)|| <= L' on the safe region St(p,n)_epsilon
    Load-bearing for Lemma 5.7 (safe step size), Lemma 5.8 (merit function bound), and the clipping operation in Algorithm 1. Fails for heavy-tailed stochastic gradients.
  • domain assumption Assumption 5.3: each local objective f_i is L_i-smooth on the safe region
    Used to bound the merit-function inner product and to derive the momentum error recursion in Lemma B.3.
  • domain assumption Assumption 5.5: f is lower bounded on the safe region
    Required for telescoping the Lyapunov function in Theorem 5.9.
  • domain assumption Assumption 5.6: unbiased stochastic gradients with bounded variance sigma^2
    Used in the momentum and compression error recursions (Lemmas B.3 and B.4).
  • domain assumption Definition 3.2: contractive compressor with factor alpha in (0,1]
    The error-feedback rates in Theorems 5.9 and 5.14 depend on alpha; compressors such as signSGD are not contractive and are outside the scope.
  • standard math Safe-region singular-value bounds and merit-function gradient formula from Ablin et al. 2024 (Lemmas B.1 and B.2)
    Imported without proof into Lemmas 5.7 and 5.8; published prior work, treated as external benchmark rather than circular.
  • standard math The merit function m is L_m-smooth on the safe region, with bound taken from Ablin et al. 2024
    Used at the start of Theorem 5.9's proof to expand m(X^{k+1}).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Distributed Retraction-Free and Communication-Efficient Optimization on the Stiefel Manifold." pith.science (2026). https://pith.science/paper/MFZIXBDL

@misc{pith2026250602879,
  author       = {Pith},
  title        = {Pith review of: Distributed Retraction-Free and Communication-Efficient Optimization on the Stiefel Manifold},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MFZIXBDL}},
  note         = {Machine review of arXiv:2506.02879}
}
read the original abstract

Optimization problems on the Stiefel manifold, ranging from principal component analysis to enhancing neural network robustness, are ubiquitous in machine learning. The Landing algorithm avoids computationally expensive retraction operations on manifolds, making it highly competitive for large-scale problems. This paper extends this method to distributed settings, introducing *EF-Landing*, the first retraction-free and communication-efficient algorithm for distributed stochastic optimization on the Stiefel manifold. By incorporating communication compression and error feedback, EF-Landing ensures convergence and constraint feasibility while significantly reducing communication overhead. We provide sharp convergence guarantees, demonstrating that EF-Landing achieves the same asymptotic linear speedup convergence rate as existing methods without communication compression. Furthermore, our analysis is highly versatile, applying to both deterministic and stochastic settings and encompassing algorithms based on gradient descent or momentum-based gradient descent. We also generalize EF-Landing to operate on block-wise Stiefel manifolds, enabling greater flexibility for structured constraints. Extensive numerical experiments validate our theoretical results.

Figures

Figures reproduced from arXiv: 2506.02879 by the authors.

Figure 1
Figure 1. The necessity of error feedback 4.1. Compression on Euclidean Gradient In the error feedback method for solving the unconstrained optimization problem (4), compressing the gradient ∇f(X) is sufficient to ensure convergence. Given that the Landing method can be expressed in a descent form (3), a natural question arises: should we mimic recursion (5) and com￾press Λ(X) for EF-Landing? We find that compressing Λ(X) doe… view at source ↗
Figure 2
Figure 2. Performance comparison of EF-Landing and other algorithms on distributed online PCA with [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Performance comparison on deep learning with ResNet-18 on CIFAR-10. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: Diagram for the toy example in Appendix A.2. The heuristic reason explaining the phenomenon of stagnation is: Greedy compressors choose the directions with largest magnitude in unconstrained Euclidean space, but these directions are not necessarily directions with larg…
Figure 5
Figure 5. Figure 5: Performance of EF-Landing and other algorithms on online PCA, with [PITH_FULL_IMAGE:figures/full_fig_p029_5.png]
Figure 6
Figure 6. Figure 6: Performance of EF-Landing and other algorithms on online PCA, with [PITH_FULL_IMAGE:figures/full_fig_p029_6.png]
Figure 7
Figure 7. Figure 7: Performance of EF-Landing and other algorithms on online PCA, with [PITH_FULL_IMAGE:figures/full_fig_p030_7.png]
Figure 8
Figure 8. Figure 8: Performance of EF-Landing and other algorithms on deep learning with VGG16 on MNIST. [PITH_FULL_IMAGE:figures/full_fig_p031_8.png]
Figure 9
Figure 9. Figure 9: Performance of EF-Landing and other algorithms on deep learning with VGG16 on CIFAR-10. [PITH_FULL_IMAGE:figures/full_fig_p032_9.png]
Figure 10
Figure 10. Figure 10: Comparison with Penalty Method: p = 100 Method/Hyperparameter γ Clipping λ compress rate /s EF-Landing-Top-K 1.0 108 0.5 0.1 EF-Landing-Random-K 1.0 108 0.5 0.1 Penalty method 0.01 – 10.0 – Penalty method 0.001 – 1000.0 – [PITH_FULL_IMAGE:figures/full_fig_p033_10.png]
Figure 11
Figure 11. Figure 11: Comparison with Penalty Method: p = 200 Method/Hyperparameter γ Clipping λ compress rate /s EF-Landing-Top-K 1.0 108 0.5 0.1 EF-Landing-Random-K 1.0 108 0.5 0.1 Penalty method 0.01 – 10.0 – Penalty method 0.001 – 1000.0 – [PITH_FULL_IMAGE:figures/full_fig_p034_11.png]
Figure 12
Figure 12. Figure 12: Comparison with Penalty Method: p = 500 Method/Hyperparameter γ Clipping λ compress rate /s EF-Landing-Top-K 1.0 108 0.5 0.1 EF-Landing-Random-K 1.0 108 0.5 0.1 Penalty method 0.01 – 10.0 – Penalty method 0.001 – 1000.0 – [PITH_FULL_IMAGE:figures/full_fig_p034_12.png]
Figure 13
Figure 13. Figure 13: Comparison with Penalty Method: p = 1000 Method/Hyperparameter γ Clipping λ compress rate /s EF-Landing-Top-K 1.0 108 0.5 0.1 EF-Landing-Random-K 1.0 108 0.5 0.1 Penalty method 0.01 – 10.0 – Penalty method 0.001 – 1000.0 – [PITH_FULL_IMAGE:figures/full_fig_p035_13.png]
Figure 14
Figure 14. Figure 14: Comparison of EF-Landing Algorithm and Penalty Method: VGG16 on CIFAR-10 [PITH_FULL_IMAGE:figures/full_fig_p036_14.png]
Figure 15
Figure 15. Figure 15: Comparison of EF-Landing Algorithm and Penalty Method: ResNet-18 on CIFAR-10 [PITH_FULL_IMAGE:figures/full_fig_p036_15.png]
Figure 16
Figure 16. Figure 16: Comparison of EF-Landing Algorithm and Decentralized Algorithm On Synthetic Dataset 1 [PITH_FULL_IMAGE:figures/full_fig_p037_16.png]
Figure 17
Figure 17. Figure 17: Comparison of EF-Landing Algorithm and Decentralized Algorithm On Synthetic Dataset 2 [PITH_FULL_IMAGE:figures/full_fig_p037_17.png]
Figure 18
Figure 18. Figure 18: Comparison of EF-Landing Algorithm and Algorithm 1 in [Zhang et al. 2024], synthetic dataset [PITH_FULL_IMAGE:figures/full_fig_p038_18.png]
Figure 19
Figure 19. Figure 19: Comparison of EF-Landing Algorithm and Algorithm 1 in [Zhang et al. 2024], synthetic dataset [PITH_FULL_IMAGE:figures/full_fig_p038_19.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. An Embarrassingly Simple Way to Optimize Orthogonal Matrices at Scale

    cs.LG 2026-02 conditional novelty 6.0 of 10

    POGO uses a two-step tangent-plus-normal update with lambda = 1/2 to keep iterates near the Stiefel manifold at the cost of five matrix multiplications, making large-scale orthogonality constraints practical.

Reference graph

Works this paper leans on

64 extracted references · 58 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    and Peyré, G

    Ablin, P. and Peyré, G. Fast and accurate optimization on the orthogonal manifold without retraction. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics , pp.\ 5636--5657. PMLR, May 2022

    work page 2022

  3. [3]

    Infeasible Deterministic , Stochastic , and Variance - Reduction Algorithms for Optimization under Orthogonality Constraints

    Ablin, P., Vary, S., Gao, B., and Absil, P.-A. Infeasible Deterministic , Stochastic , and Variance - Reduction Algorithms for Optimization under Orthogonality Constraints . Journal of Machine Learning Research, 25 0 (389): 0 1--38, 2024

    work page 2024

  4. [4]

    and Malick, J

    Absil, P.-A. and Malick, J. Projection-like Retractions on Matrix Manifolds . SIAM Journal on Optimization, 22 0 (1): 0 135--158, January 2012

    work page 2012

  5. [5]

    Trust- Region Methods on Riemannian Manifolds

    Absil, P.-A., Baker, C., and Gallivan, K. Trust- Region Methods on Riemannian Manifolds . Foundations of Computational Mathematics, 7 0 (3): 0 303--330, July 2007

    work page 2007

  6. [6]

    Optimization Algorithms on Matrix Manifolds

    Absil, P.-A., Mahony, R., and Sepulchre, R. Optimization Algorithms on Matrix Manifolds . Princeton University Press, April 2009

    work page 2009

  7. [7]

    and Sra, S

    Ahn, K. and Sra, S. From Nesterov ’s Estimate Sequence to Riemannian Acceleration . In Proceedings of Thirty Third Conference on Learning Theory , pp.\ 84--118. PMLR, July 2020

    work page 2020

  8. [8]

    Momentum Improves Optimization on Riemannian Manifolds

    Alimisis, F., Orvieto, A., Becigneul, G., and Lucchi, A. Momentum Improves Optimization on Riemannian Manifolds . In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics , pp.\ 1351--1359. PMLR, March 2021

    work page 2021

Show all 64 references
  1. [9]

    QSGD : Communication - Efficient SGD via Gradient Quantization and Encoding

    Alistarh, D., Grubic, D., Li, J., Tomioka, R., and Vojnovic, M. QSGD : Communication - Efficient SGD via Gradient Quantization and Encoding . In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017

    2017

  2. [10]

    Unitary Evolution Recurrent Neural Networks

    Arjovsky, M., Shah, A., and Bengio, Y. Unitary Evolution Recurrent Neural Networks . In Proceedings of The 33rd International Conference on Machine Learning , pp.\ 1120--1128. PMLR, June 2016

    2016

  3. [11]

    Can We Gain More from Orthogonality Regularizations in Training Deep Networks ? In Advances in Neural Information Processing Systems , volume 31

    Bansal, N., Chen, X., and Wang, Z. Can We Gain More from Orthogonality Regularizations in Training Deep Networks ? In Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018

    2018

  4. [12]

    signSGD : Compressed Optimisation for Non - Convex Problems

    Bernstein, J., Wang, Y.-X., Azizzadenesheli, K., and Anandkumar, A. signSGD : Compressed Optimisation for Non - Convex Problems . In Proceedings of the 35th International Conference on Machine Learning , pp.\ 560--569. PMLR, July 2018

    2018

  5. [13]

    M., Massena, T., and Serrurier, M

    Boissin, T., Mamalet, F., Fel, T., Picard, A. M., Massena, T., and Serrurier, M. An adaptive orthogonal convolution scheme for efficient and flexible cnn architectures. arXiv preprint arXiv:2501.07930, 2025

    2025 arXiv

  6. [14]

    Stochastic Gradient Descent on Riemannian Manifolds

    Bonnabel, S. Stochastic Gradient Descent on Riemannian Manifolds . IEEE Transactions on Automatic Control, 58 0 (9): 0 2217--2229, September 2013. Conference Name: IEEE Transactions on Automatic Control

    2013

  7. [15]

    Global rates of convergence for nonconvex optimization on manifolds

    Boumal, N., Absil, P.-A., and Cartis, C. Global rates of convergence for nonconvex optimization on manifolds. IMA Journal of Numerical Analysis, 39 0 (1): 0 1--33, January 2019

    2019

  8. [16]

    Decentralized Riemannian Gradient Descent on the Stiefel Manifold

    Chen, S., Garcia, A., Hong, M., and Shahrampour, S. Decentralized Riemannian Gradient Descent on the Stiefel Manifold . In Proceedings of the 38th International Conference on Machine Learning , pp.\ 1594--1605. PMLR, July 2021

    2021

  9. [17]

    A., and Smith, S

    Edelman, A., Arias, T. A., and Smith, S. T. The Geometry of Algorithms with Orthogonality Constraints . SIAM Journal on Matrix Analysis and Applications, 20 0 (2): 0 303--353, June 1998

    1998

  10. [18]

    Momentum Provably Improves Error Feedback ! Advances in Neural Information Processing Systems, 36: 0 76444--76495, December 2023

    Fatkhullin, I., Tyurin, A., and Richtarik, P. Momentum Provably Improves Error Feedback ! Advances in Neural Information Processing Systems, 36: 0 76444--76495, December 2023

    2023

  11. [19]

    Parallelizable Algorithms for Optimization Problems with Orthogonality Constraints

    Gao, B., Liu, X., and Yuan, Y.-x. Parallelizable Algorithms for Optimization Problems with Orthogonality Constraints . SIAM Journal on Scientific Computing, June 2019

    2019

  12. [20]

    Gao, B., Vary, S., Ablin, P., and Absil, P. A. Optimization flows landing on the Stiefel manifold. IFAC-PapersOnLine, 55 0 (30): 0 25--30, January 2022. ISSN 2405-8963

    2022

  13. [21]

    Golub, G. H. and Zha, H. The canonical correlations of matrix pairs and their numerical computation. Springer, 1995

    1995

  14. [22]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

    2016

  15. [23]

    Unbiased compression saves communication in distributed optimization: when and how much? Advances in Neural Information Processing Systems, 36, 2024

    He, Y., Huang, X., and Yuan, K. Unbiased compression saves communication in distributed optimization: when and how much? Advances in Neural Information Processing Systems, 36, 2024

    2024

  16. [24]

    Analysis of a complex of statistical variables into principal components

    Hotelling, H. Analysis of a complex of statistical variables into principal components. Journal of Educational Psychology, 24 0 (6): 0 417--441, 1933

    1933

  17. [25]

    Relations between two sets of variates

    Hotelling, H. Relations between two sets of variates. Biometrika, 28: 0 321--377, 1936

    1936

  18. [26]

    and Deng, K

    Hu, J. and Deng, K. Improving the communication in decentralized manifold optimization through single-step consensus and compression, July 2024. arXiv:2407.08904 [math]

    2024 arXiv

  19. [27]

    Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compression

    Huang, X., Chen, Y., Yin, W., and Yuan, K. Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compression . Advances in Neural Information Processing Systems, 35: 0 18955--18969, December 2022

    2022

  20. [28]

    P., Rebjock, Q., Stich, S., and Jaggi, M

    Karimireddy, S. P., Rebjock, Q., Stich, S., and Jaggi, M. Error Feedback Fixes SignSGD and other Gradient Compression Schemes . In Proceedings of the 36th International Conference on Machine Learning , pp.\ 3252--3261. PMLR, May 2019

    2019

  21. [29]

    Learning multiple layers of features from tiny images.(2009), 2009

    Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images.(2009), 2009

    2009

  22. [30]

    Mnist handwritten digit database

    LeCun, Y., Cortes, C., and Burges, C. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist, 2, 2010

    2010

  23. [31]

    and Martínez-Rubio, D

    Lezcano-Casado, M. and Martínez-Rubio, D. Cheap Orthogonal Constraints in Neural Networks : A Simple Parametrization of the Orthogonal and Unitary Group . In Proceedings of the 36th International Conference on Machine Learning , pp.\ 3794--3803. PMLR, May 2019

    2019

  24. [32]

    A Penalty - Free Infeasible Approach for a Class of Nonsmooth Optimization Problems Over the Stiefel Manifold

    Liu, X., Xiao, N., and Yuan, Y.-x. A Penalty - Free Infeasible Approach for a Class of Nonsmooth Optimization Problems Over the Stiefel Manifold . Journal of Scientific Computing, 99 0 (2): 0 30, March 2024

    2024

  25. [33]

    Accelerated First -order Methods for Geodesically Convex Optimization on Riemannian Manifolds

    Liu, Y., Shang, F., Cheng, J., Cheng, H., and Jiao, L. Accelerated First -order Methods for Geodesically Convex Optimization on Riemannian Manifolds . In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017

    2017

  26. [34]

    Lopes, C. G. and Sayed, A. H. Diffusion Least - Mean Squares Over Adaptive Networks : Formulation and Performance Analysis . IEEE Transactions on Signal Processing, 56 0 (7): 0 3122--3136, July 2008. Conference Name: IEEE Transactions on Signal Processing

    2008

  27. [35]

    McMahan, B., Moore, E., Ramage, D., Hampson, S., and Arcas, B. A. y. Communication- Efficient Learning of Deep Networks from Decentralized Data . In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics , pp.\ 1273--1282. PMLR, April 2017

    2017

  28. [36]

    Achieving Geometric Convergence for Distributed Optimization Over Time - Varying Graphs

    Nedić, A., Olshevsky, A., and Shi, W. Achieving Geometric Convergence for Distributed Optimization Over Time - Varying Graphs . SIAM Journal on Optimization, December 2017

    2017

  29. [37]

    and Yuan, X

    Patarasuk, P. and Yuan, X. Bandwidth optimal all-reduce algorithms for clusters of workstations. Journal of Parallel and Distributed Computing, 69 0 (2): 0 117--124, 2009

    2009

  30. [38]

    A., and Absil, P.-A

    Qi, C., Gallivan, K. A., and Absil, P.-A. Riemannian BFGS Algorithm with Applications . In Diehl, M., Glineur, F., Jarlebring, E., and Michiels, W. (eds.), Recent Advances in Optimization and its Applications in Engineering , pp.\ 183--192, Berlin, Heidelberg, 2010. Springer

    2010

  31. [39]

    Distributed sparsity constrained optimization over the Stiefel manifold

    Qu, W., Chen, H., Xiu, X., and Liu, W. Distributed sparsity constrained optimization over the Stiefel manifold. Neurocomputing, 602: 0 128267, October 2024

    2024

  32. [40]

    EF21 : A New , Simpler , Theoretically Better , and Practically Faster Error Feedback

    Richtarik, P., Sokolov, I., and Fatkhullin, I. EF21 : A New , Simpler , Theoretically Better , and Practically Faster Error Feedback . In Advances in Neural Information Processing Systems , volume 34, pp.\ 4384--4396. Curran Associates, Inc., 2021

    2021

  33. [41]

    Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

    Saxe, A., McClelland, J., and Ganguli, S. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In Proceedings of the International Conference on Learning Represenatations 2014. International Conference on Learning Represenatations 2014, 2014

    2014

  34. [42]

    1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs

    Seide, F., Fu, H., Droppo, J., Li, G., and Yu, D. 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs . In Interspeech 2014, pp.\ 1058--1062. ISCA, September 2014

    2014

  35. [43]

    EXTRA : An Exact First - Order Algorithm for Decentralized Consensus Optimization

    Shi, W., Ling, Q., Wu, G., and Yin, W. EXTRA : An Exact First - Order Algorithm for Decentralized Consensus Optimization . SIAM Journal on Optimization, May 2015

    2015

  36. [44]

    and Feizi, S

    Singla, S. and Feizi, S. Skew orthogonal convolutions. In International Conference on Machine Learning, pp.\ 9756--9766. PMLR, 2021

    2021

  37. [45]

    Stich, S. U. Local sgd converges fast and communicates little. In ICLR 2019-International Conference on Learning Representations, 2019 a

    2019

  38. [46]

    Stich, S. U. Local SGD Converges Fast and Communicates Little . In ICLR 2019 - International Conference on Learning Representations , May 2019 b

    2019

  39. [47]

    Stich, S. U. and Karimireddy, S. P. The Error - Feedback Framework : Better Rates for SGD with Delayed Gradients and Compressed Communication , June 2021. arXiv:1909.05350 [cs]

    2021 arXiv

  40. [48]

    U., Cordonnier, J.-B., and Jaggi, M

    Stich, S. U., Cordonnier, J.-B., and Jaggi, M. Sparsified SGD with Memory . In Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018

    2018

  41. [49]

    Retraction- Free Decentralized Non -convex Optimization with Orthogonal Constraints , December 2024

    Sun, Y., Chen, S., Garcia, A., and Shahrampour, S. Retraction- Free Decentralized Non -convex Optimization with Orthogonal Constraints , December 2024. arXiv:2405.11590 [cs]

    2024 arXiv

  42. [50]

    Tripuraneni, N., Flammarion, N., Bach, F., and Jordan, M. I. Averaging Stochastic Gradient Descent on Riemannian Manifolds . In Proceedings of the 31st Conference On Learning Theory , pp.\ 650--687. PMLR, July 2018

    2018

  43. [51]

    Distributed asynchronous deterministic and stochastic gradient optimization algorithms

    Tsitsiklis, J., Bertsekas, D., and Athans, M. Distributed asynchronous deterministic and stochastic gradient optimization algorithms. IEEE Transactions on Automatic Control, 31 0 (9): 0 803--812, September 1986. Conference Name: IEEE Transactions on Automatic Control

    1986

  44. [52]

    Optimization without Retraction on the Random Generalized Stiefel Manifold

    Vary, S., Ablin, P., Gao, B., and Absil, P.-A. Optimization without Retraction on the Random Generalized Stiefel Manifold . In Proceedings of the 41st International Conference on Machine Learning , pp.\ 49226--49248. PMLR, July 2024

    2024

  45. [53]

    Wang, J., Chen, Y., Chakraborty, R., and Yu, S. X. Orthogonal Convolutional Neural Networks . pp.\ 11505--11515, 2020

    2020

  46. [54]

    and Liu, X

    Wang, L. and Liu, X. Decentralized Optimization Over the Stiefel Manifold by an Approximate Augmented Lagrangian Function . IEEE Transactions on Signal Processing, 70: 0 3029--3041, 2022. Conference Name: IEEE Transactions on Signal Processing

    2022

  47. [55]

    Gradient Sparsification for Communication - Efficient Distributed Optimization

    Wangni, J., Wang, J., Liu, J., and Zhang, T. Gradient Sparsification for Communication - Efficient Distributed Optimization . In Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018

    2018

  48. [56]

    TernGrad : Ternary Gradients to Reduce Communication in Distributed Deep Learning

    Wen, W., Xu, C., Yan, F., Wu, C., Wang, Y., Chen, Y., and Li, H. TernGrad : Ternary Gradients to Reduce Communication in Distributed Deep Learning . In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017

    2017

  49. [57]

    Exact Penalty Function for _ 2,1 Norm Minimization over the Stiefel Manifold

    Xiao, N., Liu, X., and Yuan, Y.-x. Exact Penalty Function for _ 2,1 Norm Minimization over the Stiefel Manifold . SIAM Journal on Optimization, December 2021

    2021

  50. [58]

    A class of smooth exact penalty function methods for optimization problems with orthogonality constraints

    Xiao, N., Liu, X., and Yuan, Y.-x. A class of smooth exact penalty function methods for optimization problems with orthogonality constraints. Optimization Methods and Software, 37 0 (4): 0 1205--1241, July 2022

    2022

  51. [59]

    Constructing orthogonal convolutions in an explicit manner

    Yu, T., Li, J., Cai, Y., and Li, P. Constructing orthogonal convolutions in an explicit manner. In International Conference on Learning Representations, 2022

    2022

  52. [60]

    and Sra, S

    Zhang, H. and Sra, S. First-order Methods for Geodesically Convex Optimization . In Conference on Learning Theory , pp.\ 1617--1638. PMLR, June 2016

    2016

  53. [61]

    Reddi, S., and Sra, S

    Zhang, H., J. Reddi, S., and Sra, S. Riemannian SVRG : Fast Stochastic Optimization on Riemannian Manifolds . In Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016

    2016

  54. [62]

    M.-C., and Johansson, M

    Zhang, J., Hu, J., So, A. M.-C., and Johansson, M. Nonconvex Federated Learning on Compact Smooth Submanifolds With Heterogeneous Data . In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.), Advances in Neural Information Processin...

    2024

  55. [63]

    Distributed Riemannian Stochastic Gradient Tracking Algorithm on the Stiefel Manifold , January 2025

    Zhao, J., Wang, X., and Lei, J. Distributed Riemannian Stochastic Gradient Tracking Algorithm on the Stiefel Manifold , January 2025. arXiv:2405.16900 [math]

    2025 arXiv

  56. [64]

    Faster First - Order Methods for Stochastic Non - Convex Optimization on Riemannian Manifolds

    Zhou, P., Yuan, X.-T., and Feng, J. Faster First - Order Methods for Stochastic Non - Convex Optimization on Riemannian Manifolds . In Proceedings of the Twenty - Second International Conference on Artificial Intelligence and Statistics , pp.\ 138--147. PMLR, April 2019

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.