Pith. sign in

REVIEW 1 major objections 4 minor 65 references

A retraction-free optimizer for Stiefel constraints converges with a fixed penalty parameter and makes LoRA fine-tuning faster.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 02:51 UTC pith:D4ZXTXOE

load-bearing objection The explicit-penalty result is genuinely new, but the stochastic convergence proof has a real gap and the evaluated algorithm escapes the theory. the 1 major comments →

arxiv 2607.25299 v1 pith:D4ZXTXOE submitted 2026-07-28 cs.LG cs.AI

Retraction-Free Optimization over the Stiefel Manifold for the LoRA Fine-Tuning

Yuan Zhang , Jiang Hu , Zhijian Lai , Lin Lin , Zaiwen Wen This is my paper
classification cs.LG cs.AI MSC 90C2690C3065K05
keywords Stiefel manifoldretraction-free optimizationpenalty-parameter-freelanding methodrestricted secant inequalityLoRAparameter-efficient fine-tuningconstraint violation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to show that a simple projected-gradient-plus-penalty update can solve optimization problems on the Stiefel manifold without any retraction operator and without tuning the penalty strength. It proves that with the penalty coefficient fixed at 1/3, the iterates both fall onto the manifold and reach a stationary point, at the same O(1/K) rate as retraction-based methods in the deterministic setting. The same mechanism is then applied to LoRA: rewriting the low-rank update as a product of a manifold-constrained basis B and a free coordinate A removes the redundancy of the BA factorization. In experiments, this Manifold-LoRA reaches the same loss in roughly half the epochs and matches or beats vanilla LoRA with half the trainable parameters. If correct, the work removes two practical headaches—retraction cost and penalty tuning—from a widely used fine-tuning method.

Core claim

The quadratic penalty φ(X)=¼||X^T X−I||² satisfies a restricted secant inequality near the Stiefel manifold, so gradient descent on φ with step 1/3 converges linearly to the manifold. Combined with proximal smoothness, this yields the landing update X_{k+1}=X_k − α_k P_{T_{X_k}St}(g_k) − (1/3)∇φ(X_k): the penalty step controls feasibility, the projected loss step drives optimality. The paper proves exact convergence to a stationary point under constant step size deterministically, and O(log K/√K) stationarity plus O(log K/K) constraint violation stochastically, improving on landing methods that converged only to a neighborhood or needed tuned penalties. Manifold-LoRA constrains B to the Stie

What carries the argument

Restricted secant inequality (RSI): a strong-convexity substitute for the nonconvex penalty φ, asserting that for X within distance 1/8 of the manifold, ⟨∇φ(X), X − P_St(X)⟩ ≥ ||X − P_St(X)||². This is what fixes the penalty parameter: gradient descent on φ with step 1/3 contracts the distance to the manifold by a factor 2/3 each step. Proximal smoothness of the Stiefel manifold supplies the Lipschitz behavior of the projection and of the tangent-space projection, which connects feasibility to loss descent. The landing update itself—a projected loss gradient plus a constant-multiple penalty gradient—is the mechanism that lets the iterates remain near the manifold while decreasing f.

Load-bearing premise

The convergence theorems assume the initial point lies within distance 1/8 of the Stiefel manifold and that step sizes keep every iterate inside that neighborhood; if a practical LoRA initialization violates this, the fixed-penalty landing guarantees may not hold.

What would settle it

Run the deterministic update with μ=1/3 on a tiny Stiefel problem starting from an X with singular values 0.90 and 0.95 (distance about 0.112, inside the 1/8 tube) and zero loss gradient. The theory predicts the squared distance to the manifold contracts by factor 2/3 at every step; any iterate that fails to contract or leaves the 1/8-neighborhood refutes Lemma 2 and the deterministic half of Theorem 1.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Deterministic Stiefel-constrained problems can be solved with retraction-free updates at the same O(1/K) iteration rate as retraction-based methods, with no penalty parameter to tune.
  • Stochastic mini-batch versions retain global convergence; the two-scale step-size schedule (decaying loss step, constant penalty step) improves the constraint-violation rate to O(log K/K).
  • LoRA fine-tuning can be reparameterized to remove the BA representation redundancy; a manifold-constrained B converges faster and can match or beat vanilla LoRA at half the rank or parameter count.
  • The fixed penalty strength 1/3 transfers across tasks, so the only hyperparameter needing scheduling is the loss step size.
  • The Oblique-manifold variant, which only normalizes the columns of B, gives similar gains, indicating the benefit does not depend on strict orthogonality.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The RSI-plus-proximal-smoothness engine is probably not specific to the Stiefel manifold; any proximally smooth matrix manifold whose constraint penalty satisfies a similar secant inequality should admit an analogous landing algorithm.
  • The geometric constraint may act as an implicit regularizer; a controlled comparison of Manifold-LoRA with ordinary LoRA at matched final feasibility (e.g., by projecting B post-hoc) would tell whether the gains come from faster optimization or from the constraint itself.
  • The heatmap evidence that B^T B ≈ I at checkpoints suggests a testable prediction: with the fixed penalty step, the iterates should stay close to the manifold throughout training, not just at the end.
  • Since the theory is proven for SGD-style updates while the reported experiments use AdamW, the delayed-preconditioner Adam-style analysis in the appendix can be empirically checked for the same O(log K / √K) rate.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 4 minor

Summary. The paper proposes a retraction-free, penalty-parameter-free landing method for optimization over the Stiefel manifold. The key theoretical claims are: (i) an explicit penalty parameter mu=1/3 is sufficient; (ii) under a tube initialization dist(X_1,St)<=1/8, the iterates of Algorithm 1 with projected stochastic gradients and a constant penalty step mu=1/3 converge in deterministic and stochastic settings with the best-known iteration complexities (Theorem 1); (iii) a two-scale step-size scheme yields O(log K/K) constraint-violation decay. The paper then reformulates LoRA fine-tuning as optimization over a product manifold and Euclidean space (with B constrained to Stiefel or Oblique manifolds), proposes Algorithm 2 ('Manifold-LoRA') with SGD or AdamW updates and per-iteration clipping of the step size, and reports experiments on GLUE, SQuAD, E2E, and LLaMA-scale benchmarks. Appendices provide proofs, including a delayed-preconditioner Adam-style convergence theorem (Theorem B.1).

Significance. If the convergence theory were fully sound, the explicit penalty parameter and two-scale step-size analysis would be a useful advance in retraction-free optimization, and the LoRA reformulation is a reasonable and interesting idea with supporting experimental gains. The paper also makes explicit comparisons to prior landing methods and identifies a self-contained gap in the Adam analysis (Remark B.1), which is commendable. However, the central stochastic convergence proof contains a load-bearing gap that is not fixed within the current manuscript, and the experiments use tuned hyperparameters and a non-delayed AdamW variant that the theory explicitly excludes. The theoretical contribution is therefore not established as stated, and the experimental evaluation does not directly validate the theoretical claims.

major comments (1)
  1. [Section IV.A, Lemmas 1-2] The initialization condition ||X_1 - bar X_1|| <= 1/8 is essential for the RSI and all subsequent lemmas, but the paper does not explain how LoRA initializations (e.g., random A and B, or B fixed at initialization as in some LoRA implementations) guarantee this condition. For the practical algorithm, B_0 is typically initialized randomly and the constraint is not satisfied initially. This is an unaddressed gap between theory and practice. The paper should either provide a practical initialization scheme or state the limitation explicitly.
minor comments (4)
  1. [Notation and Assumption 1] Assumption 1 states gradients are Lipschitz over the convex hull of bar U(1/8), but Lemma 3 uses L and D_f over the manifold; the relation between these constants should be stated more carefully. Also, the notation X_1 vs X_0 is inconsistent between Algorithm 1 and Theorem 1 (Algorithm 1 initializes with X_0, Theorem 1 assumes ||X_1 - bar X_1|| <= 1/8).
  2. [Proof of Lemma 5] The proof of Lemma 5 in the manuscript is compressed; in particular the constant 7 in front of the variance term and the step where the term involving mu D_f is absorbed into the constant 4 hat D_f + 9 hat L^2 + 8L + 3 are not shown. Expanding the algebra would help verifiability.
  3. [Section V] The experimental section reports results on many tasks, but the number of random seeds for the GLUE tables and the LLaMA-scale tables is not stated; Figure 3 mentions five seeds. Also, Tables I-II compare against 'LoRA + Stiefel' from [27]; the corresponding baseline setup should be described precisely to allow fair comparison.
  4. [Abstract and Introduction] The abstract claims 'best-known iteration complexities' — this should be qualified relative to the specific setting (e.g., stochastic nonconvex with bounded variance) and compared with the most relevant prior results beyond [11,12].

Circularity Check

0 steps flagged

No circular reduction found: the theoretical derivation is self-contained; flagged gaps are proof/scope errors, not input-output identity.

full rationale

The paper's central derivation (Algorithm 1 and Theorem 1) is not circular under the rubric. The penalty parameter mu=1/3 is derived analytically from the RSI lemma and singular-value bounds, not fitted to data; the convergence theorem is proved from Assumption 1, Lemma 2's contraction of the penalty step, and Lemma 5's descent estimate, with no quantity defined as the thing it is claimed to predict. The citations to the authors' own work (e.g., [18] Deng & Hu for a standard quadratic-upper-bound lemma, and [7] in related work) are auxiliary and independently checkable; they are not used to forbid alternatives or to import an unverified uniqueness claim. The genuine weaknesses are non-circular gaps: (i) in Appendix A.F, the induction delta_{k+1} <= sqrt(2/3) delta_k + (5/2) alpha_k ||g_k|| is asserted <= 1/8 using alpha_k <= 1/(120 D_hat_f), which silently requires an almost-sure bound on ||g_k|| that Assumption 1 does not provide; (ii) Appendix B Remark B.1 explicitly says 'The non-delayed version that uses the current adaptive preconditioner depending on g_tilde_k is not covered by the conditional-independence argument above,' while Algorithm 2 uses the non-delayed v_k and constant beta1=0.9; (iii) the experiments tune mu per task (Tables III-V) rather than using the theoretical mu=1/3, so the empirical results do not directly instantiate the theory. These are correctness/scope concerns, not cases where an output equation equals its input by construction or a fitted parameter is renamed as a prediction. The alleged Lemma 5 identity is also not a circularity (and for eta in T_Xbar St and symmetric N, <eta, Xbar N>=0 actually holds).

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The theoretical algorithm depends on µ=1/3, initialization, and standard smoothness. The experimental algorithm depends on tuned µ and clipping bounds, which are not free variables in the proof but are necessary to reproduce the reported performance.

free parameters (3)
  • Penalty coefficient µ (empirical) = Varies per task: 0.8–1.2 (Tables III-V)
    Although theory fixes µ=1/3, all experiments use per-task tuned µ values for best performance.
  • Clipping bounds lb, ub for α(C) = Varies per task (Tables III-V)
    Algorithm 2 clips the step size by ||C|| with tuned lower and upper bounds.
  • Learning rate α0 = Reported per dataset (Tables III-V)
    Standard hyperparameter, but not all settings are given for every task.
axioms (3)
  • domain assumption Assumption 1: each component function fi is Lf-smooth over the convex hull of the 1/8-tube, and stochastic gradients are unbiased with bounded variance.
    Standard smoothness and noise assumptions, but not verified for LLM loss functions.
  • domain assumption Initial condition ||X1 - Xbar1|| ≤ 1/8.
    Required for the RSI and all subsequent lemmas; not discussed for LoRA initialization.
  • standard math Proximal smoothness of Stiefel manifold and 2-Lipschitz projection property in USt(1/2).
    Cited from [36,37].

pith-pipeline@v1.3.0-alltime-deepseek · 3591 in / 12219 out tokens · 228395 ms · 2026-08-01T02:51:21.589973+00:00 · methodology

0 comments
read the original abstract

Optimization over the Stiefel manifold plays a significant role in various machine learning tasks. Existing methods either use the retraction operators, requiring costly orthonormalization for large-scale matrices, or employ landing methods that rely on careful step size selection and penalty parameter tuning. To address these challenges, we propose a retraction-free and penalty parameter-free algorithm that directly lands on the manifold. By leveraging the strongly-convex-like property of the quadratic penalty function and the proximal smoothness of the Stiefel manifold, we establish global convergence guarantees with the best-known iteration complexities under both constant and diminishing step sizes. Then, we reformulate the low-rank adaptation (LoRA) fine-tuning problem for large language models as a manifold optimization problem, introducing Manifold-LoRA for geometry-accelerated adaptation. This approach employs the proposed landing technique and a carefully designed step size strategy to accelerate the training process. Numerical experiments on benchmark datasets demonstrate the efficiency and strong downstream performance of the proposed method.

Figures

Figures reproduced from arXiv: 2607.25299 by Jiang Hu, Lin Lin, Yuan Zhang, Zaiwen Wen, Zhijian Lai.

Figure 1
Figure 1. Figure 1: The figures compare the training loss, evaluation exact match, and evaluation F1 metrics against epochs for the SQuADv2.0 dataset. It can be clearly [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The figures illustrate that both sphere constrained and Stiefel constrained manifold-LoRA achieve a faster convergence rate and attain a lower training [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Performance on the validation sets across three datasets. The COLA dataset is evaluated using the matthews correlation metric, QQP is measured by [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The heat map of B⊤B with the Stiefel manifold (the first and second rows) and the Oblique manifold (the third and fourth rows) at the end of training on SQuADv2.0 dataset. TABLE VIII: GPT-2 medium (M) and large (L) models evaluated on the E2E NLG Challenge. * denotes results from previously published works. Model Parameters BLEU NIST MET ROUGE-L CIDEr GPT-2 M (FT)* 354.92M 68.2 8.62 46.2 71.0 2.47 GPT-2 M … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

65 extracted references · 16 linked inside Pith

  1. [1]

    Unitary evolution recurrent neural networks,

    M. Arjovsky, A. Shah, and Y . Bengio, “Unitary evolution recurrent neural networks,” inInternational conference on machine learning. PMLR, 2016, pp. 1120–1128

    2016

  2. [2]

    Riemannian approach to batch normalization,

    M. Cho and J. Lee, “Riemannian approach to batch normalization,” Advances in Neural Information Processing Systems, vol. 30, 2017

    2017

  3. [3]

    Robust optimization for non-convex objectives,

    R. S. Chen, B. Lucier, Y . Singer, and V . Syrgkanis, “Robust optimization for non-convex objectives,”Advances in Neural Information Processing Systems, vol. 30, 2017

    2017

  4. [4]

    Momentum Stiefel Optimizer, with Applications to Suitably-Orthogonal Attention, and Optimal Transport,

    L. Kong, Y . Wang, and M. Tao, “Momentum Stiefel Optimizer, with Applications to Suitably-Orthogonal Attention, and Optimal Transport,” inInternational Conference on Learning Representations, 2023

    2023

  5. [5]

    Absil, R

    P.-A. Absil, R. Mahony, and R. Sepulchre,Optimization algorithms on matrix manifolds. Princeton University Press, 2008

    2008

  6. [6]

    A feasible method for optimization with orthog- onality constraints,

    Z. Wen and W. Yin, “A feasible method for optimization with orthog- onality constraints,”Mathematical Programming, vol. 142, no. 1, pp. 397–434, 2013

    2013

  7. [7]

    A brief introduction to manifold optimization,

    J. Hu, X. Liu, Z.-W. Wen, and Y .-X. Yuan, “A brief introduction to manifold optimization,”Journal of the Operations Research Society of China, vol. 8, pp. 199–248, 2020

    2020

  8. [8]

    Boumal,An introduction to optimization on smooth manifolds

    N. Boumal,An introduction to optimization on smooth manifolds. Cambridge University Press, 2023

    2023

  9. [9]

    A new first-order algorithmic framework for optimization problems with orthogonality constraints,

    B. Gao, X. Liu, X. Chen, and Y .-x. Yuan, “A new first-order algorithmic framework for optimization problems with orthogonality constraints,” SIAM Journal on Optimization, vol. 28, no. 1, pp. 302–332, 2018

    2018

  10. [10]

    An orthogonalization-free parallelizable framework for all-electron calculations in density functional theory,

    B. Gao, G. Hu, Y . Kuang, and X. Liu, “An orthogonalization-free parallelizable framework for all-electron calculations in density functional theory,”SIAM Journal on Scientific Computing, vol. 44, no. 3, pp. B723– B745, 2022

    2022

  11. [11]

    Fast and accurate optimization on the orthogonal manifold without retraction,

    P. Ablin and G. Peyr ´e, “Fast and accurate optimization on the orthogonal manifold without retraction,” inInternational Conference on Artificial Intelligence and Statistics. PMLR, 2022, pp. 5636–5657. 10 TABLE V: Hyperparameter configurations of Manifold-LoRA for GLUE benchmark. Method Hyperparameter MNLI SST-2 CoLA QQP QNLI RTE MRPC STS-B Baseline Epoch...

    arXiv 2022

  12. [12]

    Infeasible deterministic, stochastic, and variance-reduction algorithms for optimization under orthogonality constraints,

    P. Ablin, S. Vary, B. Gao, and P.-A. Absil, “Infeasible deterministic, stochastic, and variance-reduction algorithms for optimization under orthogonality constraints,”Journal of Machine Learning Research, vol. 25, no. 389, pp. 1–38, 2024

    2024

  13. [13]

    Dissolving constraints for Riemannian optimization,

    N. Xiao, X. Liu, and K.-C. Toh, “Dissolving constraints for Riemannian optimization,”Mathematics of Operations Research, vol. 49, no. 1, pp. 366–397, 2024

    2024

  14. [14]

    Distributed subgradient methods for multi- agent optimization,

    A. Nedic and A. Ozdaglar, “Distributed subgradient methods for multi- agent optimization,”IEEE Transactions on Automatic Control, vol. 54, no. 1, p. 48, 2009

    2009

  15. [15]

    EXTRA: An exact first-order algorithm for decentralized consensus optimization,

    W. Shi, Q. Ling, G. Wu, and W. Yin, “EXTRA: An exact first-order algorithm for decentralized consensus optimization,”SIAM Journal on Optimization, vol. 25, no. 2, pp. 944–966, 2015

    2015

  16. [16]

    Harnessing smoothness to accelerate distributed optimization,

    G. Qu and N. Li, “Harnessing smoothness to accelerate distributed optimization,”IEEE Transactions on Control of Network Systems, vol. 5, no. 3, pp. 1245–1260, 2017

    2017

  17. [17]

    Decentralized Riemannian gradient descent on the Stiefel manifold,

    S. Chen, A. Garcia, M. Hong, and S. Shahrampour, “Decentralized Riemannian gradient descent on the Stiefel manifold,” inInternational Conference on Machine Learning. PMLR, 2021, pp. 1594–1605

    2021

  18. [18]

    Decentralized projected Riemannian gradient method for smooth optimization on compact submanifolds,

    K. Deng and J. Hu, “Decentralized projected Riemannian gradient method for smooth optimization on compact submanifolds,”arXiv preprint arXiv:2304.08241, 2023

    Pith/arXiv arXiv 2023

  19. [19]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskeveret al., “Language models are unsupervised multitask learners,”OpenAI blog, vol. 1, no. 8, p. 9, 2019

    2019

  20. [20]

    Is chatgpt a general-purpose natural language processing task solver?

    C. Qin, A. Zhang, Z. Zhang, J. Chen, M. Yasunaga, and D. Yang, “Is chatgpt a general-purpose natural language processing task solver?”arXiv preprint arXiv:2302.06476, 2023. 11 0 1 2 3 4 5 6 7 01234567 layer.2.value_proj 0 1 2 3 4 5 6 7 01234567 layer.2.attention.output 0 1 2 3 4 5 6 7 01234567 layer.2.intermediate.dense 0 1 2 3 4 5 6 7 01234567 layer.2.ou...

    Pith/arXiv arXiv 2023

  21. [21]

    Parameter-efficient transfer learning for NLP,

    N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for NLP,” inInternational conference on machine learning. PMLR, 2019, pp. 2790–2799

    2019

  22. [22]

    Prefix-tuning: Optimizing continuous prompts for generation,

    X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,”arXiv preprint arXiv:2101.00190, 2021

    Pith/arXiv arXiv 2021

  23. [23]

    Bitfit: Simple parameter- efficient fine-tuning for transformer-based masked language-models,

    E. B. Zaken, S. Ravfogel, and Y . Goldberg, “Bitfit: Simple parameter- efficient fine-tuning for transformer-based masked language-models,” arXiv preprint arXiv:2106.10199, 2021

    arXiv 2021

  24. [24]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685, 2021

    Pith/arXiv arXiv 2021

  25. [25]

    Riemannlora: A unified riemannian framework for ambiguity-free lora optimization,

    V . Bogachev, V . Aletov, A. Molozhavenko, D. Bobkov, V . Soboleva, A. Alanov, and M. Rakhuba, “Riemannlora: A unified riemannian framework for ambiguity-free lora optimization,”arXiv e-prints, pp. arXiv–2507, 2025

    2025

  26. [26]

    Lora meets riemannion: Muon optimizer for parametrization- independent low-rank adapters,

    ——, “Lora meets riemannion: Muon optimizer for parametrization- independent low-rank adapters,”arXiv preprint arXiv:2507.12142, 2025

    arXiv 2025

  27. [27]

    Riemannian optimization for LoRA on the stiefel manifold,

    J. Park, M. Kang, S. Lee, H. Lee, S. Kim, and J. Lee, “Riemannian optimization for LoRA on the stiefel manifold,” inFindings of the Association for Computational Linguistics: EMNLP 2025, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V . Peng, Eds. Suzhou, China: Association for Computational Linguistics, Nov. 2025, pp. 20 971–20 985. [Online]. Avail...

    2025

  28. [28]

    Adaptive budget allocation for parameter-efficient fine-tuning,

    Q. Zhang, M. Chen, A. Bukharin, P. He, Y . Cheng, W. Chen, and T. Zhao, “Adaptive budget allocation for parameter-efficient fine-tuning,” inThe Eleventh International Conference on Learning Representations, 2023

    2023

  29. [29]

    Sparse low-rank adaptation of pre-trained language models,

    N. Ding, X. Lv, Q. Wang, Y . Chen, B. Zhou, Z. Liu, and M. Sun, “Sparse low-rank adaptation of pre-trained language models,”arXiv preprint arXiv:2311.11696, 2023

    Pith/arXiv arXiv 2023

  30. [30]

    The lottery ticket hypothesis: Finding sparse, 12 trainable neural networks,

    J. Frankle and M. Carbin, “The lottery ticket hypothesis: Finding sparse, 12 trainable neural networks,”arXiv preprint arXiv:1803.03635, 2018

    Pith/arXiv arXiv 2018

  31. [31]

    Vera: Vector-based random matrix adaptation,

    D. J. Kopiczko, T. Blankevoort, and Y . M. Asano, “Vera: Vector-based random matrix adaptation,”arXiv preprint arXiv:2310.11454, 2023

    Pith/arXiv arXiv 2023

  32. [32]

    RTRMC: A Riemannian trust-region method for low-rank matrix completion,

    N. Boumal and P.-a. Absil, “RTRMC: A Riemannian trust-region method for low-rank matrix completion,”Advances in neural information processing systems, vol. 24, 2011

    2011

  33. [33]

    Subspace evolution and transfer (SET) for low-rank matrix completion,

    W. Dai, O. Milenkovic, and E. Kerman, “Subspace evolution and transfer (SET) for low-rank matrix completion,”IEEE Transactions on Signal Processing, vol. 59, no. 7, pp. 3120–3132, 2011

    2011

  34. [34]

    A geometric approach to low- rank matrix completion,

    W. Dai, E. Kerman, and O. Milenkovic, “A geometric approach to low- rank matrix completion,”IEEE Transactions on Information Theory, vol. 58, no. 1, pp. 237–247, 2012

    2012

  35. [35]

    A framework of constraint preserving update schemes for optimization on Stiefel manifold,

    B. Jiang and Y .-H. Dai, “A framework of constraint preserving update schemes for optimization on Stiefel manifold,”Mathematical Program- ming, vol. 153, no. 2, pp. 535–575, 2015

    2015

  36. [36]

    Proximal smoothness and the lower-C2 property,

    F. H. Clarke, R. J. Stern, and P. R. Wolenski, “Proximal smoothness and the lower-C2 property,”Journal of Convex Analysis, vol. 2, no. 1-2, pp. 117–144, 1995

    1995

  37. [37]

    The gradient projection method with armijo’s step size on manifolds,

    M. Balashov and R. Kamalov, “The gradient projection method with armijo’s step size on manifolds,”Computational Mathematics and Mathematical Physics, vol. 61, pp. 1776–1786, 2021

    2021

  38. [38]

    Nocedal and S

    J. Nocedal and S. J. Wright,Numerical optimization. Springer, 1999

    1999

  39. [39]

    Optimization methods for large-scale machine learning,

    L. Bottou, F. E. Curtis, and J. Nocedal, “Optimization methods for large-scale machine learning,”SIAM review, vol. 60, no. 2, pp. 223–311, 2018

    2018

  40. [40]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101, 2017

    Pith/arXiv arXiv 2017

  41. [41]

    Gradient methods for convex minimization: better rates under weaker conditions,

    H. Zhang and W. Yin, “Gradient methods for convex minimization: better rates under weaker conditions,”arXiv preprint arXiv:1303.4645, 2013

    Pith/arXiv arXiv 2013

  42. [42]

    N. J. Higham,Functions of matrices: theory and computation. SIAM, 2008

    2008

  43. [43]

    On the computation of the matrix k-th root,

    S. Laki ´c, “On the computation of the matrix k-th root,”ZAMM-Journal of Applied Mathematics and Mechanics/Zeitschrift f ¨ur Angewandte Mathematik und Mechanik: Applied Mathematics and Mechanics, vol. 78, no. 3, pp. 167–172, 1998

    1998

  44. [44]

    Global rates of convergence for nonconvex optimization on manifolds,

    N. Boumal, P.-A. Absil, and C. Cartis, “Global rates of convergence for nonconvex optimization on manifolds,”IMA Journal of Numerical Analysis, vol. 39, no. 1, pp. 1–33, 2019

    2019

  45. [45]

    On gradient descent ascent for nonconvex-concave minimax problems,

    T. Lin, C. Jin, and M. Jordan, “On gradient descent ascent for nonconvex-concave minimax problems,” inProceedings of the 37th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, H. D. III and A. Singh, Eds., vol

  46. [46]

    Two-timescale gradient descent ascent algorithms for nonconvex minimax optimization,

    T. Lin, C. Jin, and M. I. Jordan, “Two-timescale gradient descent ascent algorithms for nonconvex minimax optimization,”Journal of Machine Learning Research, vol. 26, no. 11, pp. 1–45, 2025. [Online]. Available: http://jmlr.org/papers/v26/22-0863.html

    2025

  47. [47]

    On the convergence of adam and beyond,

    S. J. Reddi, S. Kale, and S. Kumar, “On the convergence of adam and beyond,” inInternational Conference on Learning Representations, 2018

    2018

  48. [48]

    AdaShift: Decorrelation and convergence of adaptive learning rate methods,

    Z. Zhou, Q. Zhang, G. Lu, H. Wang, W. Zhang, and Y . Yu, “AdaShift: Decorrelation and convergence of adaptive learning rate methods,” in International Conference on Learning Representations, 2019

    2019

  49. [49]

    Pytorch: An imperative style, high-performance deep learning library,

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antigaet al., “Pytorch: An imperative style, high-performance deep learning library,”Advances in neural information processing systems, vol. 32, 2019

    2019

  50. [50]

    Transformers: State-of-the-Art Natural Language Processing,

    T. Wolf, L. Debut, V . Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y . Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush, “Transformers: State-of-the-Art Natural Language Processing,” inProceedings of the 2020 Conference on Empir- ical M...

    2020

  51. [51]

    Opendelta: A Plug-and-play Library for Parameter-efficient Adaptation of Pre-trained Models,

    S. Hu, N. Ding, W. Zhao, X. Lv, Z. Zhang, Z. Liu, and M. Sun, “Opendelta: A Plug-and-play Library for Parameter-efficient Adaptation of Pre-trained Models,”arXiv preprint arXiv:2307.03084, 2023

    Pith/arXiv arXiv 2023

  52. [52]

    Debertav3: Improving deberta using electra- style pre-training with gradient-disentangled embedding sharing,

    P. He, J. Gao, and W. Chen, “Debertav3: Improving deberta using electra- style pre-training with gradient-disentangled embedding sharing,”arXiv preprint arXiv:2111.09543, 2021

    Pith/arXiv arXiv 2021

  53. [53]

    GLUE: A multi-task benchmark and analysis platform for natural language understanding,

    A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “GLUE: A multi-task benchmark and analysis platform for natural language understanding,”arXiv preprint arXiv:1804.07461, 2018

    Pith/arXiv arXiv 2018

  54. [54]

    Squad: 100,000+ questions for machine comprehension of text,

    P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “Squad: 100,000+ questions for machine comprehension of text,”arXiv preprint arXiv:1606.05250, 2016

    Pith/arXiv arXiv 2016

  55. [55]

    Know what you don’t know: Unanswerable questions for SQuAD,

    P. Rajpurkar, R. Jia, and P. Liang, “Know what you don’t know: Unanswerable questions for SQuAD,”arXiv preprint arXiv:1806.03822, 2018

    Pith/arXiv arXiv 2018

  56. [56]

    The E2E dataset: New challenges for end-to-end generation,

    J. Novikova, O. Du ˇsek, and V . Rieser, “The E2E dataset: New challenges for end-to-end generation,”arXiv preprint arXiv:1706.09254, 2017

    Pith/arXiv arXiv 2017

  57. [57]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fanet al., “The llama 3 herd of models,”arXiv e-prints, pp. arXiv–2407, 2024

    2024

  58. [58]

    Training verifiers to solve math word problems,

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakanoet al., “Training verifiers to solve math word problems,”arXiv preprint arXiv:2110.14168, 2021

    Pith/arXiv arXiv 2021

  59. [59]

    Optimization without retraction on the random generalized stiefel manifold,

    S. Vary, P. Ablin, B. Gao, and P.-A. Absil, “Optimization without retraction on the random generalized stiefel manifold,” inInternational Conference on Machine Learning. PMLR, 2024, pp. 49 226–49 248. 13 APPENDIXA PROOFS FOR THERETRACTION-FREECONVERGENCEANALYSIS This appendix provides the detailed arguments supporting the convergence results in Section IV...

    2024

  60. [61]

    From Lemma 1,⟨∇φ(X pen k ), Xpen k − X pen k ⟩ ≥3 2 ∥X pen k − X pen k ∥2

  61. [62]

    From the singular-value bound,∥∇φ(X pen k )∥2 ≤6∥X pen k − X pen k ∥2. Substituting gives ∥X pen k+1 − X pen k+1∥2 ≤ ∥Xpen k − X pen k ∥2 − 2 3 · 3 2 ∥X pen k − X pen k ∥2 + 1 9 ·6∥X pen k − X pen k ∥2 = 1−1 + 6 9 ∥X pen k − X pen k ∥2 = 2 3 ∥X pen k − X pen k ∥2. This completes the proof. C. Proof of Lemma 3 Proof. First inequality (8): Due to the Lipsch...

  62. [63]

    The first term is bounded by the contractive property ofP TX St(d,r) and the Lipschitz continuity of∇f: PTX St(d,r)(∇f(X))− PTX St(d,r)(∇f(Y)) ≤ ∇f(X)− ∇f(Y) ≤L f ∥X−Y∥

  63. [64]

    Recall that PTX St(d,r)(g) :=g−Xsym X ⊤g =g− 1 2 X X ⊤g+g ⊤X . Thus, a direct calculation of second term shows PTX St(d,r)(∇f(Y))− PTY St(d,r)(∇f(Y)) = 1 2 X X ⊤∇f(Y) +∇f(Y) ⊤X −Y Y ⊤∇f(Y) +∇f(Y) ⊤Y ≤ 1 2 ∥X((X−Y) ⊤∇f(Y) +∇f(Y) ⊤(X−Y))∥+ 1 2 ∥(X−Y)(Y ⊤∇f(Y) +∇f(Y) ⊤Y)∥ ≤ 1 2 (2 ˆDf + 3 ˆDf )∥X−Y∥ = 5 2 ˆDf ∥X−Y∥, where ˆDf := maxY∈ ¯USt(d,r)( 1 8 ) ∥∇f(Y)...

  64. [65]

    KX k=1 ∥Xk − ¯Xk∥2 # ≤60E

    Putting both pieces together, ∥gradf(X)− ˆ∇f(Y)∥ ≤ Lf + 5 2 ˆDf ∥X−Y∥. Setting ˆL=L f + 5 2 ˆDf completes the proof. D. Proof of Lemma 4 Proof.It follows from ¯Xk+1 =argmin X∈St(d,r) ∥X−X k+1∥2 that ∥Xk+1 − ¯Xk+1∥ ≤ ∥Xk+1 − ¯Xk∥ ≤ ∥Xk −α kPTXk St(d,r)(gk)− 1 3 ∇φ(Xk)− ¯Xk∥(by update (3),µ= 1 3) ≤ ∥Xk − 1 3 ∇φ(Xk)− ¯Xk∥+ 5 2 αk∥gk∥, where we used the fact ...

  65. [119]

    6083–6093

    PMLR, 13–18 Jul 2020, pp. 6083–6093. [Online]. Available: https://proceedings.mlr.press/v119/lin20a.html

    2020