REVIEW 6 minor 19 references
On the Convergence of Stochastic Low-Rank Adaptation
T0 review · 0 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read The paper establishes that simultaneous two-factor LoRA optimization converges with polynomial oracle complexity—$O(\epsilon^{-4})$ for deterministic LoRA-GD, $O(\epsilon^{-8})$ for stochastic LoRA-NSGDM, and $O(\epsilon^{-6})$ for…
desk verdict Solid theory paper that closes the LoRA convergence gap with polynomial rates; the proofs hold up, and the main weaknesses are experimental and presentational. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing device is a modified descent inequality (Lemma A.1), which bounds the non-smooth factored objective by a fourth-order polynomial in the update direction: $J(V+U)\le J(V)+\langle\nabla J(V),U\rangle+\sqrt{2}\rho\|U\|^2\|V\|^2+\sqrt{2}\rho\|U\|^3\|V\|+(\sqrt{2}\rho/4)\|U\|^4+\|\nabla F(BA)\|\,\|U\|^2$. Every descent step in Theorems 3.1, 4.5, and 5.2 is built on this inequality, which replaces the unavailable global Lipschitz-gradient condition for the factored objective. Around it, the deterministic proof uses a trajectory bound $\|V_t\|^2=O(\sqrt{t})$ obtained by summing before Cauchy–Schwarz, while the stochastic proofs add two algorithmic devices: normalizing each update to a prescribed step length (LoRA-NSGDM) and subtracting a same-sample correction between consecutive iterates (LoRA-STORM) to control the cubic and quartic noise terms that the descent inequality introduces.
What would settle it
Check Lemma A.1 directly: for $F(X)=\|X\|_F^2$ ($\rho=1$) and random $V,U$ of varying norms, compute the left and right sides of the displayed inequality; a single violation shows the descent model on which all three theorems rest is false. Alternatively, run Algorithm 1 on a smooth nonconvex $F$ and plot $\min_{t<T}\|\nabla J(V_t)\|_F$ against $T$; decay slower than $O(T^{-1/4})$ would contradict Theorem 3.1.
Extended reading notes
Core claim
The central claim is that simultaneous two-factor LoRA updates admit polynomial first-order convergence rates. Writing the two factors as a single variable $V=[B;\,A^\top]$ and measuring stationarity by $\|\nabla J(V)\|_F\le\epsilon$, the paper proves that the standard LoRA-GD algorithm reaches an $\epsilon$-stationary point in $O(\epsilon^{-4})$ full-gradient evaluations. The improvement comes from a sharper trajectory estimate: summing the factor-norm recursion before applying Cauchy–Schwarz yields $\|V_t\|_F^2=O(\sqrt{t})$ instead of $O(t)$, so the normalized stepsize $\eta_t$ decays like $t^{-1/2}$ and the cumulative stepsize grows like $\sqrt{T}$. For stochastic oracles, the paper first shows that plain LoRA-SGD can fail—with an explicit one-dimensional example where the expected objective and expected gradient norm are infinite—and then proves that LoRA-NSGDM, whose update is normalized to a fixed length and filtered by momentum, finds an $\epsilon$-stationary point with $O(\epsilon^{-8})$ oracle calls under unbiasedness and finite variance. Under the additional mean-square smoothness condition, LoRA-STORM reuses each fresh sample at two consecutive iterates to correct the gradient estimator and improves the bound to $O(\epsilon^{-6})$.
Load-bearing premise
The load-bearing premise is that the modified descent inequality of Lemma A.1—quoted from prior work rather than rederived here—is correct, because every descent step in Theorems 3.1, 4.5, and 5.2 is built on that fourth-order upper bound on $J(V+U)$; if it fails or has different constants, the stated complexity bounds do not follow, and the guarantees are only for the weaker factor-space stationarity notion.
Editorial extensions
If this is right
- Deterministic LoRA-GD moves from exponential to polynomial oracle complexity: $O(\epsilon^{-4})$ full-gradient evaluations suffice for an $\epsilon$-stationary point in factor space.
- Stochastic LoRA can be made convergent under only unbiasedness and finite variance, via LoRA-NSGDM with $O(\epsilon^{-8})$ oracle complexity; the paper also proves that plain LoRA-SGD can diverge in expectation under the same assumptions.
- Under mean-square smoothness, LoRA-STORM improves the stochastic complexity to $O(\epsilon^{-6})$, bringing LoRA in line with typical variance-reduced nonconvex rates despite the bilinear factorization.
- The paper's experiments on logistic regression, ResNet-18, and TinyLlama indicate that LoRA-NSGDM converges faster and more steadily than adaptive-stepsize LoRA-GD baselines in practice.
Reading between the lines
- The same descent-inequality technique should transfer to other bilinear parameterizations, such as matrix factorization or dictionary learning, whenever stationarity is measured in the factored parameter space; the results are not tied to LoRA's particular architecture choices.
- Because the stationarity criterion $\|\nabla J(V)\le\epsilon$ lives in factor space, it is weaker than stationarity in the original weight space; practitioners who stop on weight-space gradients will need a separate translation of these rates.
- The constants in the $O(\epsilon^{-8})$ and $O(\epsilon^{-6})$ bounds scale with quantities such as $\|V_0\|$, $\|\nabla F(0)\|$, $\sigma$, and $\rho$, so a practical next step is to derive adaptive schedules for the momentum and normalization parameters instead of the fixed power-law choices $\alpha=T^{-1/2}$, $\gamma=T^{-7/8}$, $a=T^{-2/3}$, $\eta=T^{-5/6}$.
- A natural testable extension is a single-call variance-reduced LoRA method: LoRA-STORM uses $2T-1$ oracle evaluations, and the paper itself notes this makes it less attractive on large models; a single-call variant at a comparable rate would settle the trade-off.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper analyzes the convergence of simultaneous two-factor LoRA optimization. It proves that deterministic LoRA-GD finds a point with ||∇J(B,A)||_F ≤ ε using O(ε^{-4}) full-gradient evaluations, improving on the exponential exp{O(ε^{-2})} bound of Mu and Klabjan. For the stochastic setting, it shows that plain LoRA-SGD can fail in expectation under finite variance alone, and it proposes LoRA-NSGDM, with O(ε^{-8}) stochastic oracle complexity under finite variance, and LoRA-STORM, with O(ε^{-6}) under mean-square smoothness. The rates are for the factor-space stationarity measure of Definition 2.1. The paper also reports experiments on logistic regression, ResNet-18, and TinyLlama fine-tuning.
Significance. Subject to the explicit factor-space stationarity criterion, the results are technically sound and constitute a substantial advance: they answer the open problem left by Mu and Klabjan and provide the first polynomial stochastic guarantees for simultaneous LoRA factor updates under standard finite-variance assumptions. I checked the main chains of proof, including the summed trajectory bound in Theorem 3.1, the momentum error recursion in Lemma 4.3, and the STORM tracking inequality in Lemma E.1, and found them internally consistent; in particular, the constant in Lemma E.1 (2q^2/[a(2-a)] ≤ 1/a) is valid. The normalization and STORM correction are natural ways to bypass the higher-moment obstruction identified in prior work, and the theorems state explicit dependencies on the problem parameters. The main caveat, which the paper acknowledges at Definition 2.1, is that ||∇J(V)|| ≤ ε is weaker than smallness of the original weight-space gradient ∇F(BA); this should be stated more prominently.
minor comments (6)
- [Definition 2.1] The paper's complexity statements all use the factor-space criterion ||∇J(V)|| ≤ ε, which is explicit but is not equivalent to ||∇F(BA)|| ≤ ε; a point with A or B aligned with a nullspace of ∇F(BA) can satisfy the former without the latter, so the introduction or Section 2 should carry a remark clarifying the practical scope of the guarantees.
- [Appendix A] Lemma A.1 is the load-bearing descent inequality used by Theorems 3.1, 4.5, and 5.2, but it is quoted from Mu and Klabjan without proof; please include a short derivation or proof sketch in the appendix so the paper is self-contained.
- [Algorithms 2 and 3] Algorithms 2 and 3 divide by s_t, but no update is specified when s_t = 0; the proofs handle that case as a zero step, so the pseudocode should state this convention explicitly.
- [Algorithm 3] The claimed oracle count 1 + 2(T−1) assumes the initial evaluation H_0 is reused at t = 0, whereas line 6 as written evaluates the oracle twice at V_0 = V_{−1}; please clarify the implementation convention in the text or pseudocode.
- [Equation (23)] The display for C in Eq. (23) is garbled, with the term after 4Δ likely intended as 4√Δ · (2ρ)^{1/4} or similar; please restate it in terms of C_0 and C_1 to avoid ambiguity.
- [Section 6] The experiments report training loss rather than the factor-gradient stationarity measure used in the theorems, so they do not directly verify the predicted convergence rates; a plot of ||∇J(V_t)|| on the small logistic-regression problem would strengthen the connection between theory and practice.
Circularity Check
No circularity found: the complexity theorems are derived forward from stated assumptions and externally quoted lemmas, with no fitted constants or self-citation chain.
full rationale
The paper's derivation chain is forward and non-circular. Theorem 3.1 follows from the weighted descent budget obtained by summing Lemma A.2 and a trajectory bound obtained by summing the factor-norm recursion before applying Cauchy-Schwarz; the stepsize lower bound eta_t = Omega((t+1)^-1/2) then yields min g_t^2 = O(T^-1/2). Lemma A.1 and Lemma A.2 are quoted from Mu and Klabjan (2026), but these are external, parameter-free results whose stated assumption (rho-smoothness of F) does not include the target complexity, so citing them is legitimate support rather than circularity. The stochastic analysis is also forward: Lemma 2.2 derives unbiasedness and variance scaling from Assumption 3; Lemma 4.2 bounds the trajectory by construction of the normalized step; Lemma 4.3 bounds the momentum error by unrolling the recursion; Lemma 4.4 gives the one-step descent via local smoothness; Theorem 4.5 sums these and balances alpha = T^-1/2 and gamma = T^-7/8 by exponent arithmetic, not by fitting. The STORM analysis likewise derives the tracking bound from Assumption 4, and the parameter choice p = 2/3, b = 5/6 is the maximin solution of the five decay exponents in equation (33). No theorem's conclusion is used as its own assumption; no fitted constants enter the rates; the experimental hyperparameters are tuned on validation but do not appear in the theorems. The factor-space stationarity measure of Definition 2.1 is explicitly stated and weaker than original-weight-space stationarity, but this is a deliberate scoping choice shared with the prior work and does not make any claimed result equivalent to its input by construction.
Assumptions & free parameters
free parameters (2)
- LoRA-NSGDM step-size geometry (α, γ) =
α = T^{-1/2}, γ = T^{-7/8}
- LoRA-STORM averaging/step-size (a, η) =
a = T^{-2/3}, η = T^{-5/6}
assumptions (6)
- domain assumption F has ρ-Lipschitz gradient (Assumption 1)
- domain assumption F is lower bounded by F⋆ (Assumption 2)
- domain assumption Stochastic oracle is unbiased with finite variance σ² (Assumption 3)
- domain assumption Mean-square smoothness of the stochastic oracle (Assumption 4)
- standard math Mu-Klabjan modified descent lemma with quartic term (Lemma A.1)
- standard math Factor-space gradient formula (Eq. 1)
Cite this review
Pith. "Pith review of On the Convergence of Stochastic Low-Rank Adaptation." pith.science (2026). https://pith.science/paper/DU5WZNXV
@misc{pith2026260721975,
author = {Pith},
title = {Pith review of: On the Convergence of Stochastic Low-Rank Adaptation},
year = {2026},
howpublished = {\url{https://pith.science/paper/DU5WZNXV}},
note = {Machine review of arXiv:2607.21975}
}
abstract
Low-rank adaptation (LoRA) optimizes $J(B,A)=\mathcal L(W_\mathrm{base}+sBA)$ over two adapters $B \in \mathbb{R}^{m \times r}$ and $A \in \mathbb{R}^{r \times n}$ that form a low-rank update to a frozen pretrained weight matrix $W_\mathrm{base} \in \mathbb{R}^{m \times n}$. The prior analysis shows LoRA-GD takes $\exp\{\mathcal{O}(\epsilon^{-2})\}$ oracle calls to find an $\epsilon$-stationary point such that $\|\nabla J(B,A)\|\leq \epsilon$ in the deterministic setting. We sharpen the analysis and show that $\mathcal{O}(\epsilon^{-4})$ full-gradient evaluations suffice for the same first-order criterion. We further study stochastic LoRA under unbiased gradient estimates and finite variance. We propose LoRA-NSGDM, which finds an $\epsilon$-stationary point with $\mathcal{O}(\epsilon^{-8})$ stochastic oracle complexity. Under the additional mean-square smoothness condition, we use variance reduction strategy and propose LoRA-STORM, which improves the stochastic oracle complexity to $\mathcal{O}(\epsilon^{-6})$.
Figures
Reference graph
Works this paper leans on
-
[6]
Hyperparameters are selected on validation data; the test splits are not used. Unless explicitly noted, the tasks, architectures, and LoRA configurations follow Mu and Klabjan (2026). We fix each task and model configuration before tuning the optimizers. F.1 Task and model configurations Table 2 summarizes the principal hyperparameters for the four report...
work page 2026
-
[8]
URLhttps://proceedings.mlr.press/v267/kim25n.html. Grigory Malinovsky, Umberto Michieli, Hasan Abed Al Kader Hammoud, Taha Ceritli, Hayder Elesedy, Mete Ozay, and Peter Richtárik. Randomized asymmetric chain of LoRA: The first meaningful theoretical framework for low-rank adaptation.arXiv preprint arXiv:2410.08305,
-
[9]
URLhttps://arxiv.org/abs/2410.08305
doi: 10.48550/arXiv.2410.08305. URLhttps://arxiv.org/abs/2410.08305. Fanxu Meng, Zhaohui Wang, and Muhan Zhang. PiSSA: Principal singular values and singular vectors adaptation of large language models. InAdvances in Neural Information Processing Systems, volume 37, pages 121038–121072,
-
[10]
doi: 10.52202/079017-3846. URL https: //proceedings.neurips.cc/paper_files/paper/2024/hash/db36f4d603cc9e3a2a5e10b93 e6428f2-Abstract-Conference.html. Siqiao Mu and Diego Klabjan. On the convergence rate of LoRA gradient descent. InInternational Conference on Machine Learning,
-
[12]
Igor Sokolov, Abdurakhmon Sadiev, Yury Demidovich, Fawaz S
URLhttps://papers.nips.cc/paper_files/paper/2025/hash/ff541950d 1e885af90f523571564a401-Abstract-Conference.html. Igor Sokolov, Abdurakhmon Sadiev, Yury Demidovich, Fawaz S. Al-Qahtani, and Peter Richtárik. Bernoulli-LoRA: A theoretical framework for randomized low-rank adaptation.arXiv preprint arXiv:2508.03820,
arXiv 2025
-
[13]
URLhttps://arxiv.org/abs/2508 .03820
doi: 10.48550/arXiv.2508.03820. URLhttps://arxiv.org/abs/2508 .03820. Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving LoRA in privacy-preserving federated learning. InInternational Conference on Learning Representations,
-
[14]
doi: 10.52202/079017-1741. URL https://proceedings.neurips.cc/paper_files/paper/2024/ha sh/62c4718cc334f6a0a62fb81c4a2095a1-Abstract-Conference.html. 13 Ziqing Xu, Hancheng Min, Lachlan Ewen MacDonald, Jinqi Luo, Salma Tarmoun, Enrique Mallada, and Rene Vidal. Understanding the learning dynamics of LoRA: A gradient flow perspective on low-rank adaptation ...
-
[15]
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu
URL https://proceedings.mlr.press/v235/zhang24ax.html. Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. TinyLlama: An open-source small language model.arXiv preprint arXiv:2401.02385,
Show all 19 references
-
[16]
URL https://arxiv.org/abs/2401.02385
doi: 10.48550/arXiv.2401.02385. URL https://arxiv.org/abs/2401.02385. 14 A Auxiliary Lemmas Lemma A.1(Mu and Klabjan, 2026, Lemma 3.3).Under Assumption 1, for everyV = [B⊤, A]⊤∈ R(m+n)×r andU∈R (m+n)×r, withH=∇F(BA), J(V+U)≤J(V) +⟨∇J(V),U⟩ F + √ 2ρ∥U∥ 2 F∥V∥ 2 F + √ 2ρ∥U∥ 3 F∥...
-
[18]
Rank-20 adapters are applied to everyconv1 and conv2 in the residual blocks, and the convolutional downsample projections
(0.5,0.1)– – – CIFAR-10 classification with convolutional LoRA.We use theuoft-cs/cifar10 Hugging Face mirror and initialize a torchvision ResNet-18 from scratch following Mu and Klabjan (2026). Rank-20 adapters are applied to everyconv1 and conv2 in the residual blocks, and th...
2026
-
[19]
Terminal validation loss is used as the tie breaker
Hyperparameters are selected on validation data, and are ranked by training-loss AUC unless stated otherwise. Terminal validation loss is used as the tie breaker. LoRA-NSGDMcalibration.For every task we use the same coarse grid α∈{0.05,0.1,0.2,0.5,1.0}andγ∈{0.01,0.02,0.05,0.1,...
2026
-
[2009]
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer
doi: 10.1109/CVPR.2009.5206848. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning ofquantizedLLMs. InAdvances in Neural Information Processing Systems, volume36, pages10088– 10115,
2009
-
[2016]
Edward J
doi: 10.1109/CVPR.2016.90. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. InInternational Conference on Learning Representations,
2016 doi
-
[2019]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei
URL https://papers.nips.cc/paper_files/paper/2019/hash/b8002139cdde66b87638f7f 91d169d96-Abstract.html. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. InIEEE Conference on Computer Vision and Pattern ...
2019
-
[2021]
URLhttps://aclanthology.org/2021
doi: 10.18653/v1/2021.acl-long.568. URLhttps://aclanthology.org/2021. acl-long.568/. Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney, and John P. ...
2021 doi
-
[2023]
URLhttps://proceedings.neurips.cc/paper_fil es/paper/2023/hash/1feb87871436031bdc0f2beaa62a049b-Abstract-Conference.html
doi: 10.52202/075280-0441. URLhttps://proceedings.neurips.cc/paper_fil es/paper/2023/hash/1feb87871436031bdc0f2beaa62a049b-Abstract-Conference.html. Shu Ding, Yang Peng, Hangan Zhou, Xinyu Lu, Shangwei Chen, Junhua Huang, Mingxuan Yuan, and Wei Wang. Towards understanding the ...
2023 doi
-
[2024]
Zhanhong Jiang, Nastaran Saadati, Aditya Balu, Minh Pham, Joshua R
URL https://proceedings.mlr.press/v235/jang24d.html. Zhanhong Jiang, Nastaran Saadati, Aditya Balu, Minh Pham, Joshua R. Waite, Nasla Saleem, Chinmay Hegde, and Soumik Sarkar. A unified convergence theory for large language model efficient fine-tuning. InOPT 2024: Optimization...
2024
-
[2025]
URLhttps://aclanthology.org/2025.realm-1.24/
doi: 10.18653/v1/2025.realm-1.24. URLhttps://aclanthology.org/2025.realm-1.24/. Soufiane Hayou, Nikhil Ghosh, and Bin Yu. LoRA+: Efficient low rank adaptation of large models. InProceedings of the 41st International Conference on Machine Learning, volume 235 ofProceedings of M...
2025 doi
-
[2026]
URL https: //arxiv.org/abs/2512.18248
doi: 10.48550/arXiv.2512.18248. URL https: //arxiv.org/abs/2512.18248. Reece Shuttleworth, Jacob Andreas, Antonio Torralba, and Pratyusha Sharma. LoRA vs full fine-tuning: An illusion of equivalence. InAdvances in Neural Information Processing Systems, volume 38,
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.