REVIEW 4 major objections 7 minor 45 references
LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won't Fail)
T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Under natural curvature assumptions, every LoRA stable point is a low-rank global minimum or an obvious high-rank failure.
desk verdict A genuinely new conditional landscape dichotomy for LoRA, but the title overclaims: the theorem only holds inside an unverified D-ball with α>0, and the paper's own estimates break at r=64. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The controlling object is the ratio $2\alpha/\beta$ of the restricted strong convexity constant $\alpha$ to the restricted smoothness constant $\beta$ of the full fine-tuning loss near $X_\star$. At any second-order stationary point, the first- and second-order optimality conditions for the factored loss force the gradient $\nabla \widehat L^{\mathrm{full}}(X)$ to split into a part aligned with the singular vectors of $X$ and an orthogonal residual $S$; restricted smoothness bounds $\|S\|_2$ by $\beta\sigma_r + \lambda$. This yields a small perturbation $Z$ of a nuclear-norm subgradient, and the approximation $X - \kappa Z$ has the same top $r_\star$ singular vectors as $X$ for suitable $\kappa$. A rank-constrained projection inequality then produces the quadratic inequality $(2\kappa\alpha - 1)\|X_\star - X\|_F^2 + \|X_{r_\star} - X\|_F^2 \le 0$, so the sign of $2\kappa\alpha - 1$ decides whether every stationary point collapses to the global minimum or whether large spurious solutions are forced.
What would settle it
On a small matrix-sensing problem with known low-rank optimum and engineered curvature constants in the generic regime ($2\alpha\le\beta$), run LoRA from many initializations and collect every converged second-order stationary point; the dichotomy predicts that every non-global one has rank exactly $r$ and smallest singular value at least $\frac{2\alpha}{\beta}\sigma_{r_\star}(X)$. Finding a single non-global SOSP with lower rank, or an exact curvature computation showing $\alpha\le0$ along the path, would overturn the claim.
Extended reading notes
Core claim
The central result is Theorem 1. Let $\widehat L^{\mathrm{full}}_\lambda(X) = \widehat L^{\mathrm{full}}(X) + \lambda\|X\|_*$ be the nuclear-norm-regularized full fine-tuning objective, and assume it has a global minimizer $X_\star$ of rank $r_\star$. Let $\alpha>0$ and $\beta<\infty$ be the restricted strong convexity and smoothness constants of $\widehat L^{\mathrm{full}}$ about $X_\star$, and let LoRA use rank $r \ge r_\star$. Theorem 1 says that any second-order stationary point $(A,B)$ of the factored LoRA loss with $X = AB^\top$ and $\|X - X_\star\|_F \le D$ is either a global minimum of $\widehat L^{\mathrm{full}}_\lambda$, or, when $2\alpha \le \beta$, a non-global point with $\operatorname{rank}(X) = r$, $\sigma_r(X) \ge \frac{2\alpha}{\beta}\,\sigma_{r_\star}(X)$, and $\|X - X_\star\|_F^2 \ge \|X - \Pi_{\operatorname{rank}\le r_\star}(X)\|_F^2 \,/\, (1 - \frac{2\alpha\sigma_{r_\star}}{\beta\sigma_r})$. Spurious local minima may exist, but they are exactly full-rank and large-magnitude. The same qualitative conclusion is extended to multiple weight matrices and to approximately low-rank global minimizers, and a rank bound for SGD with weight decay supports the claim that training is biased toward the good solutions.
Load-bearing premise
The load-bearing premise is that the full fine-tuning loss curves upward strongly enough near its low-rank global minimizer, meaning the restricted strong convexity constant $\alpha$ is strictly positive inside a ball of radius $D$ that contains the optimization path, while staying smooth there; the paper estimates these constants by Monte Carlo sampling, and its estimate for $\alpha$ becomes negative at LoRA rank 64, so without positive curvature the conclusions do not follow.
Editorial extensions
If this is right
- If the theorem holds, LoRA users can treat the factored loss as essentially benign: the only stable bad endpoints are full-rank, large-magnitude points that are easy to detect by monitoring $\sigma_r$ and $\|X\|_F$.
- Zero-initialization and weight decay are not just regularization conveniences; they are what steer the optimizer into the low-rank, small-magnitude region that contains global minima.
- Using a smaller LoRA rank sharpens the curvature constants, making the special regime more likely and the failure bounds tighter.
- When the full fine-tuning objective has only an approximately low-rank minimizer, the same dichotomy persists with an $\varepsilon$-global-minimizer guarantee instead of exact global optimality.
- The theory transfers to any LoRA-style method that keeps the factored update $AB^\top$ and weight decay, since the proof only uses the factorization and the curvature constants.
Reading between the lines
- Editorial inference: because the failure mode is characterized by rank and norm, a cheap safety monitor for LoRA training could compute the smallest singular value of $X_t$ and its Frobenius norm after each epoch and alert when they exceed the theorem's thresholds.
- Editorial inference: the rank-dependent behavior of $\alpha$ suggests a testable trade-off between expressivity (larger rank) and landscape conditioning (smaller rank), and the paper notes but does not quantify this optimum.
- Editorial inference: if the implicit-bias argument is correct, then changing only the initialization while keeping weight decay fixed should raise the frequency of loud failures; a multi-seed sweep across tasks would quantify the risk.
- Editorial inference: the equivalence between weight decay in factored space and nuclear-norm regularization predicts that the same low-rank-versus-loud-failure dichotomy should appear when training with explicit nuclear-norm penalization rather than weight decay, which is a direct untested consequence.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper analyzes the loss landscape of LoRA fine-tuning, in which the update is parameterized as X = AB^T and ℓ2 weight decay is identified with nuclear-norm regularization on X. Under assumptions that the regularized full fine-tuning loss has a rank-r* global minimizer X* and satisfies (r,D)-restricted strong convexity (constant α > 0) and restricted smoothness (constant β < ∞) about X*, Theorem 1 claims a dichotomy for every second-order stationary point (A,B) of the LoRA loss with ||AB^T - X*||_F ≤ D: in the 'special regime' 2α > β all such SOSPs are global minima, and in the 'generic regime' 2α ≤ β each such SOSP is either a global minimum or a non-global solution of rank exactly r satisfying ||X - X*||_F^2 ≥ ||X - Π_{rank≤r*}(X)||_F^2 / (1 - 2ασ_{r*}(X)/(βσ_r(X))). Theorem 2 extends this to multiple weight matrices, and Theorem 3 relaxes exact low-rankness of X* to δ-approximate low-rankness. Section 3.4 argues that zero initialization and weight decay bias training away from the spurious solutions, supported by Lemma 1 on approximate rank. Experiments on RoBERTa/SST-2 and ViT/CIFAR-100 estimate the RSC/RSM constants by Monte Carlo sampling and show convergence to a global minimizer under zero initialization and to a high-rank, large-norm solution under a large random initialization.
Significance. The conditional dichotomy is a substantive theoretical contribution: it avoids the linearization/NTK assumptions of prior LoRA analyses, handles the full non-convex factored objective with weight decay, and yields a falsifiable signature of bad stable points (exactly rank r, with a tail-energy lower bound on distance from the low-rank optimum). The proof strategy—combining first- and second-order optimality of the factored loss with RSC/RSM and nuclear-norm subgradient calculus—is coherent, and the main derivation in Appendix A.1 is reconstructible. The paper is also transparent about the fragility of its core premise: Table 2 reports a negative estimated α at r = 64, and the authors explicitly say the theory no longer applies there. The theorem is not circular: α, β, D, r, and r* are stated assumptions, and the lower bounds are derived rather than assumed. However, the title, abstract, and Section 3.4 assert more than the theorems establish—the dichotomy holds only inside the D-ball, and no result places the training trajectory there—and at the authors' own estimated constants (β/α ≈ 320.8 at r = 32) the 'large magnitude' content of the dichotomy is nearly vacuous.
major comments (4)
- [Abstract, §3.1, Theorem 1] The advertised dichotomy is stated without the D-ball restriction, whereas Theorem 1's conclusion is restricted to SOSPs with ||X - X*||_F ≤ D. The abstract claims that 'LoRA training converges to a global minimizer with low rank and small magnitude, or a qualitatively distinct solution with high rank and large magnitude,' and the title claims LoRA 'Provably Converges'; however, no result in the paper establishes that a LoRA training trajectory, or its limit point, satisfies ||X_t - X*||_F ≤ D. Section 3.4 is explicitly heuristic, and Lemma 1 concerns approximate rank only. It is therefore consistent with every theorem in the paper that a trajectory leaves the D-ball and converges to a spurious SOSP of low rank and small magnitude, which Theorem 1 does not exclude. The abstract, the title, and the Section 3 preamble should be revised to match the quantifiers of Theorem 1.
- [§2.2, Table 2, Eq. (4) in Appendix A.1] The assumption α > 0 is load-bearing—Eq. (4) of Appendix A.1 uses it to convert restricted strong convexity into the distance lower bound—yet the paper's own estimates show it is fragile, and where it holds the advertised 'large magnitude' conclusion is nearly vacuous. Table 2 reports α = -0.0445 for r = 64, so the assumption is false at that rank (as the authors note), and for r = 32 reports α = 0.0002 with β/α = 320.82. At this ratio, the amplification factor in Theorem 1(ii), namely 1/(1 - 2ασ_{r*}(X)/(βσ_r(X))), is only about 1.006 when the singular-value spectrum is flat (σ_r ≈ σ_{r*}), in which case the bound reduces to the almost trivial inequality ||X - X*||_F^2 ≥ ||X - Π_{rank≤r*}(X)||_F^2; a spurious SOSP with small tail energy would not be certified as 'large magnitude' or as far from X*. Since the paper does not control which singular-value profile occurs, the 'fails loudly' guarantee is not uniform in the empirically estimated regime. In addition, the Monte-Carlo procedure of Section C.2 samples 1000 points within the D-ball and only for the last layer, so it certifies α and β only at sampled points, not over the full ball, and it does not address trajectory containment in the ball.
- [Appendix A.4, Lemma 1] The proof of Lemma 1 contains incorrect rank accounting. The accumulated term U_{t,n} = -μ Σ_{j=1}^n (1-2μλ)^{j-1} ∇L̂_{S_{t-j}}(X_{t-j}) is a sum of n batch gradients each of rank at most b, hence has rank at most nb; the proof itself states rank(U_{t,n}), rank(V_{t,n}) ≤ nb, but then asserts that the two grouped terms in the expansion of X_t are 'both matrices of rank at most b,' and the stated result rank(W) ≤ b log(ε/4)/log(1-μλ) does not follow. The displayed calculation also works with (1-2μλ)^{2n} < ε/2 and log(1-2μλ), while the lemma statement uses log(ε/4) and log(1-μλ), and the constants are never reconciled. A corrected count would give rank O(b|log ε|/|log(1-μλ)|), so the qualitative 'approximately low rank' conclusion may survive, but as printed the lemma is unproven.
- [Appendix A.1, λ = 0 case] The proof of Theorem 1 does not actually cover the λ = 0 case it states. The implication 'Aw = 0 ⇒ Bw = 0' used in the rank-deficient case (σ_r(X) = 0) relies on A^T A = B^T B, which is derived immediately beforehand only under the assumption λ > 0; the second-order perturbation argument that follows needs both A w = 0 and B w = 0. Since Theorem 1 is stated for λ ≥ 0, this case requires an explicit argument (e.g., a limiting argument in λ) or the hypothesis should be restricted to λ > 0. In addition, the closing steps of the Appendix A.1 proof are difficult to check as printed: the admissible κ satisfies κ ≤ σ_{r*}(X)/(βσ_r(X)), but the text says 'we can plug in κ = σ_{r*}/σ_r', and the sentence 'resulting in X = X* if 1 ≥ 2κα' reverses the inequality direction relative to the displayed inequality (2κα - 1)||X* - X||_F^2 + ||X^{r*} - X||_F^2 ≤ 0 from which it is supposed to follow.
minor comments (7)
- [§2.2] The definition of (β,r,D)-restricted smoothness is garbled: the bracket structure is unbalanced, the condition rank(U) = 1 appears twice, and the conditions on U and V are tangled; it should be rewritten.
- [Table 2] The β/α column is inconsistent with the displayed α and β entries (for r = 32, 0.0726/0.0002 = 363, not 320.82); the table should use a consistent level of rounding or recompute the ratios from the displayed values.
- [§3.3, Theorem 3 and Theorem A.1] The notation 'δ = o(ε3)' is a typo for δ = o(ε³), and the proof of Theorem 3 in Appendix A.2 drops the term 2κ⟨X - X^{r'}, Z⟩ in the step 'which again simplifies to' without explaining that it vanishes because Z is orthogonal to the singular spaces of X; as displayed, the simplification is not justified by algebra alone.
- [§3.2] In the multiple-matrix restricted smoothness definition, the sentence 'We denote the tuple β of the largest β(l) such that f is (β,r,D)-restricted strongly convex about X*' should say 'restricted smooth'; the slip makes the definition hard to follow.
- [Appendix A.3, Theorem 2 proof] The closing line of the Theorem 2 proof, 'if σ_r(X(l)) ≥ α(l)/(2β(l))·σ_{r*}(X(l)) for all l then X(l)* = X(l)', does not follow from the preceding inequality; the condition for X(l) = X(l)* is 2α(l)σ_{r*}(X(l))/(β(l)σ_r(X(l))) ≥ 1, i.e., σ_r(X(l)) ≤ (2α(l)/β(l))σ_{r*}(X(l)), which is the condition used in Theorem 2's statement. The proof and the statement should be brought into agreement.
- [Theorem 1 and Theorem 2, boundary case] The boundary of the dichotomy is not handled consistently: Theorem 1(ii) states σ_r ≥ (2α/β)σ_{r*} while Theorem 2(ii) states σ_r > (2α/β)σ_{r*}, and neither proof addresses the equality case, where the argument leaves rank(X) ≤ r* and does not decide whether X is a global minimum.
- [Lemma 1 and §1.2] The statement of Lemma 1 does not treat the case X̃ = 0, where X̃/||X̃|| is undefined, and the informal citations of Lee et al. (2016) and Ge et al. (2015) in Section 1.2 present convergence to SOSPs as unconditional, whereas the cited results are conditional on convergence of the iterates and on strict-saddle structure; the text should hedge both statements.
Circularity Check
No significant circularity: Theorem 1 is a conditional landscape dichotomy proved from stated RSC/RSM assumptions, not from fitted constants or self-citation.
full rationale
The paper's central claim is not circular. Theorem 1 takes alpha, beta, D, lambda, and r as assumptions and derives the special/generic-regime dichotomy from the first- and second-order optimality conditions of the LoRA objective together with restricted strong convexity and restricted smoothness. The constants alpha and beta are not fitted parameters used to produce the claimed lower bounds; the Monte-Carlo estimates in Table 2 are only used to argue that the assumptions are plausible and do not enter the proof. The 'low-rank global minimizer' assumption is stated as a premise, and the theorem explicitly limits its conclusions to SOSPs within the D-ball around X*, so the conditional nature of the result is transparent. The only self-citation with co-author overlap is the remark that Theorem 1 includes the NTK-regime analysis of Jang et al. (2024); this is presented as a special case and is not load-bearing for the main proof. Section 3.4's implicit-bias argument is heuristic and does not feed back into Theorem 1, so it cannot make the derivation circular. The proof of Lemma 1 contains a rank-counting slip (it treats a sum of n batch gradients as rank at most b rather than nb), but that is a correctness concern, not circularity. Overall, the derivation chain is self-contained conditional on its explicitly stated assumptions, and no prediction or conclusion reduces by construction to its inputs.
Assumptions & free parameters
assumptions (5)
- domain assumption The full fine-tuning loss with nuclear norm penalty has a global minimizer X* of rank r*, and the LoRA rank r is at least r*.
- domain assumption The full fine-tuning loss is (alpha, r, D)-restricted strongly convex about X* for alpha > 0 within the studied ball.
- domain assumption The full fine-tuning loss is (beta, r, D)-restricted smooth about X* with beta < inf.
- standard math SGD with random initialization converges to SOSPs almost surely or with high probability.
- domain assumption The loss function is nonnegative and twice differentiable, and the neural network is twice differentiable in the fine-tuned weights.
Cite this review
Pith. "Pith review of LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won't Fail)." pith.science (2026). https://pith.science/paper/ORWOMKGX
@misc{pith2026250209376,
author = {Pith},
title = {Pith review of: LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won't Fail)},
year = {2026},
howpublished = {\url{https://pith.science/paper/ORWOMKGX}},
note = {Machine review of arXiv:2502.09376}
}
read the original abstract
Low-rank adaptation (LoRA) has become a standard approach for fine-tuning large foundation models. However, our theoretical understanding of LoRA remains limited as prior analyses of LoRA's training dynamics either rely on linearization arguments or consider highly simplified setups. In this work, we analyze the LoRA loss landscape without such restrictive assumptions. We define two regimes: a "special regime", which includes idealized setups where linearization arguments hold, and a "generic regime" representing more realistic setups where linearization arguments do not hold. In the generic regime, we show that LoRA training converges to a global minimizer with low rank and small magnitude, or a qualitatively distinct solution with high rank and large magnitude. Finally, we argue that the zero-initialization and weight decay in LoRA training induce an implicit bias toward the low-rank, small-magnitude region of the parameter space -- where global minima lie -- thus shedding light on why LoRA training usually succeeds in finding global minima.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
-
[2]
Intrinsic dimensionality explains the effectiveness of language model fine-tuning
Aghajanyan, A., Gupta, S., and Zettlemoyer, L. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. Association for Computational Linguistics, 2021
work page 2021
-
[3]
B it F it: Simple parameter-efficient fine-tuning for transformer-based masked language-models
Ben Zaken, E., Goldberg, Y., and Ravfogel, S. B it F it: Simple parameter-efficient fine-tuning for transformer-based masked language-models. Association for Computational Linguistics, 2022
work page 2022
-
[4]
Global optimality of local search for low rank matrix recovery
Bhojanapalli, S., Neyshabur, B., and Srebro, N. Global optimality of local search for low rank matrix recovery. Neural Information Processing Systems, 2016
work page 2016
-
[5]
Burer, S. and Monteiro, R. D. C. A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization. Mathematical Programming, 95: 0 329--357, 2003
work page 2003
-
[6]
Cabral, R., De la Torre, F., Costeira, J. P., and Bernardino, A. Unifying nuclear norm and bilinear factorization approaches for low-rank matrix decomposition. International Conference on Computer Vision, 2013
work page 2013
-
[7]
Cand\` e s, E. and Recht, B. Exact matrix completion via convex optimization. Foundations of Computational Mathematics, 55 0 (6): 0 111--119, 2012
work page 2012
-
[8]
Dayi, A. K. and Chen, S. Gradient dynamics for low-rank fine-tuning beyond kernels. arXiv preprint arXiv:2411.15385, 2024
arXiv 2024
Show all 45 references
-
[9]
QL o RA : Efficient finetuning of quantized LLM s
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. QL o RA : Efficient finetuning of quantized LLM s. Neural Information Processing Systems, 2023
2023
-
[10]
An image is worth 16x16 words: Transformers for image recognition at scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Le...
2021
-
[11]
Fazel, M., Hindi, H., and Boyd, S. P. A rank minimization heuristic with application to minimum order system approximation. American Control Conference, 2001
2001
-
[12]
S., Gupte, A., and Poggio, T
Galanti, T., Siegel, Z. S., Gupte, A., and Poggio, T. A. SGD and weight decay secretly minimize the rank of your neural network. NeurIPS 2024 Workshop on Mathematics of Modern Machine Learning, 2024
2024
-
[13]
Escaping from saddle points --- online stochastic gradient for tensor decomposition
Ge, R., Huang, F., Jin, C., and Yuan, Y. Escaping from saddle points --- online stochastic gradient for tensor decomposition. Conference on Learning Theory, 2015
2015
-
[14]
No spurious local minima in nonconvex low rank problems: A unified geometric analysis
Ge, R., Jin, C., and Zheng, Y. No spurious local minima in nonconvex low rank problems: A unified geometric analysis. International Conference on Machine Learning, 2017
2017
-
[15]
Ha, W., Liu, H., and Barber, R. F. An equivalence between critical points for rank constraints versus low-rank factorizations. SIAM Journal on Optimization, 30 0 (4): 0 2927--2955, 2020
2020
-
[16]
LoRA+ : efficient low rank adaptation of large models
Hayou, S., Ghosh, N., and Yu, B. LoRA+ : efficient low rank adaptation of large models. In International Conference on Machine Learning, 2024
2024
-
[17]
J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W
Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lo RA : Low-rank adaptation of large language models. International Conference on Learning Representations, 2022
2022
-
[18]
Low Rank Regularization : A review
Hu, Z., Nie, F., Wang, R., and Li, X. Low Rank Regularization : A review. Neural Networks, 136: 0 218--232, 2021. ISSN 0893-6080
2021
-
[19]
D., and Ryu, E
Jang, U., Lee, J. D., and Ryu, E. K. Lo RA training in the NTK regime has no spurious local minima. International Conference on Machine Learning, 2024
2024
-
[20]
A rank stabilization scaling factor for fine-tuning with LoRA
Kalajdzievski, D. A rank stabilization scaling factor for fine-tuning with LoRA . arXiv preprint arXiv:2312.03732, 2023
2023 arXiv
-
[21]
Kobayashi, S., Akram, Y., and Oswald, J. V. Weight decay induces low-rank attention layers. Neural Information Processing Systems, 2024
2024
-
[22]
Learning multiple layers of features from tiny images
Krizhevsky, A. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009
2009
-
[23]
D., Simchowitz, M., Jordan, M
Lee, J. D., Simchowitz, M., Jordan, M. I., and Recht, B. Gradient descent only converges to minimizers. Conference on Learning Theory, 2016
2016
-
[24]
The power of scale for parameter-efficient prompt tuning
Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. Empirical Methods in Natural Language Processing, 2021
2021
-
[25]
Measuring the intrinsic dimension of objective landscapes
Li, C., Farkhoor, H., Liu, R., and Yosinski, J. Measuring the intrinsic dimension of objective landscapes. International Conference on Learning Representations, 2018
2018
-
[26]
Li, X. L. and Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. Association for Computational Linguistics, 2021
2021
-
[27]
A kernel-based view of language model fine-tuning
Malladi, S., Wettig, A., Yu, D., Chen, D., and Arora, S. A kernel-based view of language model fine-tuning. International Conference on Machine Learning, 2023
2023
-
[28]
Pi SSA : Principal singular values and singular vectors adaptation of large language models
Meng, F., Wang, Z., and Zhang, M. Pi SSA : Principal singular values and singular vectors adaptation of large language models. Neural Information Processing Systems, 2024
2024
-
[29]
Mordukhovich, B. S. and Shao, Y. On nonconvex subdifferential calculus in banach spaces. Journal of Convex Analysis, 2 0 (1--2): 0 211--227, 1995
1995
-
[30]
and Boyd, S
Parikh, N. and Boyd, S. Proximal algorithms. Foundations and Trends in Optimization, 1 0 (3): 0 127--239, 2014
2014
-
[31]
Non-square matrix sensing without spurious local minima via the B urer-- M onteiro approach
Park, D., Kyrillidis, A., Carmanis, C., and Sanghavi, S. Non-square matrix sensing without spurious local minima via the B urer-- M onteiro approach . International Conference on Artificial Intelligence and Statistics, 2017
2017
-
[32]
Recht, B., Fazel, M., and Parrilo, P. A. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Review, 52 0 (3): 0 471--501, 2010
2010
-
[33]
D., Ng, A., and Potts, C
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. Empirical Methods in Natural Language Processing, 2013
2013
-
[34]
and Sato, I
Tomihari, A. and Sato, I. Understanding linear probing then fine-tuning language models from NTK perspective. Neural Information Processing Systems, 2024
2024
-
[35]
GLUE : A multi-task benchmark and analysis platform for natural language understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. GLUE : A multi-task benchmark and analysis platform for natural language understanding. EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 2018
2018
-
[36]
M i L o RA : Harnessing minor singular components for parameter-efficient LLM finetuning
Wang, H., Li, Y., Wang, S., Chen, G., and Chen, Y. M i L o RA : Harnessing minor singular components for parameter-efficient LLM finetuning. Association for Computational Linguistics, 2025
2025
-
[37]
and Jacot, A
Wang, Z. and Jacot, A. Implicit bias of SGD in L_2 -regularized linear DNN s: One-way jumps from high to low rank. International Conference on Learning Representations, 2024
2024
-
[38]
How over-parameterization slows down gradient descent in matrix sensing: The curses of symmetry and initialization
Xiong, N., Ding, L., and Du, S. How over-parameterization slows down gradient descent in matrix sensing: The curses of symmetry and initialization. International Conference on Learning Representations, 2024
2024
-
[39]
and Du, S
Xu, W. and Du, S. Over-parameterization exponentially slows down gradient descent for learning a single neuron. Conference on Learning Theory, 2023
2023
-
[40]
and Lee, K
Zeng, Y. and Lee, K. The expressive power of low-rank adaptation. International Conference on Learning Representations, 2024
2024
-
[41]
Zhang, R. Y. Sharp global guarantees for nonconvex low-rank matrix recovery in the overparameterized regime. arXiv preprint arXiv: 2104.10790, 2021
2021 arXiv
-
[42]
Zhang, R. Y. Improved global guarantees for the nonconvex B urer-- M onteiro factorization via rank overparameterization. Mathematical Programming, pp.\ 1--30, 2024
2024
-
[43]
One-step full gradient suffices for low-rank fine-tuning, provably and efficiently
Zhang, Y., Liu, F., and Chen, Y. One-step full gradient suffices for low-rank fine-tuning, provably and efficiently. arXiv preprint arXiv: 2502.01235, 2025
2025 arXiv
-
[44]
Zhu, Z., Li, Q., Tang, G., and Wakin, M. B. Global optimality in low-rank matrix optimization. IEEE Transactions on Signal Processing, 66 0 (13): 0 3614--3628, 2018
2018
-
[45]
A robustly optimized BERT pre-training approach with post-training
Zhuang, L., Wayne, L., Ya, S., and Jun, Z. A robustly optimized BERT pre-training approach with post-training. pp.\ 1218--1227, 2021
2021
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.