Tight Sample Complexity for Low-Rank Adaptation: Matching Bounds and Rank Selection

Arunan J

REVIEW 4 major objections 4 minor 34 references

Tight Sample Complexity for Low-Rank Adaptation: Matching Bounds and Rank Selection

T0 review · 4 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proves that rank-r LoRA fine-tuning has minimax excess risk Θ~(rd/n), and that unregularized empirical risk minimization pays a strictly growing penalty when r exceeds the task's intrinsic rank r∗.

desk verdict Serious and mostly classical LoRA sample-complexity paper whose headline lower bound overreaches: Theorem 2 as stated contradicts Theorem 4, and the stress-test note is right that the transfer argument in Proposition 5 is backwards; still, the r=r* core is correct and the over-ranking dichotomy is repairable with standard unrestricted minimax bounds. read the letter →

arxiv 2607.27680 v1 pith:JQQ2KTVM submitted 2026-07-30 cs.LG cs.CL

Arunan J This is my paper · ORCID

classification cs.LGcs.CL MSC 68T0562C2062H12

keywords Low-rankadaptationsamplecomplexityminimaxlowerboundrankselectionlocalRademachernuclearnormregularizationtraceregressionover-parameterization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to settle the sample complexity of low-rank adaptation as a statistical estimation problem and to give a formal rule for choosing the LoRA rank. It claims that, in the regime where the population loss is locally quadratic in the low-rank update, the rank-r constrained empirical risk minimizer needs ~rd/n samples per unit of excess risk, that this rate is unavoidable among all rank-r estimators, and that the ERM's optimal rank is exactly the intrinsic rank r∗ while any r>r∗ inflates estimation error linearly. It also claims that adaptive nuclear-norm-then-truncate estimators avoid that penalty and saturate at ~r∗d/n for every r≥r∗. If true, the practical takeaway is sharp: pick the smallest rank that saturates validation performance, or regularize and let the rank choose itself.

What carries the argument

The load-bearing object is the rank-r LoRA class F_r = {f_0 + BA : rank(BA)≤r, ||BA||_F≤R} equipped with a local quadratic excess-risk assumption: λ−||Δ−Δ∗||_F² ≤ L(f_Δ)−L(f∗) ≤ λ+||Δ−Δ∗||_F² near the target. This sandwich converts a slow √(rd/n) Rademacher bound into a fast rd/n rate through localization, and it also controls the KL geometry used in the lower bound. The matching lower bound comes from a packing of rank-r matrices in trace regression with a Fano-type argument, and the rank-selection dichotomy comes from comparing the ERM's variance leak—it fills all r singular directions with noise—with the adaptive estimator's non-expansive nuclear-norm-then-truncate step.

What would settle it

Run the paper's synthetic trace-regression sweep with an intentionally flat population loss near the target (small λ−, e.g. adding a nearly zero-curvature direction) and check whether the excess-risk-vs-rank curve loses its sharp U-shape or whether the rd/n fit fails while over-ranking still hurts. A complementary check on real models: estimate the smallest local Hessian eigenvalue at the optimum of a fine-tuned LoRA adapter; if it is not bounded away from zero while the large-rank loss inflation is still steep, the variance-leak mechanism is not what drives the observed U-shape.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes a tight Θ~(rd/n) sample-complexity characterization for LoRA in the well-specified locally quadratic regime. The upper bound uses local Rademacher complexity on the rank-r manifold together with a Bernstein condition derived from a two-sided quadratic assumption; the lower bound uses a packing of rank-r matrices in a trace-regression instance and an information-theoretic KL-divergence argument. Combining the two with a bias analysis yields the rank-selection dichotomy: under-ranking incurs the singular-value truncation floor σ_{r+1}(Δ∗)^2, the constrained ERM is optimal exactly at r=r∗ and over-ranking strictly hurts; the minimax rate over all rank-r e

Load-bearing premise

The fast rd/n rate and the linear over-ranking penalty both rest on Assumption 4, the requirement that the population loss be quadratically curved in the low-rank update near the true weights; if that curvature is weak or absent, the rate degrades to √(rd/n) and the sharp rank-selection penalty is no longer guaranteed.

Editorial extensions

If this is right

For unregularized constrained ERM, doubling the rank above r∗ doubles the excess risk, so rank should be as small as possible while keeping approximation error negligible.
The minimax rate over all rank-r estimators is ~r∗d/n for r≥r∗, so the over-parameterization penalty is an artifact of unregularized ERM, not a fundamental limit of the LoRA class.
Reaching excess risk ε with rank r and dimension d needs n=Θ~(rd/ε) samples; dimension and rank enter linearly.
Under-ranking leaves an n-independent floor equal to the squared tail singular values of the target adaptation, so the empirical U-shape has a bias-dominated left side and variance-dominated right side.
A nuclear-norm-regularized estimator (or cross-validated rank selection) removes the over-ranking penalty and is predicted to flatten the large-rank degradation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the variance-leak explanation should transfer to any PEFT method whose effective parameter count is O(rd), e.g. adapters or selective weight updates, although the paper only explicitly proves it for LoRA-style BA factorizations.
Editorial inference: the three real-model sweeps hold weight decay to zero and use no LR schedule; adding standard regularization should weaken the U-shape at large ranks, which is a direct, testable prediction of the Theorem 3 vs Theorem 4 contrast.
Editorial inference: the assumption that λ− is bounded away from zero is the likely failure point when moving from theory to deep feature-learning models, so the sharp rd/n recommendation should be treated as provisional until local curvature is measured on real checkpoints.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should read this paper, but with the lower-bound proof under a microscope. The main ideas are classical and honestly assembled: local Rademacher complexity for the rank-r manifold gives the O~(rd/n) upper bound, and a Fano packing on rank-r matrices gives a matching lower bound when the target really has rank r. That much is solid and not new in its ingredients, though stating it cleanly for LoRA is useful. What is genuinely new is the rank-selection narrative: unregularized ERM pays a linear penalty in r past the intrinsic rank, while nuclear-norm-then-truncate saturates at the r* rate. The framing is attractive and matches the empirical U-shapes people report. But the stress-test note lands. Theorem 2 states a lower bound for rank-r* targets with r* ≤ r, yet the proof packs rank-r matrices, so it only proves the r* = r case. And Proposition 5's transfer argument is indeed the wrong direction: enlarging the output class from F_r* to F_r makes the infimum risk no larger, so a restricted minimax lower bound does not carry upward. The contradiction with Theorem 4 is real as written. The good news is that the adaptive lower bound is repairable: the standard Negahban-Wainwright minimax bound for rank-r* trace regression over all estimators gives Ω(r*d/n), which does transfer to F_r because restricting the estimator class cannot lower the minimax risk. So the qualitative dichotomy likely survives, but the proof needs to be rewritten and Theorem 2 restated. For the ERM half of the dichotomy, the paper's own Proposition 2 only analyzes truncated least squares, not the empirical risk minimizer over F_r. That may be a minor gap in the synthetic setting, but it is another place where the paper claims more than it proves. The empirical section is a mixed bag: 168 runs with code and data is good, but n=500 fine-tuning, AdamW standing in for ERM, post-hoc rank selection with no multiplicity control, and a p=0.016 at a coarse resolution make the verification suggestive rather than confirmatory. The theory's curvature Assumption 4 is flagged as necessary, but never checked on the real configurations. Net: the paper deserves a serious referee, but it needs real revision before I would cite the rank-selection results. I'd send it out, and I'd tell the referee to focus on the lower-bound statements and the ERM-to-truncated-LS gap. Not a reading-group slam dunk, but a good discussion piece about how minimax bounds behave under restricted output classes.

Referee Report

4 major / 4 minor

Summary. The paper studies the sample complexity of low-rank adaptation (LoRA) in a well-specified, locally quadratic regime. It claims an O~(rd/n) excess-risk upper bound for the empirical risk minimizer over rank-r LoRA classes (Theorem 1), a matching Ω(rd/n) minimax lower bound for any estimator whose output lies in F_r when the target rank is at most r (Theorem 2), and a rank-selection dichotomy: constrained ERM is optimal exactly at the intrinsic rank r* and over-ranking strictly hurts, while adaptive nuclear-norm-then-truncate estimators achieve Θ~(r*d/n) independent of the chosen rank (Theorems 3 and 4). The theoretical results are complemented by a synthetic trace-regression experiment and real DistilBERT/RoBERTa fine-tuning sweeps.

Significance. The questions addressed are timely and the intended dichotomy — over-ranking is costly for unregularized ERM but harmless for adaptive estimators — is attractive and of practical interest. The upper-bound proof is a careful, classical local-Rademacher argument with tracked constants, and the experimental package (168 runs, code, permutation tests) is a genuine strength. However, the lower-bound argument contains a serious internal contradiction: Theorem 2 as stated is incompatible with Theorem 4, and the transfer argument in Appendix A.4 is invalid. Because the matching-bound and rank-selection claims rest on this lower bound, the central contribution is not established as written.

major comments (4)

[Theorem 2 (§3.2) vs Theorem 4 (§3.3); Lemma 10 (§6.3)] The stated lower bound contradicts Theorem 4. Theorem 2 claims that for target rank r* ≤ r, any estimator with output in F_r has minimax excess risk ≥ c·rd/n. Theorem 4 claims an F_r-restricted estimator with risk O~(r*d/n) in the same trace-regression setting. These cannot both hold when r* < r. The source is visible in Lemma 10: the packing matrices are Δ = δ/(2√(rd)) Σ_{i=1}^r u_i θ_i^T, which have rank r, not rank r*. Fano therefore lower-bounds the supremum over a family of rank-r targets, not over rank-≤r* targets. Step 5 of the assembly concludes a lower bound for rank-≤r* targets, but no argument transfers the packing to the smaller family; in fact the transfer is impossible because the rank-r* family is much smaller. Theorem 2 is proved only for r* = r, and as stated is false for r > r* in the Gaussian trace-regression model, since the adaptive estimator of Theorem 4 achieves O~
[Proposition 5 (Appendix A.4)] The transfer of the lower bound from F_{r*} to F_r is in the wrong direction. The text says an estimator restricted to rank ≤ r is 'at least as free' as one restricted to rank ≤ r*, so the lower bound 'only gets easier.' Enlarging the output set from F_{r*} to F_r can only decrease the infimum risk: inf_{f∈F_r} ≤ inf_{f∈F_{r*}}. Therefore a minimax lower bound for F_{r*}-valued estimators does not imply one for F_r-valued estimators. Proposition 5 is the only lower-bound support for the r ≥ r* case of Theorem 4, so that lower bound is unsupported as written. A correct argument would invoke the unrestricted rank-r* minimax lower bound together with F_r ⊂ {all estimators}, or prove a new F_r-specific bound; neither appears in the manuscript.
[Theorem 3 (§3.3); Proposition 2 (Appendix A.3)] Even if Theorem 2 were corrected to r = r*, it would not establish the over-ranking penalty for the constrained ERM. Theorem 2 is a lower bound over all estimators whose output lies in F_r; it does not lower-bound the risk of the specific ERM. The paper's lower-bound evidence for 'over-ranking strictly hurts' is Proposition 2, which analyzes the truncated least-squares estimator Δ̂_r = Π_r(n^{-1}Σ y_i X_i). In the Gaussian trace-regression design used in the paper, this truncated estimator is not the ERM over rank-r matrices unless the design is orthogonal; no argument is given that their excess risks coincide (and when n < d^2, the full least-squares solution is not even unique). Thus the ERM half of the rank-selection dichotomy, which is a headline practical claim, lacks a valid lower-bound proof.
[§5.5, star-shapedness] The application of Lemma 7 (Bartlett et al. 2005, Thm 3.3) requires F_r − f* to be star-shaped at the origin. The proof asserts this follows because scaling Δ→tΔ preserves rank and the norm ball. But for a nonzero target Δ*, the interpolated matrix (1−t)Δ* + tΔ need not have rank ≤ r; e.g., r=1, Δ*=u1v1^T, Δ=u2v2^T with orthogonal rank-one terms gives rank 2 for 0<t<1. Consequently the offset class is not generally contained in F_r, and the fast-rate localization step is not justified for arbitrary Lipschitz nonlinearities g, or even for trace regression with a nonzero target. This is a separate load-bearing gap in the proof of Theorem 1.

minor comments (4)

[Figure 2 and Section 4] The x-axis labels in Figure 2 appear as '20 21 22 ...' and should read 2^0, 2^1, ..., 2^7. Finding 3 in Section 4 says 'Cross-seed variance grows sharply at larger.' and is missing the word 'ranks'.
[Lemma 5 and Theorem 1 statement] The log argument is written inconsistently: Λ_n := log(9RL_gρ√r√n) in Lemma 5, but later log(nRL_gρ√r) appears. The statement of Theorem 1 sets K_2 = 2^5, while the proof derives 16 ≤ 2^5; this is correct but should be harmonized.
[Section 4.5] The observed ratio 5.5 is compared to the theoretical 8 with the explanation that n=2000 is not asymptotic; this is plausible but no finite-n analysis is given. The sentence 'the shortfall is explained by...' should be softened or supported.
[Section 4.6] The real experiments use only 500 training examples and three epochs. The U-shape and permutation tests are reported transparently, and the loss-versus-accuracy caveat is acknowledged, but the claim that the experiments 'verify' the theory should be framed as qualitative consistency rather than quantitative confirmation, since Assumption 4 is not checked for these configurations.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central rates are derived from external learning-theoretic results and stated assumptions, not from the conclusions.

full rationale

Auditing each claimed derivation: Theorem 1 is a local-Rademacher argument whose ingredients are the rank-r covering bound (Szarek 1982), Dudley's entropy integral, and the Bernstein condition supplied by Assumption 4; the fast rd/n rate is not inserted as a premise but follows from the critical-radius fixed point. Theorem 2 reduces LoRA to trace regression and uses a Fano-style packing adapted from Negahban-Wainwright (Lemma 10); the lower bound is information-theoretic and does not rely on any parameter fitted to the conclusion. Theorems 3 and 4 combine an external Eckart-Young bias term, an external Marchenko-Pastur variance-leak calculation, and external nuclear-norm oracle inequalities (Negahban et al. 2012; Koltchinskii et al. 2011); the achievability result (Proposition 4) is an application of those external bounds. The empirical theory curves in Figure 1 fit a scale constant K, but the qualitative predictions (linear ERM growth in r, flat adaptive curve for r ≥ r*, ratio r/r*) do not depend on that constant, so this is not a fitted parameter renamed as a prediction in a load-bearing way. The paper's main internal weakness is Proposition 5: the transfer from F_{r*} to F_r has the comparison direction wrong ('at least as free' makes lower bounds harder, not easier), and the lower-bound construction in Theorem 2 packs rank-r matrices even when r* < r. That is a proof-validity gap, not a circular reduction: the desired Θ~(r*d/n) lower bound has independent external support in the trace-regression minimax literature. Assumption 4 is a stated premise with independent verification examples, and there is no load-bearing self-citation (the only self-reference is the use of Theorem 2 inside Proposition 5, a within-paper inference, not an imported authority). Therefore no step was found in which the paper's prediction is equivalent to its input by construction.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claims rest on two domain assumptions (realizability and local quadratic curvature) plus the Gaussian-design trace-regression model for the lower bound; all technical machinery is cited from classical low-rank estimation theory. No invented entities. The empirical verification introduces a fitted scale K and unpinned constants in §4.1, which weaken the 'verification' claims but do not enter the theorems.

free parameters (2)

K (theory-curve scale) = fit to the synthetic excess-risk data (Fig. 1)
The 'theoretical' curves bias(r)+K·rd/n and bias(r)+K·r*·d/n in §4.5 introduce a proportionality constant K matched to the measured excess risk; this is a fit, not a prediction. Theorem-level constants C1 (§4.1) and c2 (§4.2) are likewise unpinned, so the 'numerical predictions' ≤C1·0.21 / ≥c2·0.41 are non-falsifiable as stated.
λ_n (nuclear-norm penalty) = 8σ√(d/n)
Set by the proof to dominate the noise-design deviation (Lemma 15), not fitted to data; listed for completeness since it is a chosen constant of the estimator in Proposition 4.

assumptions (5)

domain assumption Assumption 4: local quadratic excess risk with λ−, λ+ > 0 in a Frobenius neighborhood of Δ*
Load-bearing for the fast rd/n rate (Lemmas 5-6, Theorem 1) and for the variance-leak interpretation (Prop. 2 treats the neighborhood as a smooth manifold). The paper states failure modes (flat valleys, λ−→0) but never verifies it on the real fine-tuning configurations.
domain assumption Assumption 3: realizability — the target f* lies in the rank-r* LoRA class
Used in Theorem 1 and in the bias-floor analysis (Eckart-Young, Lemma 13); the paper notes real tasks satisfy it only approximately, adding an Approx(F_r) term (§9.3).
domain assumption Assumption 5: effective noise scale for KL in the nonlinear lower bound (Appendix B)
Needed for Theorem 7's extension of the lower bound to nonlinear f0; the rate is preserved but constants depend on σ_eff.
domain assumption Gaussian design in trace regression for the lower bound and for Prop. 2's variance analysis
The lower bound (§6.1) and the Marchenko-Pastur variance-leak argument (Prop. 2, App. A.3) are proven only for i.i.d. N(0,1) entries; extension to general sub-Gaussian design is listed as open (§9.5).
standard math External classical results used as black boxes
Szarek Stiefel packing (Lemma 1/2), Gilbert-Varshamov, Fano (Cover-Thomas), Dudley's entropy integral, Bartlett et al. 2005 Theorem 3.3, Marchenko-Pastur/Bai-Yin, Eckart-Young, Negahban-Wainwright packing Lemma 3, and the RSC/oracle-inequality framework of Negahban et al. 2012 / Koltchinskii et al. 2011.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Tight Sample Complexity for Low-Rank Adaptation: Matching Bounds and Rank Selection." pith.science (2026). https://pith.science/paper/JQQ2KTVM

@misc{pith2026260727680,
  author       = {Pith},
  title        = {Pith review of: Tight Sample Complexity for Low-Rank Adaptation: Matching Bounds and Rank Selection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JQQ2KTVM}},
  note         = {Machine review of arXiv:2607.27680}
}

read the original abstract

Low-Rank Adaptation (LoRA) has become the standard mechanism for fine-tuning large pretrained models, yet its statistical properties remain only partially understood. Existing generalization results provide upper bounds of the form O~(sqrt(rd/n)) or O~(rd/n), but a matching lower bound is missing, and the question of how to choose the LoRA rank r has no formal answer. Both gaps are closed here. A local Rademacher argument establishes an upper bound of O~(rd/n) on the excess risk of the empirical risk minimizer over rank-r LoRA, whenever the target adaptation has rank at most r. A matching minimax lower bound of Omega(rd/n) is then proved via a Fano-type packing of the rank-r subspace of R^{d x d}; the bound applies to any estimator whose output lies in the rank-r LoRA class. Combining the two yields a rank-selection dichotomy. For the constrained empirical risk minimizer, the optimal rank equals the intrinsic rank r*, and over-ranking strictly hurts. For adaptive estimators of the nuclear-norm-then-truncate type, over-ranking is harmless and the rate saturates at Theta~(r* d / n) regardless of r. Taken together, the three results characterize the statistical complexity of LoRA fine-tuning within the well-specified locally quadratic regime, and identify the empirically observed over-parameterization penalty as a property of unregularized empirical risk minimization rather than of the LoRA class itself. Predictions of the theory are verified on a synthetic trace-regression benchmark and on real LoRA fine-tuning across three (model, task) configurations covering DistilBERT and RoBERTa on SST-2 and MRPC. All configurations exhibit the predicted U-shape in validation loss, with two showing statistically significant loss inflation at large ranks (paired permutation p = 0.016).

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 2 linked inside Pith

[1]

Annals of Statistics 33(4):1497--1537

Bartlett PL, Bousquet O, Mendelson S (2005) Local Rademacher complexities. Annals of Statistics 33(4):1497--1537

2005
[2]

Machine Learning 79(1--2):151--175

Ben-David S, Blitzer J, Crammer K, et al (2010) A theory of learning from different domains. Machine Learning 79(1--2):151--175

2010
[3]

Transactions on Machine Learning Research

Biderman D, Portes J, Ortiz JJG, et al (2024) LoRA learns less and forgets less. Transactions on Machine Learning Research

2024
[4]

IEEE Transactions on Information Theory 56(5):2053--2080

Cand \`e s EJ, Tao T (2010) The power of convex relaxation: Near-optimal matrix completion. IEEE Transactions on Information Theory 56(5):2053--2080

2010
[5]

Wiley-Interscience

Cover TM, Thomas JA (2006) Elements of Information Theory, 2nd edn. Wiley-Interscience

2006
[6]

Advances in Neural Information Processing Systems

Dettmers T, Pagnoni A, Holtzman A, et al (2023) QLoRA : Efficient finetuning of quantized LLMs . Advances in Neural Information Processing Systems

2023
[7]

Johns Hopkins University Press

Golub GH, Van Loan CF (2013) Matrix Computations, 4th edn. Johns Hopkins University Press

2013
[8]

In: International Conference on Machine Learning (ICML)

Gunasekar S, Lee JD, Soudry D, et al (2018) Characterizing implicit bias in terms of optimization geometry. In: International Conference on Machine Learning (ICML)

2018

Show all 34 references

[9]

arXiv preprint arXiv:240212354

Hayou S, Ghosh N, Yu B (2024) LoRA+ : Efficient low rank adaptation of large models. arXiv preprint arXiv:240212354

2024
[10]

In: International Conference on Machine Learning (ICML)

Houlsby N, Giurgiu A, Jastrzebski S, et al (2019) Parameter-efficient transfer learning for NLP . In: International Conference on Machine Learning (ICML)

2019
[11]

In: International Conference on Learning Representations (ICLR), ://arxiv.org/abs/2106.09685

Hu EJ, Shen Y, Wallis P, et al (2022) LoRA : Low-rank adaptation of large language models. In: International Conference on Learning Representations (ICLR), ://arxiv.org/abs/2106.09685

2022 arXiv
[12]

arXiv preprint arXiv:240211867

Jang U, Lee JD, Ryu EK (2024) LoRA training in the NTK regime has no spurious local minima. arXiv preprint arXiv:240211867

2024
[13]

arXiv preprint arXiv:250614530

Kalajdzievski D (2025) Sharp generalization bounds for foundation models with asymmetric randomized low-rank adapters. arXiv preprint arXiv:250614530

2025
[14]

\'E cole d' \'E t \'e de Probabilit \'e s de Saint-Flour

Koltchinskii V (2011) Oracle inequalities in empirical risk minimization and sparse recovery problems. \'E cole d' \'E t \'e de Probabilit \'e s de Saint-Flour

2011
[15]

Annals of Statistics 39(5):2302--2329

Koltchinskii V, Lounici K, Tsybakov AB (2011) Nuclear-norm penalization and optimal rates for noisy low-rank matrix completion. Annals of Statistics 39(5):2302--2329

2011
[16]

arXiv preprint arXiv:240603136

Koo A, Song Z, Yin R (2024) Computational limits of low-rank adaptation ( LoRA ) fine-tuning for transformer models. arXiv preprint arXiv:240603136

2024
[17]

In: International Conference on Learning Representations (ICLR)

Kopiczko DJ, Blankevoort T, Asano YM (2024) VeRA : Vector-based random matrix adaptation. In: International Conference on Learning Representations (ICLR)

2024
[18]

Empirical Methods in Natural Language Processing (EMNLP)

Lester B, Al-Rfou R, Constant N (2021) The power of scale for parameter-efficient prompt tuning. Empirical Methods in Natural Language Processing (EMNLP)

2021
[19]

In: Annual Meeting of the Association for Computational Linguistics (ACL)

Li XL, Liang P (2021) Prefix-tuning: Optimizing continuous prompts for generation. In: Annual Meeting of the Association for Computational Linguistics (ACL)

2021
[20]

Advances in Neural Information Processing Systems

Liu H, Tam D, Muqeeth M, et al (2022) Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems

2022
[21]

In: International Conference on Machine Learning (ICML)

Liu SY, Wang CY, Yin H, et al (2024) DoRA : Weight-decomposed low-rank adaptation. In: International Conference on Machine Learning (ICML)

2024
[22]

In: International Conference on Learning Representations (ICLR)

Lyu K, Li J (2020) Gradient descent maximizes the margin of homogeneous neural networks. In: International Conference on Learning Representations (ICLR)

2020
[23]

arXiv preprint arXiv:241008305

Malinovsky G, Michieli U, Hammoud HAAK, et al (2024) Randomized asymmetric chain of LoRA : The first meaningful theoretical framework for low-rank adaptation. arXiv preprint arXiv:241008305

2024
[24]

In: Conference on Learning Theory (COLT)

Mansour Y, Mohri M, Rostamizadeh A (2009) Domain adaptation: Learning bounds and algorithms. In: Conference on Learning Theory (COLT)

2009
[25]

In: Journal of Machine Learning Research

Maurer A, Pontil M, Romera-Paredes B (2016) The benefit of multitask representation learning. In: Journal of Machine Learning Research

2016
[26]

Annals of Statistics 39(2):1069--1097

Negahban S, Wainwright MJ (2011) Estimation of (near) low-rank matrices with noise and high-dimensional scaling. Annals of Statistics 39(2):1069--1097

2011
[27]

Statistical Science 27(4):538--557

Negahban SN, Ravikumar P, Wainwright MJ, et al (2012) A unified framework for high-dimensional analysis of m -estimators with decomposable regularizers. Statistical Science 27(4):538--557

2012
[28]

Journal of Machine Learning Research 12:3413--3430

Recht B (2011) A simpler approach to matrix completion. Journal of Machine Learning Research 12:3413--3430

2011
[29]

Cambridge University Press

Shalev-Shwartz S, Ben-David S (2014) Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press

2014
[30]

Journal of Machine Learning Research 19(1):2822--2878

Soudry D, Hoffer E, Nacson MS, et al (2018) The implicit bias of gradient descent on separable data. Journal of Machine Learning Research 19(1):2822--2878

2018
[31]

Proceedings of Research Workshop on Banach Space Theory (Iowa City) pp 169--185

Szarek SJ (1982) Nets of Grassmann manifolds and orthogonal groups. Proceedings of Research Workshop on Banach Space Theory (Iowa City) pp 169--185

1982
[32]

Cambridge University Press

Vershynin R (2018) High-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge University Press

2018
[33]

Cambridge University Press

Wainwright MJ (2019) High-Dimensional Statistics: A Non-Asymptotic Viewpoint. Cambridge University Press

2019
[34]

In: International Conference on Learning Representations (ICLR), ://arxiv.org/abs/2310.17513

Zeng Y, Lee K (2024) The expressive power of low-rank adaptation. In: International Conference on Learning Representations (ICLR), ://arxiv.org/abs/2310.17513

2024 arXiv

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.

[1] [1]

Annals of Statistics 33(4):1497--1537

Bartlett PL, Bousquet O, Mendelson S (2005) Local Rademacher complexities. Annals of Statistics 33(4):1497--1537

2005

[2] [2]

Machine Learning 79(1--2):151--175

Ben-David S, Blitzer J, Crammer K, et al (2010) A theory of learning from different domains. Machine Learning 79(1--2):151--175

2010

[3] [3]

Transactions on Machine Learning Research

Biderman D, Portes J, Ortiz JJG, et al (2024) LoRA learns less and forgets less. Transactions on Machine Learning Research

2024

[4] [4]

IEEE Transactions on Information Theory 56(5):2053--2080

Cand \`e s EJ, Tao T (2010) The power of convex relaxation: Near-optimal matrix completion. IEEE Transactions on Information Theory 56(5):2053--2080

2010

[5] [5]

Wiley-Interscience

Cover TM, Thomas JA (2006) Elements of Information Theory, 2nd edn. Wiley-Interscience

2006

[6] [6]

Advances in Neural Information Processing Systems

Dettmers T, Pagnoni A, Holtzman A, et al (2023) QLoRA : Efficient finetuning of quantized LLMs . Advances in Neural Information Processing Systems

2023

[7] [7]

Johns Hopkins University Press

Golub GH, Van Loan CF (2013) Matrix Computations, 4th edn. Johns Hopkins University Press

2013

[8] [8]

In: International Conference on Machine Learning (ICML)

Gunasekar S, Lee JD, Soudry D, et al (2018) Characterizing implicit bias in terms of optimization geometry. In: International Conference on Machine Learning (ICML)

2018