REVIEW 5 major objections 6 minor 23 references
SingLoRA: Low Rank Adaptation Using a Single Matrix
T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read SingLoRA replaces LoRA's two trainable matrices with one, removing the scale conflicts that destabilize LoRA fine-tuning, roughly halving parameters, and matching or beating LoRA, LoRA+, and DoRA on language and image benchmarks.
desk verdict A clean PEFT idea (AA^T) with promising GLUE results, but the stability theory is mathematically shaky and the PSD expressivity limitation is still open. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the symmetric adapter $W_0 + \frac{\alpha}{r}u(t)AA^\top$, where $A \in \mathbb{R}^{n \times r}$ is the single trainable matrix, $\alpha$ is a scaling factor, and $u(t) = \min(t/T, 1)$ is a ramp that starts at zero so the fine-tuned model coincides with the frozen pretrained model at initialization even though $A$ is Kaiming-initialized rather than zero. Multiplying $A$ by its own transpose leaves exactly one matrix scale in the optimization and brings a square layer's parameter count from $2nr$ (LoRA's $B$ and $A$) down to $nr$. The theoretical argument runs on two mechanisms: the infinite-width toy model with $\gamma$-notation, which fixes the required learning-rate scaling at $\eta = \Theta(n^{-1/2})$ for stable feature learning; and the transformation-invariance criterion from the LoRA-stability literature, whose sufficient conditions SingLoRA satisfies via the polar decomposition $A_1 = A_2Q$ whenever $A_1A_1^\top = A_2A_2^\top$. For rectangular layers, the update $W_0 + A_*A^\top$ uses the first $d_{in}$ rows of $A$, and the paper proves the same invariance properties carry over.
What would settle it
Adapt only the value and MLP projections — layers where the paper's non-commutativity argument for attention does not apply — and compare SingLoRA against LoRA at matched parameter budgets on a standard suite such as GLUE; if accuracy falls on any task, the benignness of the symmetric constraint is refuted. Alternatively, initialize the same SingLoRA adapter from two orthogonal rotations of $A$ and verify that gradient descent drives both runs to the same adapted layer, which would confirm the transformation-invariance theorem directly.
Extended reading notes
Core claim
The central claim is that low-rank adaptation does not need two matrices. SingLoRA represents the adapter as a single matrix $A$ and computes the adapted weights as $W_0 + \frac{\alpha}{r}u(t)AA^\top$, so the learned change is symmetric and positive semidefinite by construction. Because only one matrix is trained, there are no two scales to drift apart: in the infinite-width analysis the entries of $A$ stay at order $\Theta(n^{-1/2})$ under a learning rate $\eta = \Theta(n^{-1/2})$, giving stable feature learning with $\Delta f_t = \Theta(1)$, whereas the two-matrix LoRA update forces $b_t$ down to $\Theta(n^{-1})$ and breaks the bound. The paper also proves that ordinary gradient descent is transformation-invariant for SingLoRA, so different parameterizations of the same adapter receive compatible updates, which is false for LoRA. The expressiveness worry raised by the symmetric update is addressed for attention: although $W_q$ and $W_k$ each gain a symmetric term, the key-query interaction $QK^\top$ contains the product $A_qA_q^\top A_kA_k^\top$ of two symmetric matrices that need not commute, so the logits can still express general transformations.
Load-bearing premise
The load-bearing assumption is that confining every adapted weight change to a symmetric matrix times its own transpose never costs the model useful accuracy; the paper argues this only for attention layers, where the query-key product stays general, and gives no direct argument for value projections, output projections, or MLP blocks.
Editorial extensions
If this is right
- SingLoRA is a drop-in replacement for LoRA: at equal rank it needs roughly half the trainable parameters (0.075M versus 0.15M on the RoBERTa and GPT-2 GLUE runs; 12M versus 20M for LLaMA-7B on MNLI), which proportionally reduces optimizer state and checkpoint memory.
- One learning rate suffices: all SingLoRA experiments use a single learning rate where LoRA and LoRA+ grid-search many values, and on LLaMA-7B/MNLI the accuracy fluctuates by about 1% across learning rates while LoRA varies by 4.8%.
- The symmetric restriction does not limit attention: in $QK^\top$ the two symmetric updates $A_qA_q^\top$ and $A_kA_k^\top$ appear as a product that is generically non-symmetric, so attention patterns remain general even though each projection matrix is symmetric.
- No special optimizer is required: because gradient descent is transformation-invariant for SingLoRA, the method avoids the separate learning rates of LoRA+ and the Riemannian conditioning used by other stable variants, and still achieves stable feature learning in the infinite-width setting.
- The design composes with existing LoRA extensions: the paper states SingLoRA is complementary to variants such as DoRA, and its DreamBooth result (DINO 0.151 versus DoRA's 0.148) marks it as a candidate for such combinations.
Reading between the lines
- The expressiveness argument in Section 5 stops at the query-key product, so the method's generality beyond attention is an open empirical question: if a task can only be solved by an asymmetric change to a value or MLP projection, SingLoRA cannot represent it at any rank, and the paper reports no experiment in which such layers are adapted alone.
- Theorem 1, the general statement that any transformation-invariant optimizer achieves efficient feature learning, is reproduced with a proof taken from the cited literature [20]; the paper's own contribution is Theorem 2, that ordinary gradient descent is transformation-invariant for SingLoRA, and the combination of the two is what the stability guarantee rests on.
- The stability analysis predicts the optimal learning-rate scale $\eta = \Theta(n^{-1/2})$ is fixed by width for SingLoRA, which implies the best learning rate should transfer across model sizes within a model family; sweeping learning rates on small and large transformers would test whether that width-independence survives outside the toy model.
- The ramp $u(t)$ is a new design element with no LoRA analogue, and the ablation shows accuracy is flat for $T$ between 0.5% and 8% of training steps; comparing SingLoRA with a fixed $AA^\top$ against the ramped version would show whether the ramp is load-bearing or just a safety device.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces SingLoRA, a parameter-efficient fine-tuning method that replaces LoRA's two-matrix low-rank update BA with a single-matrix symmetric update AA^T (or A_*A^T for rectangular layers), claiming that this removes inter-matrix scale conflicts, guarantees stable feature learning in the infinite-width limit, and roughly halves the number of trainable parameters. The theoretical part analyzes a toy linear model to derive a learning-rate scaling, introduces a transformation-invariance argument (Theorems 1 and 2), and extends the formulation to non-square matrices (Theorem 3). The empirical part reports results on GLUE tasks with RoBERTa and GPT-2, MNLI with LLaMA-7B, and DreamBooth with Stable Diffusion, plus a synthetic attention experiment. The paper argues that the symmetric-PSD restriction is not harmful in attention because the QK^T product of two symmetric updates remains expressive.
Significance. If the claims are correct, SingLoRA is an attractive drop-in alternative to LoRA: it uses fewer parameters, requires no per-matrix learning-rate tuning, and achieves competitive or better accuracy across NLP and image-generation benchmarks. The paper has several strengths: the core formulation is simple and easy to reproduce; the experiments follow the LoRA+ codebase and protocol, which aids fair comparison; the synthetic attention experiment is verified over 1,000 random seeds; and the ablation on the ramp duration T is a useful robustness check. However, the theoretical guarantees rest on proofs that are currently not reliable, and the expressiveness limitation for individually adapted layers is not addressed empirically. The significance is therefore conditional on fixing these issues.
major comments (5)
- [Appendix B.1, Theorem 1] The proof of Theorem 1, which is the bridge from transformation invariance to stable feature learning, is not valid as written. The variables x, y, and z appear in the scaling relations without definition, and the chain-rule step 'Since Z = A1B1^T, by the chain rule, we have ∇Z = ∇Z^T = ∇ZB1A1^T' is dimensionally inconsistent. Even if the norm equalities were true, concluding that the update enables efficient feature learning, ∥δA1∥∥B1∥ = ∥A1∥∥δB1∥ = Θ(1), does not follow from the preceding display. Because Theorem 1 is invoked in the main text to guarantee stable dynamics, this is a load-bearing gap that must be fixed or the theorem must be replaced by a direct width-scaling argument for SingLoRA.
- [Section 4.2, Eq. (8)] The derivation of η = Θ(n^{-1/2}) is a self-consistency argument, not a proof that the dynamics reach the asserted scaling regime. In particular, the line 'a^T_{t-1}(f_{t-1}(x)-y) = Θ(1) by CLT' is unjustified because f_{t-1}(x)-y depends on a_{t-1} through the training dynamics; the independence or zero-mean assumption needed for the CLT is not stated or established. The analysis also implicitly assumes the desired property ft(x)=Θ(1) and then derives the learning-rate scale from it, so the claim that stable feature learning is 'guaranteed by construction' is stronger than what the toy model shows. I recommend either providing a rigorous convergence analysis or rephrasing the claim as a consistency condition.
- [Section 5 and Section 4.4] The expressiveness analysis addresses only the QK^T interaction in attention, where the product of two symmetric updates is not necessarily symmetric. It does not cover layers adapted on their own, such as value projections, output projections, or MLP blocks, where the learned update is exactly a symmetric positive-semidefinite matrix AA^T (or, in the non-square extension, has a PSD left block A_*A_*^T). The GLUE experiments adapt the full LoRA+ layer set, including V and MLP layers, and report only aggregate accuracy, so they cannot isolate whether the PSD restriction is benign on those layers. This is a direct threat to the 'drop-in LoRA replacement' claim. The authors should either prove a representational sufficiency result for single layers or provide experiments that adapt V/MLP layers in isolation and show no accuracy loss relative to LoRA.
- [Tables 2 and 3, Figure 2] The headline empirical results are single-run point estimates without error bars or multiple seeds. Differences such as 91.3% vs. 90.2% for LLaMA MNLI and 0.151 vs. 0.148 for DINO similarity are small relative to typical run-to-run variance, and Table 1 takes baseline numbers directly from LoRA+ without reporting variance. Since the central claim is that SingLoRA 'consistently exceeds' LoRA-family methods, the paper should report means and standard deviations over at least three seeds for the main comparisons, or otherwise justify why the observed margins are beyond noise.
- [Appendix B.3, Theorem 3] The proof of the non-square extension is also incomplete. From X1X1^T = X2X2^T and X1Y1^T = X2Y2^T, the paper asserts that X2 = X1Q and Y2 = Y1Q for the same orthogonal matrix Q. This does not follow unless additional rank or column-space conditions are imposed; the polar-decomposition argument only gives a Q for X2 = X1Q, and the same Q need not satisfy the second equation. Since Theorem 3 is used to justify applicability to non-square layers in the experiments, this gap needs to be fixed.
minor comments (6)
- [Abstract, Eq. (2)] The notation W0 + AA^T in the abstract assumes a square matrix W0, while Section 4.4 defines the non-square update as W0 + A_*A^T. Please state the square assumption in the abstract or use dimension-consistent notation.
- [Section 4.2] The expression '∇aL = Θ n(1)' is ambiguous; it should be written as Θ(1) with respect to n, using the same notation as elsewhere in the paper.
- [Section 4.2] Unlike Section 3, the toy model for SingLoRA does not state whether W0 is set to zero by redefining y. For clarity, please include the same reduction or explain how the analysis changes when W0 x is nonzero.
- [Figure 1 and Section 5] The text says the synthetic experiment was verified with 1K different random seeds, but the figure shows a single convergence plot with no error bars or aggregated statistics. Please report the mean and spread across seeds.
- [Table 3 caption] The caption reads 'Stable diffusion V5' but should read 'V1.5'.
- [Appendix B.2] The proof of Theorem 2 labels the conditions inconsistently: after proving condition (iii), the text says 'This completes the proof of (ii).' Please correct the labels so the three sufficient conditions are mapped correctly.
Circularity Check
No significant circularity found: the stability scaling is derived, the invariance proofs are self-contained or cited to external work, and no reported result reduces to a fitted input by the paper's own equations.
full rationale
SingLoRA's central claims are not circular in the sense defined here. The main theoretical result, the learning-rate scaling eta = Theta(n^{-1/2}) in Section 4.2, is derived from the stated gamma-scaling/CLT toy-model assumptions (eta = Theta(n^c), a entries Theta(n^p), requirement f_t(x) = Theta(1)); it is not obtained by fitting to the experimental accuracies, and the experiments do not use that theoretical learning rate as a fitted parameter to produce 'predictions'. The transformation-invariance claims in Theorems 2 and 3 are proved for the AA^T parameterization using polar decomposition and the three sufficient conditions; the proof does not assume the conclusion. Theorem 1 and the LoRA instability analysis are borrowed from external prior work ([4], [19], [20]) and are explicitly cited rather than self-cited; the present authors do not rely on an unverified self-citation chain. The parameter-halving claim is immediate arithmetic from the construction A in R^{n x r} versus B,A with 2nr parameters, so it is a definitional consequence, not a concealed prediction. The expressiveness argument in Section 5 is an independent mathematical observation: the product of two non-commuting symmetric matrices AqAq^T and AkAk^T need not be symmetric, so the symmetric parameterization does not by itself force symmetric attention logits; the synthetic experiment tests exactly this Q/K pair. No evaluation table is generated by plugging fitted values back into the model's own formula. There are genuine correctness and evidence concerns: the toy-model scaling mixes CLT cancellations with deterministic gamma-notation in a way that is not fully rigorous, and the benignness of the symmetric-PSD constraint for individually adapted V/MLP layers is asserted without isolating such layers in experiments. However, these are correctness or evidence risks, not circularity. The paper is self-contained against external benchmarks and does not reduce any central claim to its own inputs by construction.
Assumptions & free parameters
free parameters (1)
- ramp duration T =
1% of total training steps in main experiments (ablation range 0.5% to 8%)
assumptions (4)
- domain assumption Stable feature learning in the rank-1 linear toy model transfers to full multi-layer transformer fine-tuning with SGD or AdamW
- domain assumption Transformation-invariant optimizers achieve efficient feature learning (Theorem 1)
- ad hoc to paper Symmetric positive-semidefinite updates on each adapted layer are sufficiently expressive for the reported tasks
- domain assumption Bounded-error and CLT scaling assumptions: (f(x)-y) = Theta_n(1), a^T x = Theta_n(1) under Kaiming initialization
Cite this review
Pith. "Pith review of SingLoRA: Low Rank Adaptation Using a Single Matrix." pith.science (2026). https://pith.science/paper/72HNXUG6
@misc{pith2026250705566,
author = {Pith},
title = {Pith review of: SingLoRA: Low Rank Adaptation Using a Single Matrix},
year = {2026},
howpublished = {\url{https://pith.science/paper/72HNXUG6}},
note = {Machine review of arXiv:2507.05566}
}
read the original abstract
Low-Rank Adaptation (LoRA) has significantly advanced parameter-efficient fine-tuning of large pretrained models. LoRA augments the pre-trained weights of a model by adding the product of two smaller matrices that together form a low-rank matrix update. Recent research has shown that scale disparities between these two matrices often cause unstable training dynamics, leading to suboptimal performance. In this paper, we propose SingLoRA, which reformulates low-rank adaptation by learning the weights update as a decomposition of a single low-rank matrix multiplied by its transpose. This simple design inherently removes inter-matrix scale conflicts, ensuring stable optimization, and roughly halves the parameter count. We analyze SingLoRA within the infinite-width neural network framework, showing that it guarantees stable feature learning by construction. Extensive experiments on multiple tasks validate these benefits. In common sense reasoning, fine-tuning LLama 7B on MNLI with SingLoRA achieves 91.3% accuracy - surpassing LoRA (89.1%) and LoRA+ (90.2%) - while using only 60% of their parameter budget. In image generation, fine-tuning Stable Diffusion with SingLoRA significantly improves image fidelity on DreamBooth, achieving a DINO similarity score of 0.151, compared to scores of 0.148 and 0.143 for DoRA and LoRA, respectively.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[20]
Riemannian preconditioned lora for fine-tuning foundation models
Fangzhao Zhang and Mert Pilanci. Riemannian preconditioned lora for fine-tuning foundation models. arXiv preprint arXiv:2402.02347, 2024
arXiv 2024
-
[1]
Emerging properties in self-supervised vision transformers
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650–9660, 2021
2021
-
[2]
Krona: Parameter efficient tuning with kronecker adapter
Ali Edalati, Marzieh Tahaei, Ivan Kobyzev, Vahid Partovi Nia, James J Clark, and Mehdi Rezagholizadeh. Krona: Parameter efficient tuning with kronecker adapter. arXiv preprint arXiv:2212.10650, 2022
arXiv 2022
-
[3]
On the impact of the activation function on deep neural networks training
Soufiane Hayou, Arnaud Doucet, and Judith Rousseau. On the impact of the activation function on deep neural networks training. In International conference on machine learning , pages 2672–2680. PMLR, 2019
work page 2019
-
[4]
Lora+: Efficient low rank adaptation of large models
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. International Conference on Machine Learning , 2024
work page 2024
-
[5]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770–778, 2016
2016
-
[6]
Lora: Low-rank adaptation of large language models
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021
arXiv 2021
-
[7]
Fedpara: Low-rank hadamard prod- uct for communication-efficient federated learning
Nam Hyeon-Woo, Moon Ye-Bin, and Tae-Hyun Oh. Fedpara: Low-rank hadamard prod- uct for communication-efficient federated learning. International Conference on Learning Representations, 2021
work page 2021
Show all 23 references
-
[8]
Adam: A method for stochastic optimization
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014
2014 arXiv
-
[9]
Dora: Weight-decomposed low-rank adaptation
Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. International Conference on Machine Learning , 2024
2024
-
[10]
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , p...
2021
-
[11]
High- resolution image synthesis with latent diffusion models
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684–10695, 2022
2022
-
[12]
Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2250...
2023
-
[13]
Deep informa- tion propagation
Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep informa- tion propagation. arXiv preprint arXiv:1611.01232, 2016
2016 arXiv
-
[14]
Llama: Open and efficient foundation language models
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023
2023 arXiv
-
[15]
Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation
Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. Findings of the Association for Computational Linguistics , 2022
2022
-
[16]
Glue: A multi-task benchmark and analysis platform for natural language under- standing
Alex Wang. Glue: A multi-task benchmark and analysis platform for natural language under- standing. International Conference on Learning Representations , 2018. 11
2018
-
[17]
Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer
Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.03466, 2022
2022 arXiv
-
[18]
Tensor programs vi: Feature learning in infinite-depth neural networks
Greg Yang, Dingli Yu, Chen Zhu, and Soufiane Hayou. Tensor programs vi: Feature learning in infinite-depth neural networks. arXiv preprint arXiv:2310.02244, 2023
2023 arXiv
-
[19]
Lora done rite: Robust invariant transformation equilibration for lora optimization
Jui-Nan Yen, Si Si, Zhao Meng, Felix Yu, Sai Surya Duvvuri, Inderjit S Dhillon, Cho-Jui Hsieh, and Sanjiv Kumar. Lora done rite: Robust invariant transformation equilibration for lora optimization. International Conference on Learning Representations , 2025
2025
-
[21]
Adalora: Adaptive budget allocation for parameter- efficient fine-tuning
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter- efficient fine-tuning. International Conference on Learning Representations , 2023
2023
-
[22]
Lora-drop: Efficient lora parameter pruning based on output evaluation
Hongyun Zhou, Xiangyu Lu, Wang Xu, Conghui Zhu, Tiejun Zhao, and Muyun Yang. Lora-drop: Efficient lora parameter pruning based on output evaluation. arXiv preprint arXiv:2402.07721, 2024
2024 arXiv
-
[23]
a photo of a sks human
Bojia Zi, Xianbiao Qi, Lingzhi Wang, Jianan Wang, Kam-Fai Wong, and Lei Zhang. Delta- lora: Fine-tuning high-rank parameters with the delta of low-rank matrices. arXiv preprint arXiv:2309.02411, 2023. 12 Supplementary A Additional experiments A.1 Initialization and choice of T...
2023 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.