Pith. sign in

REVIEW 4 major objections 3 minor 78 references

Transformed Low-rank Adaptation via Tensor Decomposition and Its Applications to Text-to-image Models

T0 review · 4 major / 3 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A learned full-rank transform applied before a compact low-rank residual can make parameter-efficient fine-tuning both smaller and more accurate than LoRA.

desk verdict A genuinely new PEFT combination with real gains on diffusion fine-tuning, but the central rank-reduction mechanism is asserted rather than demonstrated and the abstract overclaims. read the letter →

arxiv 2501.08727 v2 pith:XPEMGAQ3 submitted 2025-01-15 cs.LG

Zerui Tao , Yuhta Takida , Naoki Murata , Qibin Zhao , Yuki Mitsufuji This is my paper · ORCID
classification cs.LG
keywords parameter-efficientfine-tuninglow-rankadaptationtext-to-imagegenerationtensor-ringdecompositiontransformsubject-drivencontrollableStableDiffusion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes TLoRA, a parameter-efficient fine-tuning method for text-to-image models that replaces LoRA's single low-rank additive update with two parts: a learnable, full-rank, dense transform applied to the pre-trained weight, followed by a compact tensor-ring residual. The intended mechanism is that the transform aligns the pre-trained weight with the desired weight closely enough that the remaining residual is lower-rank, so it can be approximated with very few parameters without losing accuracy. If the claim holds, users can fine-tune large diffusion models with far fewer trainable parameters than LoRA while matching or improving subject alignment, prompt alignment, and controllability. The paper tests this by fine-tuning Stable Diffusion models for subject-driven and controllable generation, reporting that TLoRA beats LoRA and several baselines in most settings at a fraction of the parameter budget.

What carries the argument

The machinery is the tensor-ring matrix (TRM) format for the transform $T$ and the tensor-ring (TR) decomposition for the residual $\Delta$. A TRM represents an $I\times I$ matrix by contracting a chain of small fourth-order core tensors; with dense cores it can express dense full-rank matrices at a cost far below $O(I^2)$. A TR represents the residual by contracting even smaller third-order cores, giving an even more compact parameterization. Two engineering choices carry the argument: initialize every TRM core so the product is the identity, so fine-tuning starts from the pretrained weight, and zero-initialize the first TR core so the residual starts at zero, unlike earlier tensor-decomposition PEFT that used random initialization and became unstable. Identity and orthogonality regularization are computed on the cores rather than on the full matrix, making them cheap enough to use during fine-tuning.

What would settle it

Take a fully fine-tuned checkpoint (for instance SDXL-Inpaint) and, at matched parameter budgets, measure how well TLoRA reconstructs each attention weight compared with LoRA. The claim predicts that the residual after the transform, $\Delta'_* = W_* - W_0 T$, has much smaller effective rank than the raw difference $\Delta_* = W_* - W_0$ on most layers, so TLoRA's reconstruction error is lower; if that rank drop does not appear across most layers of a model outside the paper's experiments, the central mechanism is not general.

Watch

Extended reading notes

Core claim

The central claim is that the approximation gap of LoRA is caused by forcing the entire fine-tuning change into a low-rank additive matrix, and that this gap can be closed by learning a full-rank transform first. The paper writes the fine-tuned layer as $y' = (W_0 T + \Delta)x$, where $T$ is a tensor-ring-matrix transform initialized to identity and $\Delta$ is a tensor-ring residual. Its discovery is that after optimizing $T$, the residual $\Delta'_* = W_* - W_0 T$ has a smaller effective rank than the original difference $\Delta_* = W_* - W_0$, and therefore can be represented by a much more compact structure with a smaller approximation error. The paper also shows that DoRA's magnitude-direction decomposition can be seen as a special case with a diagonal transform, and that fixed-transform methods lack the adaptivity of a learnable dense transform.

Load-bearing premise

The load-bearing premise is that after the learned transform, the remaining residual is actually lower-rank, so a compact tensor-ring approximation can capture it; the paper demonstrates this with a simulation on a single SDXL attention layer, but does not prove it across tasks and layers.

Editorial extensions

If this is right

  • TLoRA with 0.4M parameters outperforms LoRA with 1.45M parameters on subject-driven fine-tuning of SDXL, so ultra-low parameter budgets need not sacrifice quality.
  • Adding the transform adaptation consistently improves LoRA and tensor-ring baselines in approximation and in controllable generation tasks, indicating the transform is the load-bearing component.
  • The transform-plus-residual scheme unifies existing methods: LoRA is the case $T=I$, DoRA uses a diagonal transform, and fixed-transform methods are non-learnable special cases, giving a shared way to design PEFT.
  • Because the residual after the transform is smaller-rank, users can smoothly trade parameter budget against error by adjusting the TRM and TR ranks, rather than relying only on LoRA's coarse rank steps.
  • The method is presented as general beyond text-to-image models, with tensor decomposition particularly suited to convolutional layers whose weights are naturally multi-way arrays.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the rank-reduction premise could be tested directly by computing singular-value spectra of $\Delta'_* = W_* - W_0 T$ across many layers of a fully fine-tuned model; the paper only shows one SDXL attention layer.
  • Editorial inference: if the premise holds, TLoRA should transfer to large-language-model fine-tuning, where weight matrices are larger and tensor decompositions become even more parameter-efficient; the paper lists this as future work.
  • Editorial inference: the identity initialization of TRM could serve as a warm start for a continuous path from the pretrained model to the target, potentially enabling interpolation between LoRA and full fine-tuning.
  • Editorial inference: pairing the transform with per-layer rank selection might let practitioners allocate the tiny parameter budget to layers where the residual remains high-rank.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes TLoRA, a parameter-efficient fine-tuning method that combines a full-rank dense transform parameterized as a tensor-ring matrix (TRM) with a compact residual parameterized as a tensor-ring (TR) decomposition. The motivating idea, formalized in Eq. (3), is that a learned transform T aligns the pretrained weight W0 with the desired weight W*, so that the residual W* - W0T is low-rank and can be approximated with very few parameters. The paper provides an identity initialization for the TRM (Prop. 1), an efficient orthogonal regularization scheme (Prop. 2), a discussion relating DoRA and fixed-transform methods to the transform-plus-residual framework, and experiments on subject-driven generation with SDXL and controllable generation with SD v1.5. The central empirical claim is that TLoRA achieves better performance and parameter efficiency than LoRA and several baselines.

Significance. If the rank-reduction premise is valid, the transform-plus-residual view is a useful conceptual unification of LoRA, DoRA, and OFT-style methods, and the tensor-ring parameterizations offer a flexible parameter-efficiency frontier. The paper has concrete strengths: the code is released, Propositions 1 and 2 are proved in Appendix B.2, the identity initialization addresses a real instability in prior TR-based PEFT methods, and the DoRA connection in Eq. (6) is clearly stated. The main weakness is that the load-bearing rank-reduction claim is supported only by weight-space simulations on off-the-shelf checkpoints (SDXL-Inpaint and Llama2-chat), not by evidence on the fine-tuned weights from the paper's own tasks, and the controllable-generation results contain a task (L2I) where TLoRA only ties or loses to LoRA.

major comments (4)
  1. [Sec. 3.2, Eq. (3)] The central premise that Delta'_* = W* - W0T has smaller rank than Delta* = W* - W0 is asserted in the sentence 'After the transform, Delta'* should have a smaller rank' but is never demonstrated on the tasks used in Sec. 4. The simulation in Fig. 2 and Appendix A uses SDXL-Inpaint and Llama2-chat checkpoints, not the fine-tuned weights produced by the paper's own subject-driven or controllable-generation experiments, and it reports MSE versus parameter budget rather than the singular-value decay or effective rank of Delta'_*. This is load-bearing: if real fine-tuned weights are not amenable to the assumed rank reduction, the advantage of Eq. (3) over LoRA is unexplained. Please add a direct test on the actual fine-tuned weights from a few subjects/tasks, e.g., compare the singular-value spectra of Delta* and Delta'_* after optimizing T, or report the same MSE-vs-budget approximation curves on the checkpoints obtained in Sec. 4.
  2. [Table 2, L2I row] The abstract and conclusion claim general 'better performances and parameter efficiency' compared to LoRA, but the L2I results in Table 2 contradict this. LoRA r=4 achieves the best L2I error (5.32) with 0.80M parameters, TLoRA*(2,4) only ties it with 0.94M parameters, and TLoRA(2,6) is worse (5.84). The paper acknowledges that LoRA is best on L2I, but the universal wording of the central claim remains. Please restrict the claim to tasks where the transform helps (S2I and C2I), or explain why L2I is a case where the transform is unnecessary and show that the parameter-efficiency claim is intended only in a Pareto-dominance sense rather than uniformly.
  3. [Fig. 4] The subject-driven Pareto curves are presented without error bars or statistical tests. The curves show trajectories of CLIP-I, DINOv2, CLIP-T, and LPIPS across training epochs, but with 30 DreamBooth subjects and per-method tuned learning rates, the headline 'our method achieves the best overall results' is not statistically secured. Please report per-subject means and standard deviations (or confidence intervals), the number of random seeds, and paired significance tests for the final-epoch metrics or for Pareto dominance. This is especially important because the visually reported gains appear modest on some of the subplots.
  4. [Sec. 3.3, R_O equation] The orthogonal regularization formula likely has an error. It subtracts I_{R^2}/R from the core expression of Prop. 2. However, substituting the identity initialization from Prop. 1, A_d[i,i,:,:] = 1_{R x R}/R, into the Prop. 2 product core gives the target delta_{ij} * 1_{R^2 x R^2}/R^2, not delta_{ij} * I_{R^2}/R. As written, the regularizer encourages a different matrix and for D > 2 does not enforce T T^T ≈ I. Since the paper states that identity regularization is preferred and is the one used in the experiments, this does not change the empirical conclusions, but the equation and its derivation should be corrected.
minor comments (3)
  1. [Table 4 and Sec. 4.2] In Sec. 4.2 the text says 'we also test TRM with rank 2 and TR residual adaptation with ranks 6 and 8, denoted as TLoRA(2, 6) and TLoRA(2, 8)', but Table 4 lists r_TRM = 1 for TLoRA(2,8). Please align the notation so the reported hyperparameter settings match the text.
  2. [Sec. 3.4] In the initialization paragraph, 'B_{1:D} and C_{1:C}' contains a typo; it should be 'C_{1:D}'.
  3. [Sec. 5] The conclusion states that the method is 'general for PEFT of other tasks', but the only non-T2I evidence is the Llama2 weight-approximation simulation in Appendix A, which is not a fine-tuning experiment. Please soften this claim or add a small LLM fine-tuning experiment.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TLoRA is evaluated on independent fine-tuning tasks; the rank-reduction premise is a stated assumption supported by a separate approximation study, not derived from the method's own outputs.

full rationale

The paper's central claim is that adding a full-rank dense TRM transform before a compact residual in Eq. (3) improves parameter-efficient fine-tuning. This is an architectural hypothesis, not a circular derivation. The transform and residual parameters are learned from task data (DreamBooth, CelebA-HQ, ADE20K) and evaluated with external metrics (CLIP-I/T, DINOv2, LPIPS, mIoU, IoU/F1). No fitted quantity is renamed as a prediction: Fig. 2 is an approximation study on pre-trained SDXL and SDXL-Inpaint checkpoints that motivates the design, while Sec. 4 reports actual fine-tuning results on independent benchmarks. The sentence 'After the transform, Delta'_* should have a smaller rank' (Sec. 3.2) is an assumption, not a theorem, and the paper itself reports in Tab. 2 that LoRA ties TLoRA on L2I; an unverified assumption is a correctness or verifiability concern, not circularity. The citations to tensor-ring literature [9, 73] include an author (Q. Zhao), but TR/TRM are standard, parameter-free mathematical representations with proofs in the appendix (Prop. 1), so this self-citation is not load-bearing. No step of the derivation reduces to its own input by construction.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new physical or mathematical entities are postulated. The method introduces a new parameterization (TRM transform plus TR residual), which is a structural choice, not an invented entity. The key free parameters are the tensor ranks and shapes, which are selected empirically.

free parameters (4)
  • TRM rank R_T = 1 (subject-driven), 2 (controllable)
    Chosen by hand to trade off parameter count and expressiveness; not derived from theory.
  • TR residual rank R_R = 2 (subject-driven), 6 or 8 (controllable)
    Selected based on experiments to achieve low parameter counts while preserving performance.
  • Tensorization shape dict = {'320': [4,8,10], '640': [8,8,10], '768': [8,8,12], '1280': [8,10,16], ...}
    Fixed without search; affects how well TRM/TR represent matrices and hence the parameter efficiency and accuracy.
  • Identity regularization scale lambda = 0 or 1e-3
    Tuned from {0, 1e-3} based on validation performance; contributes to the final results.
assumptions (5)
  • domain assumption The desired fine-tuned weight W* can be approximated as W0*T + Delta, with T a full-rank dense transform and Delta a low-rank residual.
    Central motivation in Sec. 3.2; no proof is provided, only simulation evidence on one layer of SDXL.
  • domain assumption Minimizing weight-space MSE in the simulation study is indicative of downstream fine-tuning performance.
    Used to justify the simulation in Fig. 2 and to guide architectural choices; the paper does not test this transfer directly.
  • standard math Tensor-ring matrices can compactly represent dense full-rank matrices that are useful for aligning pre-trained weights.
    Based on known expressiveness of tensor networks, cited from [9,42,73]; the paper adds a simulation comparing to BOFT.
  • standard math Zero-initializing the first TR core preserves the pre-trained model at initialization.
    By construction of the TR contraction; the paper uses this to avoid training instability from non-zero initialization.
  • domain assumption Injecting adapters only into attention layers (key, value, query, output) is sufficient for fine-tuning T2I models.
    Adopted from prior works [4,32,47] and used for fair comparison; not re-derived in this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Transformed Low-rank Adaptation via Tensor Decomposition and Its Applications to Text-to-image Models." pith.science (2026). https://pith.science/paper/XPEMGAQ3

@misc{pith2026250108727,
  author       = {Pith},
  title        = {Pith review of: Transformed Low-rank Adaptation via Tensor Decomposition and Its Applications to Text-to-image Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XPEMGAQ3}},
  note         = {Machine review of arXiv:2501.08727}
}
read the original abstract

Parameter-Efficient Fine-Tuning (PEFT) of text-to-image models has become an increasingly popular technique with many applications. Among the various PEFT methods, Low-Rank Adaptation (LoRA) and its variants have gained significant attention due to their effectiveness, enabling users to fine-tune models with limited computational resources. However, the approximation gap between the low-rank assumption and desired fine-tuning weights prevents the simultaneous acquisition of ultra-parameter-efficiency and better performance. To reduce this gap and further improve the power of LoRA, we propose a new PEFT method that combines two classes of adaptations, namely, transform and residual adaptations. In specific, we first apply a full-rank and dense transform to the pre-trained weight. This learnable transform is expected to align the pre-trained weight as closely as possible to the desired weight, thereby reducing the rank of the residual weight. Then, the residual part can be effectively approximated by more compact and parameter-efficient structures, with a smaller approximation error. To achieve ultra-parameter-efficiency in practice, we design highly flexible and effective tensor decompositions for both the transform and residual adaptations. Additionally, popular PEFT methods such as DoRA can be summarized under this transform plus residual adaptation scheme. Experiments are conducted on fine-tuning Stable Diffusion models in subject-driven and controllable generation. The results manifest that our method can achieve better performances and parameter efficiency compared to LoRA and several baselines.

Figures

Figures reproduced from arXiv: 2501.08727 by the authors.

Figure 1
Figure 1. Qualitative comparison of the subject-driven generation results. Results are generated using fine-tuned checkpoint by each [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Simulation study on pre-trained SDXL [46] and fine-tuned SDXL-Inpaint [56] weights. SDXL-Inpaint is fine-tuned on image￾mask pairs to facilitate imputation ability of the SDXL base model. We investigate the approximation of UNet attention layers, which are counterpart of our fine-tuning targets in experiments. We test the approximation error on three cases: (a) Additive low-rank difference, which LoRA can effectivel… view at source ↗
Figure 4
Figure 4. Subject-driven generation results. The larger DINOv2 [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: Qualitative results of controllable generation. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Simulation on different layers of the SDXL and SDXL-Inpaint models. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Simulation on different layers of the Llama2 7B and Llama2-chat 7B models. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Training process of LoRETTA on the C2I task. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Qualitative comparison of the subject-driven generation results. [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Qualitative comparison of the subject-driven generation results. [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Qualitative comparison of the subject-driven generation results. [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

78 extracted references · 60 canonical work pages

  1. [1]

    Tqcompressor: improving tensor de- composition methods in neural networks via permutations

    V Abronin, A Naumov, D Mazur, D Bystrov, K Tsarova, Ar Melnikov, I Oseledets, Sergey Dolgov, R Brasher, and Michael Perelshtein. Tqcompressor: improving tensor de- composition methods in neural networks via permutations. arXiv preprint arXiv:2401.16367, 2024. 5

    arXiv 2024

  2. [2]

    Tensor train low-rank approximation (tt-lora): Democratizing ai with accelerated llms.arXiv preprint arXiv:2408.01008, 2024

    Afia Anjum, Maksim E Eren, Ismael Boureima, Boian Alexandrov, and Manish Bhattarai. Tensor train low-rank approximation (tt-lora): Democratizing ai with accelerated llms.arXiv preprint arXiv:2408.01008, 2024. 2, 3, 7

    arXiv 2024

  3. [3]

    Sparse high rank adapters

    Kartikeya Bhardwaj, Nilesh Pandey, Sweta Priyadarshi, Viswanath Ganapathy, Shreya Kadambi, Rafael Esteves, Shubhankar Borse, Paul Whatmough, Risheek Garrepalli, Mart van Baalen, Harris Teague, and Markus Nagel. Sparse high rank adapters. InAdvances in Neural Information Pro- cessing Systems, pages 13685–13715. Curran Associates, Inc., 2024. 3, 6

    work page 2024

  4. [4]

    Ether: Efficient finetuning of large-scale models with hyperplane reflections

    Massimo Bini, Karsten Roth, Zeynep Akata, and Anna Khoreva. Ether: Efficient finetuning of large-scale models with hyperplane reflections. InForty-first International Con- ference on Machine Learning. 3, 5, 6, 4, 7

    work page

  5. [5]

    Foura: Fourier low rank adapta- tion.arXiv preprint arXiv:2406.08798, 2024

    Shubhankar Borse, Shreya Kadambi, Nilesh Prasad Pandey, Kartikeya Bhardwaj, Viswanath Ganapathy, Sweta Priyadarshi, Risheek Garrepalli, Rafael Esteves, Munawar Hayat, and Fatih Porikli. Foura: Fourier low rank adapta- tion.arXiv preprint arXiv:2406.08798, 2024. 2, 3, 6, 7

    arXiv 2024

  6. [6]

    How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks)

    Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks). InInternational Conference on Computer Vision, 2017. 4, 5

    work page 2017

  7. [7]

    Para: Personalizing text-to-image diffusion via parameter rank reduction

    Shangyu Chen, Zizheng Pan, Jianfei Cai, and Dinh Phung. Para: Personalizing text-to-image diffusion via parameter rank reduction. InThe Thirteenth International Conference on Learning Representations, 2025. 3, 6

    work page 2025

  8. [8]

    Quanta: Efficient high- rank fine-tuning of llms with quantum-informed tensor adap- tation.arXiv preprint arXiv:2406.00132, 2024

    Zhuo Chen, Rumen Dangovski, Charlotte Loh, Owen Dugan, Di Luo, and Marin Soljaˇci´c. Quanta: Efficient high- rank fine-tuning of llms with quantum-informed tensor adap- tation.arXiv preprint arXiv:2406.00132, 2024. 2, 3, 6, 7

    arXiv 2024

Show all 78 references
  1. [9]

    Tensor networks for dimensionality reduction and large-scale optimization: Part 1 low-rank tensor decompositions.Foundations and Trends® in Machine Learning, 9(4-5):249–429, 2016

    Andrzej Cichocki, Namgil Lee, Ivan Oseledets, Anh-Huy Phan, Qibin Zhao, and Danilo P Mandic. Tensor networks for dimensionality reduction and large-scale optimization: Part 1 low-rank tensor decompositions.Foundations and Trends® in Machine Learning, 9(4-5):249–429, 2016. 2, 3, 4, 7

    2016

  2. [10]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. InForty-first International Conference on Machi...

    2024

  3. [11]

    A note on lora.arXiv preprint arXiv:2404.05086, 2024

    Vlad Fomenko, Han Yu, Jongho Lee, Stanley Hsieh, and Weizhu Chen. A note on lora.arXiv preprint arXiv:2404.05086, 2024. 2, 3, 6

    2024 arXiv

  4. [12]

    An image is worth one word: Personalizing text-to-image gener- ation using textual inversion

    Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image is worth one word: Personalizing text-to-image gener- ation using textual inversion. InThe Eleventh International Conference on Learning Representations, 2023. 2, 6

    2023

  5. [13]

    Parameter-efficient fine-tuning with discrete fourier transform

    Ziqi Gao, Qichao Wang, Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, and Jia Li. Parameter-efficient fine-tuning with discrete fourier transform. InForty-first International Conference on Machine Learning, 2024. 2, 3, 6, 7

    2024

  6. [14]

    Ultimate tensorization: compress- ing convolutional and fc layers alike.arXiv preprint arXiv:1611.03214, 2016

    Timur Garipov, Dmitry Podoprikhin, Alexander Novikov, and Dmitry Vetrov. Ultimate tensorization: compress- ing convolutional and fc layers alike.arXiv preprint arXiv:1611.03214, 2016. 3, 7

    2016 arXiv

  7. [15]

    Deep learning book notation.https:// github.com/goodfeli/dlbook_notation

    Ian Goodfellow. Deep learning book notation.https:// github.com/goodfeli/dlbook_notation. 3

  8. [16]

    Deep Learning

    Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.http : / / www . deeplearningbook.org. 3

    2016

  9. [17]

    Mix-of-show: Decentralized low- rank adaptation for multi-concept customization of diffusion models.Advances in Neural Information Processing Sys- tems, 36, 2024

    Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yun- peng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, et al. Mix-of-show: Decentralized low- rank adaptation for multi-concept customization of diffusion models.Advances in Neural Information Processing S...

    2024

  10. [18]

    Svdiff: Compact param- eter space for diffusion fine-tuning

    Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact param- eter space for diffusion fine-tuning. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 7323–7334, 2023. 3, 6

    2023

  11. [19]

    Parameter-efficient transfer learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. InInternational conference on machine learning, pages 2790–2799. PMLR, 2019. 3, 6

    2019

  12. [20]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. InIn- ternational Conference on Learning Representations, 2022. 2, 3, 5, 6, 4

    2022

  13. [21]

    SaRA: High-efficient diffusion model fine-tuning with progressive sparse low-rank adapta- tion

    Teng Hu, Jiangning Zhang, Ran Yi, Hongrui Huang, Yabiao Wang, and Lizhuang Ma. SaRA: High-efficient diffusion model fine-tuning with progressive sparse low-rank adapta- tion. InThe Thirteenth International Conference on Learn- ing Representations, 2025. 3, 6

    2025

  14. [22]

    HiRA: Parameter-efficient hadamard high-rank adap- tation for large language models

    Qiushi Huang, Tom Ko, Zhan Zhuang, Lilian Tang, and Yu Zhang. HiRA: Parameter-efficient hadamard high-rank adap- tation for large language models. InThe Thirteenth Interna- tional Conference on Learning Representations, 2025. 3, 6

    2025

  15. [23]

    Fed- para: Low-rank hadamard product for communication- efficient federated learning

    Nam Hyeon-Woo, Moon Ye-Bin, and Tae-Hyun Oh. Fed- para: Low-rank hadamard product for communication- efficient federated learning. InInternational Conference on Learning Representations, 2022. 3, 6 9

    2022

  16. [24]

    Fact: Factor-tuning for lightweight adaptation on vision transformer

    Shibo Jie and Zhi-Hong Deng. Fact: Factor-tuning for lightweight adaptation on vision transformer. InProceed- ings of the AAAI conference on artificial intelligence, pages 1060–1068, 2023. 3, 7

    2023

  17. [25]

    Vera: Vector-based random matrix adaptation

    Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. Vera: Vector-based random matrix adaptation. In The Twelfth International Conference on Learning Represen- tations. 3, 6

  18. [26]

    Multi-concept customization of text-to-image diffusion

    Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1931–1941, 2023. 2, 6

    1931

  19. [27]

    Direct consistency optimization for compositional text- to-image personalization.arXiv preprint arXiv:2402.12004,

    Kyungmin Lee, Sangkyung Kwak, Kihyuk Sohn, and Jinwoo Shin. Direct consistency optimization for compositional text- to-image personalization.arXiv preprint arXiv:2402.12004,

    arXiv

  20. [28]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. InProceed- ings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing. Association for Computational Linguistics, 2021. 3, 6

    2021

  21. [29]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. InInterna- tional conference on machine learning, pages 12888–12900. PMLR, 2022. 4

    2022

  22. [30]

    Prefix-tuning: Optimiz- ing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimiz- ing continuous prompts for generation. InProceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Lon...

    2021

  23. [31]

    DoRA: Weight-decomposed low-rank adaptation

    Shih-yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. DoRA: Weight-decomposed low-rank adaptation. InForty-first International Conference on Ma- chine Learning, 2024. 2, 3, 6, 4

    2024

  24. [32]

    Black, Adrian Weller, and Bernhard Sch¨olkopf

    Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng, Yandong Wen, Michael J. Black, Adrian Weller, and Bernhard Sch¨olkopf. Parameter-efficient orthog- onal finetuning via butterfly factorization. InThe Twelfth ...

    2024

  25. [33]

    Badam: A memory ef- ficient full parameter optimization method for large language models

    Qijun Luo, Hengxu Yu, and Xiao Li. Badam: A memory ef- ficient full parameter optimization method for large language models. InAdvances in Neural Information Processing Sys- tems, pages 24926–24958. Curran Associates, Inc., 2024. 3, 6, 7

    2024

  26. [34]

    Parameter efficient quasi-orthogonal fine- tuning via givens rotation

    Xinyu Ma, Xu Chu, Zhibang Yang, Yang Lin, Xin Gao, and Junfeng Zhao. Parameter efficient quasi-orthogonal fine- tuning via givens rotation. InForty-first International Con- ference on Machine Learning. 3, 5, 7

  27. [35]

    A tensorized transformer for language modeling.Advances in neural information pro- cessing systems, 32, 2019

    Xindian Ma, Peng Zhang, Shuai Zhang, Nan Duan, Yuexian Hou, Ming Zhou, and Dawei Song. A tensorized transformer for language modeling.Advances in neural information pro- cessing systems, 32, 2019. 3, 7

    2019

  28. [36]

    Peft: State-of-the-art parameter-efficient fine-tuning meth- ods.https://github.com/huggingface/peft,

    Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-art parameter-efficient fine-tuning meth- ods.https://github.com/huggingface/peft,

  29. [37]

    Scaling recurrent models via orthogonal approximations in tensor trains

    Ronak Mehta, Rudrasis Chakraborty, Yunyang Xiong, and Vikas Singh. Scaling recurrent models via orthogonal approximations in tensor trains. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 10571–10579, 2019. 3, 7

    2019

  30. [38]

    T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models

    Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. InProceedings of the AAAI Conference on Artificial Intelligence, pages 4296–4304, 2024. 2

    2024

  31. [39]

    RoSA: Accurate parameter-efficient fine-tuning via robust adaptation

    Mahdi Nikdan, Soroush Tabesh, Elvir Crnˇcevi´c, and Dan Al- istarh. RoSA: Accurate parameter-efficient fine-tuning via robust adaptation. InForty-first International Conference on Machine Learning, 2024. 2, 3, 6

    2024

  32. [40]

    Tensorizing neural networks.Advances in neural information processing systems, 28, 2015

    Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural networks.Advances in neural information processing systems, 28, 2015. 3, 7

    2015

  33. [41]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research Jou...

    2024

  34. [42]

    Tensor-train decomposition.SIAM Jour- nal on Scientific Computing, 33(5):2295–2317, 2011

    Ivan V Oseledets. Tensor-train decomposition.SIAM Jour- nal on Scientific Computing, 33(5):2295–2317, 2011. 2, 3, 4, 7

    2011

  35. [43]

    Lisa: Layerwise importance sampling for memory-efficient large language model fine- tuning

    Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. Lisa: Layerwise importance sampling for memory-efficient large language model fine- tuning. InAdvances in Neural Information Processing Sys- tems, pages 57018–57049. Curran Associates, Inc., 202...

    2024

  36. [44]

    Compressing recurrent neural networks with tensor ring for action recognition

    Yu Pan, Jing Xu, Maolin Wang, Jinmian Ye, Fei Wang, Kun Bai, and Zenglin Xu. Compressing recurrent neural networks with tensor ring for action recognition. InProceedings of the AAAI Conference on Artificial Intelligence, pages 4683– 4690, 2019. 3, 7

    2019

  37. [45]

    Tt-vit: Vision transformer compression using tensor-train decomposition

    Hoang Pham Minh, Nguyen Nguyen Xuan, and Son Tran Thai. Tt-vit: Vision transformer compression using tensor-train decomposition. InInternational Conference on Computational Collective Intelligence, pages 755–767. Springer, 2022. 3, 7

    2022

  38. [46]

    SDXL: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. InThe Twelfth Interna- tional Conference on Learning Representations, 2024. 2, 4, 6

    2024

  39. [47]

    Controlling text-to-image diffusion by orthogo- nal finetuning.Advances in Neural Information Processing Systems, 36:79320–79362, 2023

    Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Sch¨olkopf. Controlling text-to-image diffusion by orthogo- nal finetuning.Advances in Neural Information Processing Systems, 36:79320–79362, 2023. 3, 5, 6, 7, 8, 1, 4 10

    2023

  40. [48]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, p...

    2021

  41. [49]

    Hierarchical text-conditional image gener- ation with clip latents.arXiv preprint arXiv:2204.06125, 1 (2):3, 2022

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gener- ation with clip latents.arXiv preprint arXiv:2204.06125, 1 (2):3, 2022. 2, 6

    2022 arXiv

  42. [50]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 2, 7, 6

    2022

  43. [51]

    Rb-modulation: Training-free personalization of diffu- sion models using stochastic optimal control.arXiv preprint arXiv:2405.17401, 2024

    Litu Rout, Yujia Chen, Nataniel Ruiz, Abhishek Kumar, Constantine Caramanis, Sanjay Shakkottai, and Wen-Sheng Chu. Rb-modulation: Training-free personalization of diffu- sion models using stochastic optimal control.arXiv preprint arXiv:2405.17401, 2024. 3, 6

    2024 arXiv

  44. [52]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22500–...

    2023

  45. [53]

    Photorealistic text-to-image diffusion models with deep language understanding.Advances in neural information processing systems, 35:36479–36494, 2022

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding.Advances in neural information ...

    2022

  46. [54]

    Structured unrestricted-rank matrices for parameter efficient fine-tuning.arXiv preprint arXiv:2406.17740, 2024

    Arijit Sehanobish, Avinava Dubey, Krzysztof Choromanski, Somnath Basu Roy Chowdhury, Deepali Jain, Vikas Sind- hwani, and Snigdha Chaturvedi. Structured unrestricted-rank matrices for parameter efficient fine-tuning.arXiv preprint arXiv:2406.17740, 2024. 2, 3, 6, 7

    2024 arXiv

  47. [55]

    Convolutional tensor- train lstm for spatio-temporal learning.Advances in Neural Information Processing Systems, 33:13714–13726, 2020

    Jiahao Su, Wonmin Byeon, Jean Kossaifi, Furong Huang, Jan Kautz, and Anima Anandkumar. Convolutional tensor- train lstm for spatio-temporal learning.Advances in Neural Information Processing Systems, 33:13714–13726, 2020. 3, 7

    2020

  48. [56]

    SD-XL inpainting 0.1 model card

    The Diffusers team. SD-XL inpainting 0.1 model card. https://huggingface.co/diffusers/stable- diffusion-xl-1.0-inpainting-0.1. 4, 1

  49. [57]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 1

    2023 arXiv

  50. [58]

    Wide compression: Tensor ring nets

    Wenqi Wang, Yifan Sun, Brian Eriksson, Wenlin Wang, and Vaneet Aggarwal. Wide compression: Tensor ring nets. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9329–9338, 2018. 3, 7

    2018

  51. [59]

    Tedigan: Text-guided diverse face image generation and ma- nipulation

    Weihao Xia, Yujiu Yang, Jing-Hao Xue, and Baoyuan Wu. Tedigan: Text-guided diverse face image generation and ma- nipulation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2256–2265,

  52. [60]

    Segformer: Simple and efficient design for semantic segmentation with transform- ers.Advances in neural information processing systems, 34: 12077–12090, 2021

    Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transform- ers.Advances in neural information processing systems, 34: 12077–12090, 2021. 7, 5

    2021

  53. [61]

    Difffit: Un- locking transferability of large diffusion models via sim- ple parameter-efficient fine-tuning

    Enze Xie, Lewei Yao, Han Shi, Zhili Liu, Daquan Zhou, Zhaoqiang Liu, Jiawei Li, and Zhenguo Li. Difffit: Un- locking transferability of large diffusion models via sim- ple parameter-efficient fine-tuning. InProceedings of the IEEE/CVF International Conference on Computer Visio...

    2023

  54. [62]

    Raphael: Text-to-image generation via large mixture of diffusion paths.Advances in Neural Information Processing Systems, 36, 2024

    Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuo- fan Zong, Yu Liu, and Ping Luo. Raphael: Text-to-image generation via large mixture of diffusion paths.Advances in Neural Information Processing Systems, 36, 2024. 2

    2024

  55. [63]

    A spectral condition for feature learning.arXiv preprint arXiv:2310.17813, 2023

    Greg Yang, James B Simon, and Jeremy Bernstein. A spectral condition for feature learning.arXiv preprint arXiv:2310.17813, 2023. 5

    2023 arXiv

  56. [64]

    Tensor- train recurrent neural networks for video classification

    Yinchong Yang, Denis Krompass, and V olker Tresp. Tensor- train recurrent neural networks for video classification. InIn- ternational Conference on Machine Learning, pages 3891–

  57. [65]

    Loretta: Low-rank economic tensor-train adaptation for ultra-low-parameter fine-tuning of large language models

    Yifan Yang, Jiajun Zhou, Ngai Wong, and Zheng Zhang. Loretta: Low-rank economic tensor-train adaptation for ultra-low-parameter fine-tuning of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lingui...

    2024

  58. [66]

    Navigating text-to-image customization: From lyCORIS fine-tuning to model evaluation

    SHIH-YING YEH, Yu-Guan Hsieh, Zhidong Gao, Bernard B W Yang, Giyeong Oh, and Yanmin Gong. Navigating text-to-image customization: From lyCORIS fine-tuning to model evaluation. InThe Twelfth International Conference on Learning Representations, 2024. 3, 6

    2024

  59. [67]

    Bridging the gap between low-rank and orthogonal adaptation via householder reflection adaptation.arXiv preprint arXiv:2405.17484,

    Shen Yuan, Haotian Liu, and Hongteng Xu. Bridging the gap between low-rank and orthogonal adaptation via householder reflection adaptation.arXiv preprint arXiv:2405.17484,

    arXiv

  60. [68]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 2, 7, 4, 5, 6

    2023

  61. [69]

    Adaptive budget allocation for parameter-efficient fine- tuning

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine- tuning. InInternational Conference on Learning Represen- tations. Openreview, 2023. 3, 6

    2023

  62. [70]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. InProceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 586–595, 2018. 7

    2018

  63. [71]

    Soda: Spectral 11 orthogonal decomposition adaptation for diffusion models

    Xinxi Zhang, Song Wen, Ligong Han, Felix Juefei-Xu, Akash Srivastava, Junzhou Huang, Vladimir Pavlovic, Hao Wang, Molei Tao, and Dimitris Metaxas. Soda: Spectral 11 orthogonal decomposition adaptation for diffusion models. InProceedings of the Winter Conference on Applications...

    2025

  64. [72]

    Galore: Memory- efficient llm training by gradient low-rank projection

    Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory- efficient llm training by gradient low-rank projection. InIn- ternational Conference on Machine Learning, pages 61121– 61143. PMLR, 2024. 7

    2024

  65. [73]

    Tensor ring decomposition.arXiv preprint arXiv:1606.05535, 2016

    Qibin Zhao, Guoxu Zhou, Shengli Xie, Liqing Zhang, and Andrzej Cichocki. Tensor ring decomposition.arXiv preprint arXiv:1606.05535, 2016. 2, 3, 4, 5, 7

    2016 arXiv

  66. [74]

    Scene parsing through ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 633–641,

  67. [75]

    Time-varying lora: Towards effec- tive cross-domain fine-tuning of diffusion models.Advances in Neural Information Processing Systems, 37:73920–73951,

    Zhan Zhuang, Yulong Zhang, Xuehao Wang, Jiangang Lu, Ying Wei, and Yu Zhang. Time-varying lora: Towards effec- tive cross-domain fine-tuning of diffusion models.Advances in Neural Information Processing Systems, 37:73920–73951,

  68. [77]

    Therefore, T[ i1 · · ·iD, j1 · · ·jD] = 0, sinceA d[id, jd,:,:] =0if id ̸=j d

    For non-diagonal elements, i.e., i1 · · ·iD ̸= j1 · · ·jD, there is at least one sub-indexi d ̸=j d. Therefore, T[ i1 · · ·iD, j1 · · ·jD] = 0, sinceA d[id, jd,:,:] =0if id ̸=j d

  69. [78]

    For diagonal elements, i.e., i1 · · ·iD = j1 · · ·jD, we havei d =j d,∀d= 1, . . . , D. Now the core ten- sors becomeA d[id, jd,:,:] =1 R×R/R,∀d= 1, . . . , D andi d, jd = 1, . . . , Id, where1 R×R is a matrix of shapeR×Rwith all elements being one. Therefore, T[ i1 · · ·iD, j...

    2017

  70. [2024]

    Details and more results in Sec

    3, 6 12 Transformed Low-rank Adaptation via Tensor Decomposition and Its Applications to Text-to-image Models Supplementary Material Contents A . Details and more results in Sec. 3.2 1 B . Proposed model 1 B.1. Expressiveness of TRM . . . . . . . . . . . 1 B.2. Proof of Props....

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.