Pith. sign in

REVIEW 4 major objections 6 minor 51 references

ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraints

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CNTLoRA derives LoRA initialization in closed form from fine-tuning activations and pre-trained weights, without any training, and reports faster convergence and higher scores across image generation, classification, and understanding…

desk verdict Genuinely new initialization idea with broad experiments, but the constraint-based derivation is mathematically infeasible as written; still worth a serious referee. read the letter →

arxiv 2507.08044 v1 pith:MBCF2YU6 submitted 2025-07-09 cs.CV cs.AI

Debasmit Das , Hyoungwoo Park , Munawar Hayat , Seokeon Choi , Sungrack Yun , Fatih Porikli This is my paper
classification cs.CVcs.AI
keywords LoRAinitializationlow-rankadaptersparameter-efficientfine-tuningdomainshiftclosed-formsolutionvariableadapterstructuredata-driven
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Low-rank adapters (LoRA) are almost always initialized with random matrices of a fixed rank. This paper argues that the initialization can instead be computed in closed form, with no training, by treating fine-tuning as a domain shift between pre-training and fine-tuning activations. It proposes three constraint sets relating source and target activations and pre-trained weights, each yielding a direct formula for the target weight estimate. The estimate is then decomposed to initialize the adapter's up and down matrices, with an optional variable-rank allocation across layers. Across image generation, classification, and understanding benchmarks, the authors report faster convergence and higher final scores than random, SVD-based, and activation-based initialization baselines.

What carries the argument

The central object is a constraint set over four quantities per LoRA layer: source and target input activations and source and target weights. Because source activations are unavailable during fine-tuning, the paper imposes 'rough' statistical assumptions (e.g., $X^i_{src}X^{iT}_{tar}=I$ in Cross mode, or $X^i_{src}X^{iT}_{src}=I$ in Self and Shift modes) to eliminate them, leaving a closed-form least-squares estimate like $W^i_{tar}=W^i_{src}(X^i_{tar}X^{iT}_{tar})^\dagger$. This pseudo-inverse formula, together with the SVD or QR decomposition of $\Delta W^i_{est}=W^i_{src}-W^i_{tar}$, is what carries the argument: no gradient step is needed before fine-tuning begins.

What would settle it

Compute the actual source activation matrices from the pre-training data for a representative layer and measure the whitening error $\|X^i_{src}X^{iT}_{src}-I\|_F$ (or the cross-correlation error $\|X^i_{src}X^{iT}_{tar}-I\|_F$). If these errors are large on the very models and layers where CNTLoRA is applied, the closed-form target weights are built on a false premise and the reported gains would not transfer to layers or tasks where the assumptions fail.

Watch

Extended reading notes

Core claim

The paper claims that a good LoRA initialization can be derived as a closed-form function of the pre-trained weight matrix $W^i_{src}$ and fine-tuning input activations $X^i_{tar}$, with no training during initialization. The derivation solves for an estimated target weight $W^i_{tar}$ by imposing two constraints per mode: cross-domain output matching plus input cross-correlation (Cross mode), output covariance matching plus source whitening (Self mode), or a constant domain-shift plus source whitening (Shift mode). Solving these gives $W^i_{tar}$ via a pseudo-inverse, e.g. $W^i_{tar}=W^i_{src}(X^i_{tar}X^{iT}_{tar})^\dagger$ in Cross mode. The difference $\Delta W^i_{est}$ is decomposed (SVD or QR) to set the down and up matrices, so both are non-zero and data-dependent. The authors further propose allocating different ranks to different attachment points based on relative singular-value variance, and report that CNTLoRA variants outperform prior initialization methods on Dreambooth, VTAB-1K, and image understanding benchmarks.

Load-bearing premise

The pre-training activations satisfy the assumed constraints (such as being whitened or cross-correlated with fine-tuning activations to the identity) even though those activations are never observed during fine-tuning.

Editorial extensions

If this is right

  • CNTLoRA removes the need for any training or gradient computation during initialization, so fine-tuning starts from a point closer to the final solution.
  • Both up and down adapter matrices are non-zero and data-dependent, so gradients for both matrices are active from the first step, unlike zero-init baselines.
  • The variable adapter structure allocates ranks per attachment point from singular-value variance, saving memory while improving or matching fixed-rank performance.
  • The method is model-agnostic: it improves DINO scores in few-shot Dreambooth fine-tuning and raises average accuracy on VTAB-1K for DINOv2.
  • The same closed-form estimate can be combined with other adapter variants such as DoRA and rsLoRA to improve their performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the constraints hold only approximately, the initialization is still a warm start; the more interesting question is how sensitive the closed-form estimate is to the constant $C$ in Shift mode, which the paper only explores for image generation.
  • Because the method uses only fine-tuning activations, it could be extended to online or continually changing target distributions, re-initializing adapters as new data arrives without retraining.
  • The variable-rank allocation via relative singular values suggests a data-driven way to prune adapter capacity; a testable extension is whether fine-tuning with the allocated ranks retains the same final performance as fixed high rank.
  • The 'Self mode' solution is one arbitrary root of a quadratic matrix equation; a direct comparison against other roots could determine whether the chosen root is essential or whether any root works.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes ConsNoTrainLoRA (CNTLoRA), a data-driven initialization scheme for LoRA adapters. Three modes (Cross, Self, Shift) define constraint sets relating source (pre-training) and target (fine-tuning) activations and weights, and the authors derive closed-form estimates of the target weights from these constraints. The estimated weight update is decomposed into LoRA up/down matrices, optionally with a variable-rank allocation (VAS). Experiments cover image generation (Dreambooth), image classification (VTAB-1K), image understanding (APD and myVLM), and language understanding (GLUE), reporting faster convergence and higher final scores than random LoRA, EVA, CORDA, LoRA-GA, PiSSA, OLoRA, and related baselines. The central theoretical claim is that the initialization requires no training and that the closed-form estimate is obtained by reformulating the stated constraints.

Significance. The empirical scope is a strength: the method is tested across four task families, with ablations of rank, learning rate, number of initialization samples, SVD/QR decomposition, and variable-rank allocation. If the gains are reproducible, the initializer is cheap and practically useful, and VAS provides a sensible memory-saving extension. However, the claimed theoretical foundation is not sound as written. The constraints used to derive the closed-form estimates are infeasible under the stated dimensions (b < d), the Self-mode solution is one arbitrary choice among infinitely many, and the Shift-mode pseudo-inverse formula is a least-norm selection rather than a consequence of the constraints. The empirical claim is further weakened by the absence of error bars, significance tests, and released code. The significance is therefore conditional on a substantially reworked derivation and additional validation.

major comments (4)
  1. [Section 3.2] The constraints X_src X_tar^T = I (Eq. 2) and X_src X_src^T = I (Eqs. 7 and 15) are infeasible for the dimensions used in the paper. With X_src, X_tar in R^{d x b} and b < d (batch size 1 for image generation, 8 for image understanding, and small batches for classification), these products are d x d matrices of rank at most b, whereas I_d has rank d. The minimum Frobenius distance from a rank-b matrix to I_d is at least sqrt(d - b), which is large in all reported settings. Consequently, Eq. (4) is not a solution of the system (1)-(2), and Eqs. (12) and (17) do not follow from their respective constraint sets. The 'rough assumptions' caveat in Section 3.2 does not resolve this issue, because the failure is not a small perturbation but an algebraic impossibility. The pseudo-inverse formulas instead solve a different least-squares problem (e.g., approximating W_src by W_tar X_tar X_tar^T), so the central claim that the closed-form estimate is 'obtained' by reformulating the constraints is not supported as written.
  2. [Section 3.2] In Self mode, Eq. (8) is a quadratic matrix equation with infinitely many solutions for W_tar. The regrouping in Eq. (11), which matches W_src with W_tar P D^{0.5}, selects one arbitrary solution, and the resulting Eq. (12) is a least-norm choice rather than a consequence of the constraints. No argument is given that this solution is the correct target weight or that the other solutions are equivalent. Moreover, for k > b the left-hand side W_src W_src^T has rank up to min(k, d) while the right-hand side W_tar X_tar X_tar^T W_tar^T has rank at most b, so Eq. (8) is also infeasible in the reported settings. As written, Eq. (12) is a heuristic projection rule whose connection to the stated covariance-matching constraint is not established.
  3. [Section 3.2] The Shift mode constraint (14) is introduced without a statistical model that connects it to the domain-shift picture; in particular, the asymmetric placement of W_src and W_tar in (14) is not motivated. After substituting (15), the equation W_tar (W_src^T - X_tar X_tar^T W_src^T) = C leaves W_tar underdetermined by the null space of the right multiplier, so the pseudo-inverse formula (17) is a least-norm selection rather than a derived solution. The role of C is also unclear: it is treated as a scalar hyperparameter in the experiments (e.g., '10 times identity'), but in Eq. (14) it must be a matrix on the output space, and this dimensional interpretation is never stated.
  4. [Section 4.2] The empirical claim that CNTLoRA 'consistently outperforms' baselines is supported only by point estimates. Table 5 reports an average over five seeds but no standard deviation or per-seed breakdown, while Tables 1, 6, and 10 report single runs. Several differences are small (e.g., Table 6: CNTLoRA-S 0.8256 vs CNTLoRA-Sh 0.8273 on APD; Table 1: CNTLoRA-Sh 62.95 vs DoRA 62.98 on DINO), and without error bars or significance tests these cannot be distinguished from noise. No code release is indicated, which further limits reproducibility. The convergence curves (Figs. 5-7, 16-17) are informative but would benefit from error bands.
minor comments (6)
  1. [Section 3.2] Equations (5) and (13) list 'F1(...) = 0' and then give the expression in the following display; please use a single notational convention (e.g., remove the F1=0 lines or number the expressions directly).
  2. [Table 1] Table 1 contains a formatting error in the PiSSA row ('74.4127.98'), and Table 2's column header 'OutQuery' appears to have lost its whitespace; please fix these issues.
  3. [Section 3.3, Eq. (19)] The definition of the normalizing constant in Eq. (19) is unclear: 'the proportion constant is obtained by summing over all the singular values and dividing by the sum' reads as circular; please state explicitly that the relative variance is the squared singular value normalized by the total sum of squared singular values.
  4. [Section 3.3] The noise schedule notation x_t = sqrt(alpha_t) z_t + sqrt(1 - alpha_t) epsilon should be checked against the standard DDPM convention, where the coefficient of z_t is typically the square root of the cumulative product of the noise schedule; the current notation may confuse readers.
  5. [Section 3.2, Eq. (18)] The batch-averaging formula divides by B+1 because W_0 is included; please explain this choice and specify how the weights w_j are selected, since the default w_j=1 is mentioned only in the text.
  6. [General] The paper does not state whether code will be released; for a data-driven initialization method, code release would materially aid reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the closed-form LoRA initialization is derived algebraically from explicitly stated constraints and is evaluated on held-out fine-tuning metrics; the constraints are admittedly rough assumptions, which is a correctness concern rather than circular reasoning.

full rationale

The derivation chain is self-contained in the algebraic sense. In Cross mode, Eqs. (1)-(2) are combined by multiplying Eq. (1) by X_tar^T and substituting the constraint X_src X_tar^T = I to obtain Eq. (3), from which the least-squares pseudo-inverse estimate Eq. (4) follows. Self mode and Shift mode follow the same pattern: the constraints (6)-(7) and (14)-(15) are manipulated algebraically to produce the closed-form estimates (12) and (17). Each estimate depends only on the pre-trained weight W_src and the fine-tuning activation X_tar, which are explicit inputs; the evaluation is then performed after actual fine-tuning on held-out metrics (DINO, CLIP, VTAB-1K accuracy, SentSim), so no 'prediction' is merely a renamed fit. The paper explicitly acknowledges that the constraints are 'rough assumptions just to provide a good initialization' (Section 3.2), and the pseudo-inverse steps solve least-squares versions of the derived equations. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in via citation. The skeptic's point that X_src X_tar^T = I is infeasible when b < d is a legitimate mathematical correctness risk about whether the constraints approximately hold, but it is not a circularity: the derivation does not assume the conclusion. Therefore the circularity score is 0.

Assumptions & free parameters 7 free parameters · 7 assumptions · 0 invented entities

The central derivation rests on a small set of rough constraints about source and target activations, plus several tunable hyperparameters: mode, C, p, rank, VAS budget, batch weights, and input representation. These constraints are labeled assumptions in the paper and are not derived from external benchmarks. No invented entities are introduced. The key gap is that the quadratic equation in Self mode is resolved by grouping factors with no proof of uniqueness or optimality.

free parameters (7)
  • Constraint mode (Cross/Self/Shift) = None; all three evaluated
    The paper proposes three constraint sets and selects among them after experiments; the choice is not derived from theory.
  • Shift constant C = identity (default); also Normal, 10I, 0.1I in Table 9
    Introduced in Eq. (14) as a hyper-parameter; no principled criterion for setting it.
  • Fractional singular value allocation p = 0.5 default; 0.1, 0.25, 0.75 tested
    Controls how much of the singular matrix goes to A versus B in decomposition (Section 3.2, Table 9).
  • Rank per attachment point r = 128 (SD), 32 (APD), 128 (myVLM), etc.
    Standard LoRA rank chosen per task; not derived.
  • VAS rank budget K = rank times number of attachment points
    Defines total rank budget in variable adapter structure (Section 3.3).
  • Batch averaging weight w_j = 1
    Used in Eq. (18) to average estimates over batches; default is uniform.
  • UNet initialization input (noisy vs clean latent) = clean latent z_t (default)
    Section 3.3 special case for text-to-image; the choice changes the activation statistics used.
assumptions (7)
  • domain assumption Fine-tuning updates have low intrinsic rank
    Standard LoRA premise, cited to Aghajanyan et al.; used in Section 3.1.
  • ad hoc to paper Source and target output activations are equal (Cross mode)
    Eq. (1), explicitly called a rough assumption.
  • ad hoc to paper Source and target input activations satisfy X_src X_tar^T = I (Cross mode)
    Eq. (2); strong orthogonality/correlation assumption not derived from data.
  • ad hoc to paper Source input activations are whitened
    Eqs. (7) and (15); used in Self and Shift modes.
  • ad hoc to paper Source and target output covariance matrices are equal (Self mode)
    Eq. (6); second-order matching assumption.
  • ad hoc to paper Domain shift between source and target features is a constant C (Shift mode)
    Eq. (14); the constant is a hyper-parameter, default identity.
  • ad hoc to paper A specific solution of the quadratic equation W_src W_src^T = W_tar A A^T W_tar^T is obtained by grouping factors
    Eqs. (8)-(12); the paper notes multiple solutions and chooses one without a uniqueness or optimality proof.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraints." pith.science (2026). https://pith.science/paper/MBCF2YU6

@misc{pith2026250708044,
  author       = {Pith},
  title        = {Pith review of: ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraints},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MBCF2YU6}},
  note         = {Machine review of arXiv:2507.08044}
}
read the original abstract

Foundation models are pre-trained on large-scale datasets and subsequently fine-tuned on small-scale datasets using parameter-efficient fine-tuning (PEFT) techniques like low-rank adapters (LoRA). In most previous works, LoRA weight matrices are randomly initialized with a fixed rank across all attachment points. In this paper, we improve convergence and final performance of LoRA fine-tuning, using our proposed data-driven weight initialization method, ConsNoTrainLoRA (CNTLoRA). We express LoRA initialization as a domain shift problem where we use multiple constraints relating the pre-training and fine-tuning activations. By reformulating these constraints, we obtain a closed-form estimate of LoRA weights that depends on pre-training weights and fine-tuning activation vectors and hence requires no training during initialization. This weight estimate is decomposed to initialize the up and down matrices with proposed flexibility of variable ranks. With the proposed initialization method, we fine-tune on downstream tasks such as image generation, image classification and image understanding. Both quantitative and qualitative results demonstrate that CNTLoRA outperforms standard and data-driven weight initialization methods. Extensive analyses and ablations further elucidate the design choices of our framework, providing an optimal recipe for faster convergence and enhanced performance.

Figures

Figures reproduced from arXiv: 2507.08044 by the authors.

Figure 1
Figure 1. The DINO score plot of different initialization methods [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Generation evolution for different initialization methods on Dreambooth [ [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Visualization of our proposed framework. The input samples are fed to the network and the activations before the LoRA layer [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Visualization of how variable ranking is done to obtain [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Plot showing how the CLIP score evolves for different [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Evolution of loss & evaluation accuracy with epochs on [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Figure showing how image captions evolve with increas [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 7
Figure 7. Figure 7: Progression of evaluation metrics over epochs on [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 9
Figure 9. Figure 9: Figure showing the input prompts used in the myVLM [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: Plot showing how the rank distribution takes place for the dog class in the Dreambooth dataset. [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: Plot showing how the generation evolves for different initialization methods on the Dreambooth dataset for the duck class. The [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: Plot showing how the generation evolves for different initialization methods on the Dreambooth dataset for the teapot class. The [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]
Figure 13
Figure 13. Figure 13: Plot showing how the generation evolves for different initialization methods on the Dreambooth dataset for the dog class. The [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 14
Figure 14. Figure 14: Figure showing how image captions evolve with in [PITH_FULL_IMAGE:figures/full_fig_p015_14.png]
Figure 15
Figure 15. Figure 15: Plot showing how the rank distribution takes place when fine-tuned on the myVLM dataset. [PITH_FULL_IMAGE:figures/full_fig_p016_15.png]
Figure 16
Figure 16. Figure 16: Plots showing how the training loss varies with different epochs for different initialization methods on the MRPC and RTE [PITH_FULL_IMAGE:figures/full_fig_p017_16.png]
Figure 17
Figure 17. Figure 17: Plots showing how the accuracy varies with different epochs for different initialization methods on the MRPC and RTE datasets. [PITH_FULL_IMAGE:figures/full_fig_p017_17.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

51 extracted references · 20 canonical work pages

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    arXiv

  2. [2]

    In- trinsic dimensionality explains the effectiveness of language model fine-tuning.arXiv preprint arXiv:2012.13255, 2020

    Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. In- trinsic dimensionality explains the effectiveness of language model fine-tuning.arXiv preprint arXiv:2012.13255, 2020. 3

    arXiv 2012

  3. [3]

    Myvlm: Personalizing vlms for user-specific queries

    Yotam Alaluf, Elad Richardson, Sergey Tulyakov, Kfir Aber- man, and Daniel Cohen-Or. Myvlm: Personalizing vlms for user-specific queries. InEuropean Conference on Computer Vision (ECCV), pages 73–91. Springer, 2024. 5, 11

    work page 2024

  4. [4]

    Slora: Federated param- eter efficient fine-tuning of language models.arXiv preprint arXiv:2308.06522, 2023

    Sara Babakniya, Ahmed Roushdy Elkordy, Yahya H Ezzeldin, Qingfeng Liu, Kee-Bong Song, Mostafa El- Khamy, and Salman Avestimehr. Slora: Federated param- eter efficient fine-tuning of language models.arXiv preprint arXiv:2308.06522, 2023. 2, 3

    arXiv 2023

  5. [5]

    Lora-xs: Low-rank adaptation with ex- tremely small number of parameters.arXiv preprint arXiv:2405.17604, 2024

    Klaudia Bałazy, Mohammadreza Banaei, Karl Aberer, and Jacek Tabor. Lora-xs: Low-rank adaptation with ex- tremely small number of parameters.arXiv preprint arXiv:2405.17604, 2024. 3

    arXiv 2024

  6. [6]

    On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258, 2021

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Alt- man, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258, 2021. 1

    arXiv 2021

  7. [7]

    Rt-1: Robotics transformer for real-world control at scale

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakr- ishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022. 1

    arXiv 2022

  8. [8]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control.arXiv preprint arXiv:2307.15818, 2023

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control.arXiv preprint arXiv:2307.15818, 2023. 1

    arXiv 2023

Show all 51 references
  1. [9]

    Olora: Orthonormal low-rank adaptation of large language models.arXiv preprint arXiv:2406.01775, 2024

    Kerim B ¨uy¨ukaky¨uz. Olora: Orthonormal low-rank adaptation of large language models.arXiv preprint arXiv:2406.01775, 2024. 1, 3

    2024 arXiv

  2. [10]

    One-for-all: Generalized lora for parameter- efficient fine-tuning.arXiv preprint arXiv:2306.07967,

    Arnav Chavan, Zhuang Liu, Deepak Gupta, Eric Xing, and Zhiqiang Shen. One-for-all: Generalized lora for parameter- efficient fine-tuning.arXiv preprint arXiv:2306.07967,

    arXiv

  3. [11]

    Scaling vision transformers to 22 billion pa- rameters

    Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdul- mohsin, et al. Scaling vision transformers to 22 billion pa- rameters. InInternational Conference on Machine ...

    2023

  4. [12]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale.Advances in Neural Information Pro- cessing Systems, 35:30318–30332, 2022. 3

    2022

  5. [13]

    Qlora: Efficient finetuning of quantized llms

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36,

  6. [14]

    Domain-adversarial train- ing of neural networks.Journal of Machine Learning Re- search, 17(59):1–35, 2016

    Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pas- cal Germain, Hugo Larochelle, Francois Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial train- ing of neural networks.Journal of Machine Learning Re- search, 17(59):1–35, 2016. 2

    2016

  7. [15]

    Understanding the diffi- culty of training deep feedforward neural networks

    Xavier Glorot and Yoshua Bengio. Understanding the diffi- culty of training deep feedforward neural networks. InPro- ceedings of the thirteenth international conference on artifi- cial intelligence and statistics, pages 249–256. JMLR Work- shop and Conference Proceedings, 2010. 3

    2010

  8. [16]

    Lora+: Effi- cient low rank adaptation of large models.arXiv preprint arXiv:2402.12354, 2024

    Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Effi- cient low rank adaptation of large models.arXiv preprint arXiv:2402.12354, 2024. 2

    2024 arXiv

  9. [17]

    Delving deep into rectifiers: Surpassing human-level perfor- mance on imagenet classification

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level perfor- mance on imagenet classification. InProceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015. 3

    2015

  10. [18]

    Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021. 1, 2, 3, 5

    2021 arXiv

  11. [19]

    A rank stabilization scaling factor for fine-tuning with lora.arXiv preprint arXiv:2312.03732,

    Damjan Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora.arXiv preprint arXiv:2312.03732,

    arXiv

  12. [20]

    Elora: Efficient low-rank adaptation with random matrices

    Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. Elora: Efficient low-rank adaptation with random matrices. InThe Twelfth International Conference on Learn- ing Representations, 2024. 2, 3

    2024

  13. [21]

    Loftq: Lora- fine-tuning-aware quantization for large language models

    Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, Nikos Karampatziakis, Weizhu Chen, and Tuo Zhao. Loftq: Lora- fine-tuning-aware quantization for large language models. arXiv preprint arXiv:2310.08659, 2023. 2

    2023 arXiv

  14. [22]

    Few- shot parameter-efficient fine-tuning is better and cheaper than in-context learning.Advances in Neural Information Processing Systems, 35:1950–1965, 2022

    Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few- shot parameter-efficient fine-tuning is better and cheaper than in-context learning.Advances in Neural Information Processing Systems, 35:1950–1965, 2022. 3

    1950

  15. [23]

    Visual instruction tuning.arXiv preprint arXiv:2304.08485,

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.arXiv preprint arXiv:2304.08485,

    arXiv

  16. [24]

    Dora: Weight-decomposed low-rank adaptation.arXiv preprint arXiv:2402.09353, 2024

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation.arXiv preprint arXiv:2402.09353, 2024. 2

    2024 arXiv

  17. [25]

    Roberta: A robustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692,

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettle- moyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692,

    1907 arXiv

  18. [26]

    Alora: Allocating low-rank adaptation for fine-tuning large language models.arXiv preprint arXiv:2403.16187, 2024

    Zequan Liu, Jiawen Lyn, Wei Zhu, Xing Tian, and Yvette Graham. Alora: Allocating low-rank adaptation for fine-tuning large language models.arXiv preprint arXiv:2403.16187, 2024. 2

    2024 arXiv

  19. [27]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017. 5

    2017 arXiv

  20. [28]

    A survey on lora of large language models.Frontiers of Computer Science, 19(1): 197605, 2025

    Yuren Mao, Yuhang Ge, Yijiang Fan, Wenyi Xu, Yu Mi, Zhonghao Hu, and Yunjun Gao. A survey on lora of large language models.Frontiers of Computer Science, 19(1): 197605, 2025. 1

    2025

  21. [29]

    Pissa: Prin- cipal singular values and singular vectors adaptation of large language models.arXiv preprint arXiv:2404.02948, 2024

    Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Prin- cipal singular values and singular vectors adaptation of large language models.arXiv preprint arXiv:2404.02948, 2024. 1, 3

    2024 arXiv

  22. [30]

    E. H. Moore. On the reciprocal of the general algebraic ma- trix.Bulletin of the American Mathematical Society, 26:394– 395, 1920. 4

    1920

  23. [31]

    Rosa: Accurate parameter-efficient fine-tuning via ro- bust adaptation.arXiv preprint arXiv:2401.04679, 2024

    Mahdi Nikdan, Soroush Tabesh, Elvir Crnˇcevi´c, and Dan Al- istarh. Rosa: Accurate parameter-efficient fine-tuning via ro- bust adaptation.arXiv preprint arXiv:2401.04679, 2024. 2, 3

    2024 arXiv

  24. [32]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 1, 5, 11

    2023 arXiv

  25. [33]

    One initialization to rule them all: Fine- tuning via explained variance adaptation.arXiv preprint arXiv:2410.07170, 2024

    Fabian Paischer, Lukas Hauzenberger, Thomas Schmied, Benedikt Alkin, Marc Peter Deisenroth, and Sepp Hochre- iter. One initialization to rule them all: Fine- tuning via explained variance adaptation.arXiv preprint arXiv:2410.07170, 2024. 1, 3, 5, 11

  26. [34]

    Learn- ing transferable visual models from natural language super- vision.arXiv preprint arXiv:2103.00020, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision.arXiv preprint arXiv:2103.00020, 2021. 5

    2021 arXiv

  27. [35]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. InProceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing. Association for Computational Linguis- tics, 2019. 5

    2019

  28. [36]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 5

    2022

  29. [37]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22500–...

    2023

  30. [38]

    Amazon product descriptions vlm, 2024

    Philipp Schmid. Amazon product descriptions vlm, 2024. Accessed: 2025-02-14. 5, 11

    2024

  31. [39]

    Deep coral: Correlation alignment for deep domain adaptation

    Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. InEuropean Confer- ence on Computer Vision (ECCV), pages 443–450. Springer,

  32. [40]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text.arXiv preprint arXiv:2403.05530, 2024

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text.arXiv preprint arXiv:2403.05530, 2024. 1

    2024 arXiv

  33. [41]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 1

    2023 arXiv

  34. [42]

    Deep domain confusion: Maximizing for domain invariance

    Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Work- shops, pages 1–9, 2014. 2

    2014

  35. [43]

    Adversarial discriminative domain adaptation

    Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Dar- rell. Adversarial discriminative domain adaptation. InPro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7167–7176, 2017. 2

    2017

  36. [44]

    Dylora: Parameter efficient tuning of pre- trained models using dynamic search-free low-rank adapta- tion.arXiv preprint arXiv:2210.07558, 2022

    Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter efficient tuning of pre- trained models using dynamic search-free low-rank adapta- tion.arXiv preprint arXiv:2210.07558, 2022. 3

    2022 arXiv

  37. [45]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    A Wang, A Singh, J Michael, F Hill, O Levy, and SR Bow- man. Glue: A multi-task benchmark and analysis platform for natural language understanding. arxiv preprint arxiv: 180407461, 2018. 11

    2018

  38. [46]

    Lora-ga: Low- rank adaptation with gradient approximation.arXiv preprint arXiv:2407.05000, 2024

    Shaowen Wang, Linxi Yu, and Jian Li. Lora-ga: Low- rank adaptation with gradient approximation.arXiv preprint arXiv:2407.05000, 2024. 2, 3

    2024 arXiv

  39. [47]

    Corda: Context-oriented decomposition adaptation of large language models for task-aware parameter-efficient fine- tuning

    Yibo Yang, Xiaojie Li, Zhongzhu Zhou, Shuaiwen Leon Song, Jianlong Wu, Liqiang Nie, and Bernard Ghanem. Corda: Context-oriented decomposition adaptation of large language models for task-aware parameter-efficient fine- tuning. InThe Thirty-eighth Annual Conference on Neural In...

  40. [48]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools.CoRR, 2024

    Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools.CoRR, 2024. 5, 11

    2024

  41. [49]

    A large-scale study of representation learning with the visual task adaptation benchmark.arXiv preprint arXiv:1910.04867, 2019

    Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djo- longa, Andre Susano Pinto, Maxim Neumann, Alexey Doso- vitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark.arXiv p...

    1910 arXiv

  42. [50]

    Adalora: Adaptive budget alloca- tion for parameter-efficient fine-tuning.arXiv preprint arXiv:2303.10512, 2023

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget alloca- tion for parameter-efficient fine-tuning.arXiv preprint arXiv:2303.10512, 2023. 2

    2023 arXiv

  43. [51]

    A S* toy floating on top of water

    Bojia Zi, Xianbiao Qi, Lingzhi Wang, Jianan Wang, Kam-Fai Wong, and Lei Zhang. Delta-lora: Fine-tuning high-rank pa- rameters with the delta of low-rank matrices.arXiv preprint arXiv:2309.02411, 2023. 2 A. Additional Experimental Details Image GenerationFor the image generatio...

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.