Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that LoRA's zero-initialization convention is unnecessary and that starting both low-rank matrices at comparable non-zero values improves robustness to small learning rates.

desk verdict The practical claim about non-zero LoRA initialization is credible and well-tested, but the scaling-theory derivation has correctable gamma-notation errors that need fixing before the theory can be cited as support. read the letter →

arxiv 2505.23194 v2 pith:WUWYRPQP submitted 2025-05-29 cs.LG

Shiwei Li , Xiandi Luo , Xing Tang , Haozhao Wang , Hao Chen , Weihong Luo , Yuhua Li , Xiuqiang He
show 1 more author
Ruixuan Li
This is my paper
classification cs.LG
keywords LoRAlow-rankadaptationparameter-efficientfine-tuningnon-zeroinitializationlearningraterobustnessinfinite-widthanalysisAdamoptimizerdynamics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that the standard LoRA convention of initializing one of the two low-rank matrices to zero is not necessary and is often suboptimal. The central claim is that starting both matrices A and B at non-zero values with comparable variance makes LoRA fine-tuning more robust to suboptimal learning rates, especially small ones, and reaches equal or better accuracy than zero initialization across the tested models and tasks. If true, this removes a long-standing design constraint: fine-tuning need not start exactly at the pretrained model, and a zero-cost change to initialization can improve convergence under learning-rate decay.

What carries the argument

The load-bearing object is the $\gamma$-operator from neural-network scaling theory, which records the width exponent of a quantity ($v=\Theta(n^{\gamma[v]})$) and reduces stability, efficiency, and robustness of LoRA to three exponent equations. The paper also formalizes LoRA features $Z_A=AZ$ and $Z_B=BAZ$, defines stability as $Z_B=\Theta(1)$, efficiency as both linear update terms $\delta_1$ and $\delta_2$ being $\Theta(1)$, and robustness as $\delta_1$ being independent of $\eta_B$, $\delta_2$ independent of $\eta_A$, and $Z_B$ independent of both. Theorem 1 then identifies $\gamma[A_0]=\gamma[B_0]=-1/2$ as the initialization that achieves all three simultaneously, so the paper's proposed scheme, Init[AB], treats $AB$ as a single Kaiming-initialized matrix rather than initializing A or B alone.

What would settle it

Repeat the paper's T5-Base QNLI experiment at $\eta=3\times 10^{-7}$ with $\beta=16$, comparing Init[A] and Init[AB] over the same three seeds: if Init[AB] does not match or exceed Init[A] accuracy, the robustness claim fails in its headline setting. A more direct check is to measure $\gamma[\delta_1]$ in the paper's infinite-width toy at width $n=4096$ with $\gamma[A_0]=\gamma[B_0]=-1/2$ and verify that it tracks $\gamma[\eta_A]+1/2$ across $\gamma[\eta_A]\le -1/2$; any deviation falsifies the exponent law on which the theory rests.

Watch

Extended reading notes

Core claim

Analyzing LoRA training from an infinite-width scaling perspective, the paper derives necessary exponents for stability and efficiency: with Adam and a uniform learning rate, $\gamma[\eta_A]=\gamma[\eta_B]=-1/2$ and $\gamma[A_0],\gamma[B_0]\le -1/2$ are required. Under the standard Init[A], the inequality is satisfied but the process is sensitive to shrinking the learning rate because the update sizes $\delta_1$ and $\delta_2$ depend on the other matrix's learning rate. The paper's main theoretical result is that setting both $A_0$ and $B_0$ to $\Theta(n^{-1/2})$ (so $\sigma_A^2=\sigma_B^2$) makes $\delta_1$ depend only on $\eta_A$, $\delta_2$ only on $\eta_B$, and leaves $Z_B=BAZ$ at $\Theta(1)$ for every learning rate exponent $\gamma[\eta]\le -1/2$, which it defines as optimal robustness. The paper then verifies empirically that this same-variance non-zero initialization matches or improves accuracy on GLUE with T5-Base and on commonsense and arithmetic reasoning with Llama 3-8B, with the largest gains at small learning rates.

Load-bearing premise

The theory is derived for one LoRA layer, one data point, and infinite width, with Adam-normalized gradients assumed to stay at order $\Theta(1)$; the paper then transfers these conclusions to multi-layer transformer models trained on batches at finite width, even though the mini-batch extension is acknowledged to need assumptions that are never stated.

Editorial extensions

If this is right

  • Fine-tuning can start slightly away from the pretrained model, so the pretrained weights do not need to be modified in multi-LoRA serving scenarios.
  • Init[AB] is a zero-cost change to standard LoRA that improves accuracy at small learning rates and, in the QNLI experiment, roughly doubles convergence speed.
  • Standard LoRA initialization (A Kaiming, B zero) is just one point in a feasible initialization space and is not the point with the best learning-rate robustness.
  • Existing non-zero initialization methods such as LoftQ, PiSSA, and LoRA-GA receive a theoretical justification for why their starting points help.
  • The robustness benefit also extends to LoRA+ with decoupled learning rates, as shown in the appendix experiments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the exponent argument should transfer to any Adam-trained low-rank adapter whose gradient normalization keeps coordinate-wise updates at $\Theta(1)$, so a testable extension is sweeping the initialization scale $\beta$ on vision or diffusion models.
  • Editorial inference: the broad safe range of initialization variances suggests the non-zero start behaves like a warm start or mild implicit regularization, which would predict that the acceptable noise level shrinks as the total fine-tuning step budget decreases.
  • Editorial inference: the same robustness logic implies that learning-rate decay schedules could be made more aggressive under Init[AB] without losing stability, a consequence the paper does not directly test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper studies whether the standard zero-initialization of LoRA matrices is necessary, using an infinite-width scaling analysis with the Adam optimizer. The authors define stability, efficiency, and robustness for LoRA feature learning, derive conditions of the form γ[ηA]+γ[ηB]=−1 and γ[A0], γ[B0]≤−1/2, and conclude that initializing both A and B with equal non-zero variance (Init[AB]) improves robustness to suboptimal, especially smaller, learning rates. They further argue that the noise introduced by non-zero initialization does not hurt fine-tuning accuracy, so fine-tuning need not start exactly from the pretrained model. The claims are supported by experiments on a toy model, T5-Base on GLUE tasks, and Llama 3-8B on commonsense and arithmetic reasoning benchmarks, with code released.

Significance. If the empirical finding holds, the paper identifies a zero-cost modification of LoRA initialization that consistently improves accuracy at small learning rates and broadens learning-rate robustness, which would be practically valuable. The experimental work is a clear strength: it covers multiple model families and tasks, reports consistent gains, includes ablations on the initialization scale and on the subtraction of A0B0, and releases code. The theoretical framework extends prior scaling analyses of LoRA and, if repaired, could provide a useful explanation of the observed effect. However, the theoretical derivation as written contains internally inconsistent uses of the γ-operator, to the point that Theorem 1 cannot be verified from the displayed equations. The practical recommendation may survive a corrected derivation, but the paper's central 'theoretical support' claim needs substantial revision.

major comments (3)
  1. [§2.1, §3.2, Eq. (4)–(6), §4.1] The γ-operator is defined on asymptotic magnitudes (v=Θ(n^{γ[v]})), and Lemmas 1–2 treat γ[U] as an entry-magnitude exponent. Under this convention, Kaiming initialization with variance σ_A²=1/n gives entries of magnitude Θ(n^{-1/2}), hence γ[A0]=−1/2, not −1. The paper nevertheless states γ[A0]=−1 for Init[A] in Section 3.2 and again in Section 4.1, and uses this value inside the max expressions of Eq. (4). If γ[A0]=−1 is interpreted as an entry-magnitude exponent, the experimental protocol σ_A²=1/n does not match the theory; if it is interpreted as a variance exponent, the product and sum rules used to derive Eq. (4) do not apply. A consistent convention for γ is required before Eq. (6) and Theorem 1 can be verified.
  2. [§3.3, after Eq. (6)] After setting γ[A0]=γ[B0]=−1/2, the paper states γ[ZB]=γ[A0]+γ[B0]=1. Under Lemma 2, γ[ZB]=γ[A0]+γ[B0]+γ[Z]+1=0 (since A∈R^{r×n} multiplies Z∈R^n, producing a sum over n), and Definition 3 requires γ[ZB]=0. The displayed identity is also arithmetically inconsistent because −1/2 + −1/2 = −1. The following sentence, which says that γ[ZB]=1 is 'unaffected, thereby ensuring consistent stability,' is therefore incorrect. This step is load-bearing for the claim that stability is preserved under all smaller learning rates, so the derivation of Theorem 1 must be corrected.
  3. [Footnote 3, §3.1, §4] The theoretical setup is restricted to a single data point, a single LoRA layer, and infinite width, with Adam-normalized gradients assumed to be Θ(1). Footnote 3 acknowledges that extending the analysis to mini-batch gradients 'requires additional assumptions' but does not provide them. Section 4 then applies the predictions to T5-Base and Llama 3-8B trained on batches with many LoRA layers and finite width, presenting the results as confirming the theory. Without the missing assumptions, the link between Theorem 1 and the experimental protocol is incomplete. The authors should either supply the mini-batch/multi-layer extension or explicitly reframe the experiments as independent empirical evidence, with the theory serving only as motivation for the initialization family.
minor comments (6)
  1. [Figure 1 caption; Appendix D.1] There are typographical errors such as 'initlization' and 'aubstracted'; the manuscript should be proofread.
  2. [§5.1] The sentence 'Previous studies have unintentionally employed the non-zero initialization for LoRA' is inaccurate for LoftQ, PiSSA, and LoRA-GA, which deliberately choose non-zero initializations; consider rephrasing to something like 'without analyzing the effect of non-zero initialization on learning-rate robustness.'
  3. [Definition 5] The probability notation in Definition 5 is not defined for asymptotic exponents, which are deterministic functions of the initialization and learning-rate scaling; please state explicitly what randomness the conditional independence refers to.
  4. [§4.3, §D.2] The default experimental protocol subtracts A0B0 from the pretrained weights, yet the paper's headline claim is that fine-tuning need not start from the pretrained model; the relationship between this default and the headline should be clarified earlier in Section 3.5.
  5. [Figure 11 caption] The caption says both Init[A] and Init[AB] use σ²_A=σ²_B=1/n, but for Init[A] σ²_B=0; the caption should be corrected.
  6. [Definition 6 and paragraph after Theorem 1] The expression 'A0/B0=Θ(1)' should be written as a ratio of matrices (e.g., A0 B0^{-1}=Θ(1)) to avoid confusion with entrywise Θ notation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the theoretical conditions are derived from the paper's own formal definitions, and the core empirical claims are independently benchmarked.

full rationale

The paper's derivation chain is not circular. The stability and efficiency conditions (Eq. (6)) are obtained by applying the paper's own γ-operator to the update decomposition in Eq. (2), with Definitions 3 and 4 serving as explicit criteria rather than as derived conclusions. Definition 5 introduces a formal robustness criterion — independence of the exponents of δ1, δ2, and ZB from suboptimal learning-rate exponents — and Theorem 1 is the algebraic solution of Eq. (4) under that criterion together with the efficiency constraints. A theorem that solves a formally stated definition is a normal mathematical derivation, not a circular prediction; the empirical content lies in the subsequently tested claim that equal-variance non-zero initialization improves actual fine-tuning accuracy at small learning rates, which is evaluated on T5-Base, Llama 3-8B, GLUE, commonsense reasoning, and arithmetic reasoning benchmarks independently of the robustness definition. There are no load-bearing self-citations; the cited Hayou et al. works are external prior art used for notation, setup, and baseline criteria. The main caveats are correctness issues rather than circularity: footnote 3 concedes that the mini-batch extension 'requires additional assumptions' that are not supplied, and the paper's γ-exponents are internally inconsistent in places (σ²_A = 1/n is reported as γ[A0] = −1 in §3.2 and §4.1, while Theorem 1 requires γ[A0] = −1/2, and §3.3 writes γ[ZB] = 1 where stability Definition 3 requires γ[ZB] = 0). These flaws undermine the rigor of the theoretical derivation but do not make the derivation equivalent to its inputs by construction.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

No free parameters are fitted in the derivation. The paper's contributions rest on the \gamma-calculus framework inherited from Hayou et al. (2024a,b), on the domain assumptions listed above, and on empirical grids over learning rate and \beta; the latter are hyperparameters, not fitted constants.

assumptions (6)
  • standard math The \gamma-operator satisfies \gamma[uv] = \gamma[u] + \gamma[v] and \gamma[u+v] = \max(\gamma[u], \gamma[v]) for non-cancelling terms, and all analyzed random variables scale as exact powers of n.
    Invoked throughout Section 3 and Appendix B.1; the additivity assumes no sign cancellation between terms.
  • domain assumption Adam-normalized gradients g_A and g_B are \Theta(1), independent of n and of the initialization scale.
    Stated in Section 3.1; this is what makes the updates n-invariant and yields \gamma[\eta] = -1/2 under uniform learning rates.
  • domain assumption The input Z to each LoRA layer is \Theta(1) and remains constant during fine-tuning.
    Section 3.1 restricts the analysis to a single data point, fixing Z; real fine-tuning uses batches and layer-by-layer evolving inputs.
  • domain assumption The LoRA rank r is fixed and independent of the width n.
    Assumed in Definition 1 and Lemma 1; if r scaled with n, the \gamma-exponents in Eqs. (3)-(6) would change.
  • ad hoc to paper Stability, efficiency, and robustness as defined in Definitions 3-5 are the appropriate criteria for practical LoRA fine-tuning.
    Definitions 3-5 are introduced by this paper, and the optimal initialization in Theorem 1 follows from them; the practical relevance depends on these criteria matching real performance.
  • domain assumption The single-LoRA-layer, single-datapoint, infinite-width analysis transfers to the multi-layer, batched, finite-width models used in the experiments.
    Footnote 3 in Section 3.1 acknowledges that the mini-batch extension needs additional assumptions; the paper does not provide them and relies on empirical agreement instead.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics." pith.science (2026). https://pith.science/paper/WUWYRPQP

@misc{pith2026250523194,
  author       = {Pith},
  title        = {Pith review of: Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WUWYRPQP}},
  note         = {Machine review of arXiv:2505.23194}
}
abstract

Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning method. In standard LoRA layers, one of the matrices, $A$ or $B$, is initialized to zero, ensuring that fine-tuning starts from the pretrained model. However, there is no theoretical support for this practice. In this paper, we investigate the impact of non-zero initialization on LoRA's fine-tuning dynamics from an infinite-width perspective. Our analysis reveals that, compared to zero initialization, simultaneously initializing $A$ and $B$ to non-zero values improves LoRA's robustness to suboptimal learning rates, particularly smaller ones. Further analysis indicates that although the non-zero initialization of $AB$ introduces random noise into the pretrained weight, it generally does not affect fine-tuning performance. In other words, fine-tuning does not need to strictly start from the pretrained model. The validity of our findings is confirmed through extensive experiments across various models and datasets. The code is available at https://github.com/Leopold1423/non_zero_lora-icml25.

Figures

Figures reproduced from arXiv: 2505.23194 by the authors.

Figure 1
Figure 1. LoRA’s fine-tuning dynamics with zero initlization (i.e., Init[A]) and non-zero initialization (i.e., Init[AB]). Com￾pared to Init[A], Init[AB] improves LoRA’s robustness to suboptimal learning rates, generally leading to better performance. storage-intensive, presenting significant challenges for their practical deployment. To address these issues, various parameter-efficient fine-tuning (PEFT) methods have been pr… view at source ↗
Figure 2
Figure 2. Test loss of LoRA fine-tuned with zero initlization (Init[A]) and non-zero initialization (Init[AB]). The lower￾left corner presents the improvement ratio of Init[AB] relative to Init[A], while the lower-right corner presents the conver￾gence curves of Init[AB] and Init[A] under specific settings. 3.5. Starting Point of LoRA Fine-Tuning Previous studies have unintentionally employed the non￾zero initialization for L… view at source ↗
Figure 4
Figure 4. Fine-tuning accuracy of T5-Base on different datasets with various learning rates (η) and initialization sizes (β). Note that the initialization variance is set to (βσk) 2 , where σ 2 k is the variance of matrix A when initialized with Kaiming initialization. Init[AB]- Init[A] denotes the accuracy improvement achieved by non-zero initlization, where B is initialized with the same variance as A. there is no need to s… view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Test accuracy of T5-Base finetuned on the QNLI dataset with learning rate η = 3e − 7 and the initlization size β = 16. 4.2. Natural Language Generation Tasks. Models and Datasets. In this section, we fine-tune the Llama 3-8B model (Dubey et al., 2024) on commonsense an…
Figure 7
Figure 7. Figure 7: Fine-tuning accuracy with different starting points. Init[AB+] denotes the variant which do not subtract non-zero initialized AB from W, and Init[AB+]-Init[AB] indicates the accuracy gap between these two initialization strategies. 5. Related Works 5.1. Low-Rank Adapta…
Figure 8
Figure 8. Figure 8: Fine-tuning accuracy of T5-Base on the GLUE benchmark with non-zero initialization [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Fine-tuning accuracy of Llama 3-8B on the commonsense and arithmetic reasoning benchmarks with non-zero initialization [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Fine-tuning accuracy with different starting points. Init[AB+] denotes the variant which do not subtract non-zero initialized A0B0 from W, and Init[AB+]-Init[AB] indicates the accuracy gap between these two initialization strategies [PITH_FULL_IMAGE:figures/full_fig_…
Figure 11
Figure 11. Figure 11: Experimental results with LoRA+. The hyperparameter λ represents the value of ηB/ηA (i.e., ηA = η, ηB = λη). Both Init[A] and Init[AB] use the initialization size of β = 1, that is, σ 2 A = σ 2 B = 1/n, where n is the network width. 18 [PITH_FULL_IMAGE:figures/full_f…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PoLoRA: A Preconditioned Orthogonalized LoRA Optimizer

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A preconditioned, product-aware spectral optimizer for LoRA reaches Adam's final loss in 1.2–1.7x fewer steps on instruction-tuning, with ~3% per-step overhead.

Reference graph

Works this paper leans on

14 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [1]

    Let Ut = ft(x) − y

    (24) Notably, the Adam optimizer guarantees that gA = Θ(1) and gB = Θ(1) through gradient normalization, whereas SGD does not ensure this. Let Ut = ft(x) − y. At step t, the update of LoRA’s output is given by: ∆ft = ft(x) − ft−1(x) = − ηab2 t−1Ut−1∥x∥2 | {z } δ1 t − ηb(a⊤ t−1x)2Ut−1 | {z } δ2 t + ηaηbU 2 t−1bt−1(a⊤ t−1x)∥x∥2 | {z } δ3 t . (25) To ensure ...

    work page 2011

  2. [2]

    In general, µP specifies the parameter weights to be randomly initialized at a scale of Θ(n−1/2), with weight updates at a scale of Θ(n−1)

    Feature learning: ∆Yl = Θ(1), where ∆ represents the parameter update after one gradient descent step. In general, µP specifies the parameter weights to be randomly initialized at a scale of Θ(n−1/2), with weight updates at a scale of Θ(n−1). The random initialization and update scales for input and output weights should be Θ(1) and Θ(n−1), respectively. ...

    work page 2017

  3. [6]

    One-step full gradient suffices for low-rank fine-tuning, provably and efficiently

    Zhang, Y ., Liu, F., and Chen, Y . One-step full gradient suffices for low-rank fine-tuning, provably and efficiently. CoRR, abs/2502.01235,

    arXiv

  4. [9]

    (18) Thus, we conclude that: ( γ[ηA] + γ[ηB] = −1, γ[A0] ≤ γ[ηA], γ [B0] ≤ γ[ηB]

    (17) Substituting this equation back into γ[δ1 t ] and γ[δ2 t ] yields: ( max(γ[A0], γ[ηA]) = γ[ηA], max(γ[B0], γ[ηB]) = γ[ηB]. (18) Thus, we conclude that: ( γ[ηA] + γ[ηB] = −1, γ[A0] ≤ γ[ηA], γ [B0] ≤ γ[ηB]. (19) B.2. Internal Stability In this paper, we focus solely on the stability of LoRA’s final output,ZB, while excluding the stability of LoRA’s int...

    work page 2022

  5. [11]

    The hyperparameter λ represents the value of ηB/ηA (i.e., ηA = η, ηB = λη)

    Experimental results with LoRA+. The hyperparameter λ represents the value of ηB/ηA (i.e., ηA = η, ηB = λη). Both Init[A] and Init[AB] use the initialization size of β = 1, that is, σ2 A = σ2 B = 1/n, where n is the network width. 18 Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics F. Connection ...

    work page 2023

  6. [13]

    β is tuned among {1, 2, 4, 8, 16}

    • Initialization Strategy: The initialization variance is set to (βσk)2, where σ2 k represent the variance of matrix A under Kaiming initialization (i.e., σ2 k = 1 n). β is tuned among {1, 2, 4, 8, 16}. For non-zero initialization, B is initialized with the same variance as A. By default, we subtract the non-zero initialized LoRA from the pretrained weigh...

    work page 2024

  7. [64]

    During fine-tuning, the learning rate is tuned among the set {3e − 4, 4e − 4, 5e − 4, 6e − 4, 7e − 4, 8e − 4, 9e − 4, 1e − 3, 2e − 3, 3e − 3}, with W0, Win, Wout frozen

    During pretraining, the learning rate is set to 1e − 3, with 2000 training steps. During fine-tuning, the learning rate is tuned among the set {3e − 4, 4e − 4, 5e − 4, 6e − 4, 7e − 4, 8e − 4, 9e − 4, 1e − 3, 2e − 3, 3e − 3}, with W0, Win, Wout frozen. Each learning rate is trained for 100 steps. All experiments are conducted with 10 independent repetition...

    work page 2000

  8. [2015]

    The power of scale for parameter-efficient prompt tuning

    Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3045–3059,

    work page 2021

Show all 14 references
  1. [2017]

    The model dimensions (d, n, r) are set to (784, 4096, 32)

    datasets are used for pretraining and fine-tuning, with input images flattened into one-dimensional vectors. The model dimensions (d, n, r) are set to (784, 4096, 32). 16 Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynami...

    2019

  2. [2021]

    Loftq: Lora-fine-tuning-aware quantiza- tion for large language models

    Li, Y ., Yu, Y ., Liang, C., Karampatziakis, N., He, P., Chen, W., and Zhao, T. Loftq: Lora-fine-tuning-aware quantiza- tion for large language models. In International Confer- ence on Learning Representations (ICLR), 2024c. Liu, W., Wang, H., Wang, J., Li, R., Li, X., Zhang, ...

    2024

  3. [2022]

    Hu, Z., Wang, L., Lan, Y ., Xu, W., Lim, E., Bing, L., Xu, X., Poria, S., and Lee, R. K. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large lan- guage models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, (EMNLP),

    2023

  4. [2023]

    Embedding compression in recommender systems: A survey

    Li, S., Guo, H., Tang, X., Tang, R., Hou, L., Li, R., and Zhang, R. Embedding compression in recommender systems: A survey. ACM Comput. Surv., 56(5):130:1– 130:21, 2024a. Li, S., Hu, Z., Tang, X., Wang, H., Xu, S., Luo, W., Li, Y ., He, X., and Li, R. Mixed-precision embedding...

    arXiv

  5. [2024]

    H., and Gao, J

    Wang, Y ., Agarwal, S., Mukherjee, S., Liu, X., Gao, J., Awadallah, A. H., and Gao, J. Adamix: Mixture-of- adaptations for parameter-efficient model tuning. In Pro- ceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP),

    2022

  6. [2025]

    Background A.1

    12 Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamics A. Background A.1. Scaling Theory of Neural Networks In neural networks, scaling refers to enhancing the model’s representational capacity by increasing the scale of...

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.