Pith. sign in

REVIEW 4 major objections 4 minor 49 references

LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LoRASuite claims old LoRA adapters can be recycled across LLM upgrades, beating full retraining on some backbones.

desk verdict Real problem, first framing, but the transfer matrix is undefined for the actual rectangular embeddings and the headline gains seem to come from the high-LR fine-tuning step rather than the transfer. read the letter →

arxiv 2505.13515 v1 pith:BMFFUYQU submitted 2025-05-17 cs.LG cs.AIcs.CL

Yanan Li , Fanxu Meng , Muhan Zhang , Shiai Zhu , Shangguang Wang , Mengwei Xu This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords LoRAparameter-efficientfine-tuningLLMupgradetransfermatrixcenteredkernelalignmentHungarianalgorithmattentionheadmappinglow-rankadaptation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Every time a model family releases a new version, the LoRA adapters trained for the old version are usually discarded and retrained from scratch. LoRASuite argues they can instead be transplanted: given both checkpoints, it builds a transfer matrix from the embedding weights, aligns layers with representation similarity and attention heads with cosine similarity, then pays for a tiny fine-tuning pass. The paper reports that recycled adapters consistently beat same-scale vanilla LoRA and, on MiniCPM and Qwen math benchmarks, also beat full-scale LoRA retraining by +1.4 and +6.6 average points while cutting memory by 5.5 GB and time by about 78%. If correct, adapter repositories for app-specific tasks would survive backbone upgrades almost for free.

What carries the argument

The central object is the embedding-derived transfer matrix $W_h = E_o^{-1}E_n$, which is composed into every head-level update, e.g. $(\Delta W_Q^j)_n = W_h^T (\Delta W_Q^i)_o ((W_Q^i)_o)^T W_h (W_Q^j)_n$, so that a single linear map is responsible for moving LoRA updates across hidden-space changes. Around it sit the layer-mapping machinery (minibatch CKA with a dynamic-programming alignment under a maximum-offset constraint), the head-mapping machinery (cosine similarity over input-independent interaction matrices $W_{QK}$ and $W_{VO}$, assigned by the Hungarian algorithm), and an SVD step that re-factors each transformed update into low-rank $B_n, A_n$. A deliberate small fine-tuning stage compensates for the numerical instability of purely matrix-multiplication transforms.

What would settle it

For an upgrade such as MiniCPM-S-1B to MiniCPM-2B (hidden size 1536 to 2304), identify the exact matrix used as $W_h$ and check whether it satisfies $W_h E_o = E_n$ on the shared-token rows; if no such matrix is specified, substitute a random matrix of the same shape in 'LoRASuite w/o LFT' and compare — if accuracy barely changes, the transfer matrix is not carrying the claimed effect.

Watch

Extended reading notes

Core claim

LoRASuite's central claim is that a LoRA update $B_o A_o$ trained on an older LLM can be transplanted into a newer LLM of the same architecture by transforming it through known parameter differences rather than retraining. Hidden-size and vocabulary changes are handled with a transfer matrix $W_h = E_o^{-1}E_n$ built from the two models' embedding weights (with a shared-token intersection step when vocabularies differ); intermediate-size changes use $W_i = W_o^{-1} W_h W_n$; layer-depth changes are aligned by a dynamic program over minibatch CKA similarities; and attention-head differences are assigned with the Hungarian algorithm on cosine similarities between per-head interaction matrices $W_{QK}$ and $W_{VO}$. The relocated head-level updates are factored back into low-rank form by SVD, and a short fine-tuning pass is added because the transformed parameters come from matrix products rather than backpropagation. On five backbone upgrade pairs the method consistently beats same-scale vanilla LoRA, and on MiniCPM-S-1B to MiniCPM-2B and Qwen-1.5-1.8B to Qwen-2.5-3B it beats full-scale LoRA retraining on math tasks by +1.4 and +6.6 average points.

Load-bearing premise

The load-bearing premise is that one linear map computed from the two models' embedding weights can carry LoRA updates from the old hidden space to the new one; in the reported upgrades the embedding matrices are rectangular, so the stated inverse $W_h = E_o^{-1}E_n$ does not exist and the paper does not specify its replacement.

Editorial extensions

If this is right

  • LoRA weights trained for an older model version can be transplanted into the new version rather than discarded, removing the main retraining cost of model upgrades.
  • With only 100 fine-tuning samples, LoRASuite beats full 10,000-sample LoRA retraining on MiniCPM math tasks (43.80 vs 42.39 average) and on Qwen math tasks (56.88 vs 50.32).
  • The same recipe transfers DoRA adapters to within less than 1 point of full-scale retraining, so the approach generalizes beyond vanilla LoRA.
  • Memory drops by 5.5 GB and training time by 78.23% compared with full LoRA retraining, which is what makes per-app adapters on devices economically plausible.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the reported hidden-size upgrades all involve rectangular embedding matrices, the written formula $W_h = E_o^{-1}E_n$ cannot be evaluated as stated; replacing it with a least-squares pseudo-inverse is the natural minimal variant to test and is not reported in the paper.
  • If the embedding-derived map is genuinely carrying the transfer, the same construction should extend to cross-family upgrades that share a tokenizer; the paper restricts itself to same-architecture upgrades, so this is an open testable extrapolation.
  • The finding that LoRASuite's advantage jumps from +0.27 points at learning rate $10^{-4}$ to +21.30 at $9 \times 10^{-4}$ suggests the transferred initialization matters mainly through the fine-tuning trajectory, so a sweep over optimizers and schedules is a direct way to separate initialization value from fine-tuning value.
  • The method's modularity means any improved component (better similarity metric than CKA, better assignment than Hungarian) can be swapped in independently; comparing components one at a time would isolate which stage contributes most.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes LoRASuite, a method for adapting LoRA adapters trained on an older LLM to a newer version of the same architecture family without retraining from scratch. The authors categorize model upgrades into six types (vocabulary size, hidden size, intermediate size, layer depth, head count, attention type) and propose modular remedies: a transfer matrix W_h = E_o^{-1}E_n computed from the embedding weights of the two models (for vocab/hidden changes), a similar W_i for intermediate sizes, a dynamic-programming layer mapping driven by minibatch CKA similarities, and a Hungarian-based attention-head mapping using cosine similarity of input-independent interaction matrices; the transferred parameters are then decomposed by SVD and optionally refined by a small-scale fine-tuning step (LFT). Experiments across six backbone upgrade pairs and fifteen math/commonsense datasets report that LoRASuite with 100-1000 examples outperforms same-scale vanilla LoRA and, for MiniCPM and Qwen, exceeds full-scale LoRA retraining, while cutting memory by 5.5 GB and time by 78.23%. The paper also reports ablations of the layer- and head-mapping components.

Significance. If the central claims held, the contribution would be practically significant: recycling existing LoRA adapters across model upgrades would avoid expensive retraining, and the modular treatment of six upgrade factors is a sensible decomposition of a problem that has received little direct attention. The evaluation is broad (six backbone pairs, fifteen datasets), and the ablations (Figures 11-13) at least demonstrate that the layer- and head-mapping components affect post-fine-tuning accuracy. Credit is also due for the honest limitations paragraph, which concedes that the fine-tuning step is required for optimal performance. However, the paper's core construction is currently not well-defined: W_h uses an inverse of rectangular embedding matrices that does not exist in the stated sense, Eq. (3) is asserted without derivation or code, the CKA inputs are unspecified, and the headline comparisons are run with different learning-rate schedules for LoRASuite and its baselines. The Yi-6B to Yi-1.5-9B results (w/o LFT far above vanilla) also contradict the paper's own explanation that the transfer is numerically unstable.

major comments (4)
  1. [Section 3.1, Eq. (3), Algorithm 2] The central transfer construction is not instantiable as written. The paper defines W_h = E_o^{-1} E_n, but in every reported upgrade with a hidden-size change the embedding matrices are rectangular with the vocabulary dimension far exceeding the hidden dimension (e.g., MiniCPM-S-1B: 73,008x1,536 to MiniCPM-2B: 123,040x2,304; Bloom-560m to Bloomz-1B1: 250k x 1,024 to 250k x 1,536), so the inverse E_o^{-1} does not exist. The shared-token intersection step does not fix this, since the number of shared tokens is still far larger than the hidden dimension. The same problem affects W_i = W_o^{-1} W_h W_n for up/down projections (e.g., Llama-2-7B intermediate 11,008 to Llama-3-8B intermediate 14,336). No pseudo-inverse, least-squares objective, rank condition, or regularization is stated, and since no code is released, Tables 4-18 cannot be traced to any definite construction. The authors should specify the exact construction (e.g., E_o^+ E_n with a stated SVD cutoff or ridge objective), verify it on the reported settings, and state whether any of the reported numbers change.
  2. [Section 4.1, Tables 4-5, Tables 7-8, Figure 4(b)] The main comparison that supports the abstract's claim ('consistently outperforms small-scale vanilla LoRA') is confounded by unequal training recipes. In Tables 7 and 8, LoRASuite's fine-tuning step uses LR 1e-3 with zero warmup, while LoRA baselines use LR 3e-4 with warmup ratio 0.1 (or 100 warmup steps); the main tables therefore do not show a like-for-like comparison at the same data scale. The paper's own sensitivity analysis (Figure 4(b)) shows that when the learning rate is lowered to 1e-4 the LoRASuite advantage over LoRA (Small) shrinks to 0.27 points, and no matched-recipe baseline is reported for the other five backbone pairs in Figures 2-3. As a result, the reported gains and the 'exceeds full-scale LoRA retraining' claims rest on a narrow, unstated hyperparameter window. Please report matched-schedule baselines (same LR, same warmup, same data scale) in all main tables, plot both methods across the LR grid with per-task numbers, and report variance over seeds.
  3. [Section 3.1, Algorithm 1, Eq. (3)] The CKA computation and the head-level update are under-specified. It is not stated which corpus or prompt set is used to collect the activations for the CKA similarity matrix S, how many minibatches are used, what batch size and sequence length, or whether activations come from the base models or from LoRA-adapted models; the heatmaps in Figures 7-10 therefore cannot be reproduced. Equation (3) is asserted without derivation: the composition W_h^T (Delta W_Q)_o (W_Q)_o^T W_h (W_Q)_n mixes the LoRA update, the original projection weights, and the target projection weights in a way that is not explained, and the dimension bookkeeping when both head count and hidden size change (H_o != H_n, d_o != d_n) is not spelled out. A derivation of Eq. (3) (or a reference to the linear-algebra identity it relies on), together with the concrete CKA configuration, is needed before the method can be evaluated independently.
  4. [Section 4.1, Tables 9-10] The interpretation of the 'w/o LFT' rows is internally inconsistent. On MiniCPM the paper notes (correctly) that 'LoRASuite w/o LFT' matches the vanilla model (Tables 4-5: 23.96 vs 23.85 math; 32.69 vs 32.72 commonsense) and attributes this to numerical instability of matrix-multiplication-only transforms. But for Yi-6B to Yi-1.5-9B the same 'w/o LFT' configuration is far above vanilla (math 73.07 vs 65.86; commonsense 74.29 vs 56.78), and in Table 9 the fine-tuned LoRASuite is actually worse than 'w/o LFT' on AQuA (30.31 vs 38.58). One of these two patterns must be wrong, or the mechanism must be something other than 'numerical instability' (e.g., the eval pipeline has high variance, or the Yi numbers reflect an uncontrolled component). The paper should explain this discrepancy and report per-seed variance, since the flagship claim that the transfer itself is valuable rests entirely on the post-LFT numbers.
minor comments (4)
  1. [Throughout] Typos and formatting issues: 'increase the learning rate to to compensate' (Section 3.2), 'fintuing' (Section 4.1), 'Sensitive Analysis' (Section 4.2), inconsistent 'LoRa'/'LoRA' capitalization, and the 'SV AMP' column header should be typeset correctly.
  2. [Tables 4-18 and Figures 4, 11-13] No randomness control is reported: the tables show single-run numbers without seeds or confidence intervals, which matters in particular for the small-scale (100-example) settings where accuracy differences of 1-2 points are used to support the component ablations.
  3. [Algorithm 1, Section 3.3] The pseudocode's input/output notation ('/# Store...') is inconsistent with the surrounding text, and the complexity claim in Section 3.3 writes O(nlayer(Delta_layer^2 + n_head^3)) without defining the scope of the constants; please clean up the pseudocode formatting.
  4. [Abstract, Figure 1] The abstract's memory saving of 5.5 GB and time reduction of 78.23% are reported as headline numbers, but Figure 1 measures a single setting (MiniCPM); please state explicitly that these figures are setting-specific.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the transfer mappings are computed from known backbone weights and similarity metrics, not from the benchmark numbers they are evaluated against.

full rationale

LoRASuite's derivation chain does not reduce to its own inputs. The transfer matrix Wh = E_o^-1 E_n (Section 3.1) and the head-level update in Equation (3) are constructed from the embedding and projection weights of the old and new backbones, together with CKA and cosine-similarity measures; none of these quantities is fitted to the downstream task accuracies reported in Tables 4-5 or Figures 2-3, so the 'LoRASuite w LFT' results are genuine evaluations rather than in-sample predictions of fitted values. The paper's own limitation statement (Section 5: 'LoRASuite requires an additional small-scale fine-tuning step to achieve optimal performance') and the near-vanilla 'LoRASuite w/o LFT' rows weaken the causal claim that the algebraic transformation itself supplies the gains, and the comparison is complicated by unequal learning-rate and warmup settings between LoRASuite and LoRA (Small) in Tables 7-8, but these are correctness and experimental-design concerns, not circularity. The undefined E_o^-1 for rectangular embedding matrices is an instantiation gap rather than a definitional equivalence. No self-citation is load-bearing: references [28], [29], and [33] are external methodological sources for CKA and the Hungarian algorithm. Therefore no circular step is exhibited.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or architectural entities; its load-bearing elements are the embedding-derived transfer matrix, CKA layer mapping, and Hungarian head matching, all of which rely on assumptions about representational similarity and matrix invertibility that are not established in the paper. The main free choices are experimental hyperparameters and an unspecified CKA configuration.

free parameters (3)
  • LFT learning rate = 1e-3
    LoRASuite's fine-tuning uses 1e-3 with no warm-up (Tables 7, 8); sensitivity analysis shows gains largely disappear at 1e-4, so the reported results depend on this choice.
  • Fine-tuning data scale = 100 or 1,000 samples
    The main results use 100 samples for MiniCPM, Llama, and Qwen, and 1,000 for Pythia and Bloom; performance is reported to decline with larger scale (Section 4.2).
  • CKA minibatch configuration = not specified
    The minibatch CKA requires a dataset, batch size, and number of batches to estimate layer similarity; none are reported, so the layer mapping is not reproducible.
assumptions (4)
  • ad hoc to paper Embedding matrices E_o and E_n admit a meaningful inverse or pseudo-inverse such that W_h = E_o^{-1} E_n maps old hidden representations to new ones.
    Section 3.1 uses this for the transfer matrix, but E_o is rectangular in all hidden-size-changing upgrades in Table 1, and no pseudo-inverse or least-squares formulation is given.
  • domain assumption CKA similarity between layers computed on a shared input distribution is a valid guide for transferring LoRA weights.
    Section 3.1 constructs the layer mapping by maximizing total CKA similarity; this assumes representational similarity predicts weight-transfer quality, which is not proven.
  • domain assumption Attention head behavior is captured by input-independent interaction matrices W_QK and W_VO, and cosine similarity among them is a valid matching criterion.
    Section 3.1 defines the interaction matrices and uses cosine similarity for Hungarian matching; the paper provides no theoretical or empirical justification beyond downstream accuracy.
  • domain assumption The small-scale fine-tuning step preserves and stabilizes transformed LoRA weights.
    Section 3.2 states that LFT is needed for numerical stability, but the transformation-only runs in Tables 4 and 5 show near-vanilla performance, indicating the transformation alone does not transfer task ability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades." pith.science (2026). https://pith.science/paper/BMFFUYQU

@misc{pith2026250513515,
  author       = {Pith},
  title        = {Pith review of: LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BMFFUYQU}},
  note         = {Machine review of arXiv:2505.13515}
}
read the original abstract

As Large Language Models (LLMs) are frequently updated, LoRA weights trained on earlier versions quickly become obsolete. The conventional practice of retraining LoRA weights from scratch on the latest model is costly, time-consuming, and environmentally detrimental, particularly as the diversity of LLMs and downstream tasks expands. This motivates a critical question: "How can we efficiently leverage existing LoRA weights to adapt to newer model versions?" To address this, we propose LoRASuite, a modular approach tailored specifically to various types of LLM updates. First, we compute a transfer matrix utilizing known parameters from both old and new LLMs. Next, we allocate corresponding layers and attention heads based on centered kernel alignment and cosine similarity metrics, respectively. A subsequent small-scale, skillful fine-tuning step ensures numerical stability. Experimental evaluations demonstrate that LoRASuite consistently surpasses small-scale vanilla LoRA methods. Notably, on backbone LLMs such as MiniCPM and Qwen, LoRASuite even exceeds the performance of full-scale LoRA retraining, with average improvements of +1.4 and +6.6 points on math tasks, respectively. Additionally, LoRASuite significantly reduces memory consumption by 5.5 GB and computational time by 78.23%.

Figures

Figures reproduced from arXiv: 2505.13515 by the authors.

Figure 1
Figure 1. Memory and time comparison between LoRASuite and LoRA retraining. Memory and time consumption [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Average performance comparison on math tasks for different types of LLM upgrades. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Average performance comparison on common tasks for different types of LLM upgrades. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Average performance on math tasks under different settings for the MiniCPM-S-1B to [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Performance comparison on math tasks for different LLMs with up_proj and down_proj as [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Performance comparison on commonsense tasks for different LLMs with up_proj and [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Heatmap of CKA layer similarity between MiniCPM-S-1B and MiniCPM-2B. 0 10 20 30 40 Yi-1.5-9B 0 5 10 15 20 25 30 Yi-6B 0.93 0.94 0.95 0.96 0.97 0.98 0.99 [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 10
Figure 10. Figure 10: Heatmap of CKA layer similarity be￾tween Qwen1.5-1.8B and Qwen2.5-3B. A.5 Detailed Results for Different LLM Backbones Tables 9–18 provide the raw data underlying the aggregated results in Figures 2 and 3 [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Performance comparison with dif￾ferent layer mapping algorithms when up￾grading the LLM from MiniCPM-S-1B to MiniCPM-2B. Math Tasks (a) 0 20 40 Avg. Accuracy (%) 43.80 43.80 41.06 41.94 Commonsense Tasks (b) 0 20 40 Avg. Accuracy (%) 42.34 37.81 41.47 40.29 LoRASuite …
Figure 13
Figure 13. Figure 13: Performance comparison with different attention head mapping algorithms when upgrading [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 20 canonical work pages

  1. [1]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” in International Conference on Learning Representations, 2022. [Online]. Available: https://openreview.net/forum?id=nZeVKeeFYf9

    work page 2022

  2. [2]

    Gemini nano with the google ai edge sdk,

    Google, “Gemini nano with the google ai edge sdk,” 2025. [Online]. Available: https://developer.android.com/ai/gemini-nano

    work page 2025

  3. [3]

    Autodroid: Llm-powered task automation in android,

    H. Wen, Y . Li, G. Liu, S. Zhao, T. Yu, T. J.-J. Li, S. Jiang, Y . Liu, Y . Zhang, and Y . Liu, “Autodroid: Llm-powered task automation in android,” in Proceedings of the 30th Annual International Conference on Mobile Computing and Networking, 2024, pp. 543–557

    work page 2024

  4. [4]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosaleet al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023

    arXiv 2023

  5. [5]

    The llama 4 herd: The beginning of a new era of natively multimodal ai innovation,

    Meta, “The llama 4 herd: The beginning of a new era of natively multimodal ai innovation,”

    work page

  6. [6]

    Qwen2 technical report,

    A. Yang, B. Yang, B. Huiet al., “Qwen2 technical report,” arXiv preprint arXiv:2407.10671, 2024

    arXiv 2024

  7. [7]

    Qwen2.5 technical report,

    A. Yang, B. Yang, B. Zhanget al., “Qwen2.5 technical report,”arXiv preprint arXiv:2412.15115, 2024

    arXiv 2024

  8. [8]

    Fwdllm: Efficient federated finetuning of large language models with perturbed inferences,

    M. Xu, D. Cai, Y . Wu, X. Li, and S. Wang, “Fwdllm: Efficient federated finetuning of large language models with perturbed inferences,” in 2024 USENIX Annual Technical Conference (USENIX ATC 24), 2024, pp. 579–596

    work page 2024

Show all 49 references
  1. [9]

    Understanding the performance and estimating the cost of llm fine-tuning,

    Y . Xia, J. Kim, Y . Chen, H. Ye, S. Kundu, C. Hao, and N. Talati, “Understanding the performance and estimating the cost of llm fine-tuning,” arXiv preprint arXiv:2408.04693, 2024

    2024 arXiv

  2. [10]

    Energy and policy considerations for modern deep learning research,

    E. Strubell, A. Ganesh, and A. McCallum, “Energy and policy considerations for modern deep learning research,” in Proceedings of the AAAI conference on artificial intelligence, vol. 34, no. 09, 2020, pp. 13 693–13 696

    2020

  3. [11]

    What you can cram into a single vector: Probing sentence embeddings for linguistic properties,

    A. Conneau, G. Kruszewski, G. Lample, L. Barrault, and M. Baroni, “What you can cram into a single vector: Probing sentence embeddings for linguistic properties,” arXiv preprint arXiv:1805.01070, 2018

    2018 arXiv

  4. [12]

    Towards automated circuit discovery for mechanistic interpretability,

    A. Conmy, A. Mavor-Parker, A. Lynch, S. Heimersheim, and A. Garriga-Alonso, “Towards automated circuit discovery for mechanistic interpretability,”Advances in Neural Information Processing Systems, vol. 36, pp. 16 318–16 352, 2023

    2023

  5. [13]

    Locating and editing factual associations in gpt,

    K. Meng, D. Bau, A. Andonian, and Y . Belinkov, “Locating and editing factual associations in gpt,” Advances in Neural Information Processing Systems, vol. 35, pp. 17 359–17 372, 2022

    2022

  6. [14]

    Analyzing transformers in embedding space,

    G. Dar, M. Geva, A. Gupta, and J. Berant, “Analyzing transformers in embedding space,”arXiv preprint arXiv:2209.02535, 2022

    2022 arXiv

  7. [15]

    A practical review of mechanistic inter- pretability for transformer-based language models,

    D. Rai, Y . Zhou, S. Feng, A. Saparov, and Z. Yao, “A practical review of mechanistic inter- pretability for transformer-based language models,” arXiv preprint arXiv:2407.02646, 2024

    2024

  8. [16]

    Parameter-efficient transfer learning for nlp,

    N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. At- tariyan, and S. Gelly, “Parameter-efficient transfer learning for nlp,” inInternational conference on machine learning. PMLR, 2019, pp. 2790–2799

    2019

  9. [17]

    The power of scale for parameter-efficient prompt tuning,

    B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter-efficient prompt tuning,” arXiv preprint arXiv:2104.08691, 2021

    2021 arXiv

  10. [18]

    Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning,

    H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. A. Raffel, “Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning,” Advances in Neural Information Processing Systems, vol. 35, pp. 1950–1965, 2022. 10

    1950

  11. [19]

    Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models,

    E. B. Zaken, S. Ravfogel, and Y . Goldberg, “Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models,” arXiv preprint arXiv:2106.10199, 2021

    2021

  12. [20]

    Training neural networks with fixed sparse masks,

    Y .-L. Sung, V . Nair, and C. A. Raffel, “Training neural networks with fixed sparse masks,” Advances in Neural Information Processing Systems, vol. 34, pp. 24 193–24 205, 2021

    2021

  13. [21]

    Input-tuning: Adapting unfamiliar inputs to frozen pretrained models,

    S. An, Y . Li, Z. Lin, Q. Liu, B. Chen, Q. Fu, W. Chen, N. Zheng, and J.-G. Lou, “Input-tuning: Adapting unfamiliar inputs to frozen pretrained models,” arXiv preprint arXiv:2203.03131, 2022

    2022 arXiv

  14. [22]

    Adaptive budget allocation for parameter-efficient fine-tuning,

    Q. Zhang, M. Chen, A. Bukharin, P. He, Y . Cheng, W. Chen, and T. Zhao, “Adaptive budget allocation for parameter-efficient fine-tuning,” in The Eleventh International Conference on Learning Representations , 2023. [Online]. Available: https://openreview.net/forum?id= lq62uWRJjiY

    2023

  15. [23]

    Losparse: Structured compres- sion of large language models based on low-rank and sparse approximation,

    Y . Li, Y . Yu, Q. Zhang, C. Liang, P. He, W. Chen, and T. Zhao, “Losparse: Structured compres- sion of large language models based on low-rank and sparse approximation,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 20 336–20 350

    2023

  16. [24]

    Dora: Weight-decomposed low-rank adaptation,

    S.-Y . Liu, C.-Y . Wang, H. Yin, P. Molchanov, Y .-C. F. Wang, K.-T. Cheng, and M.-H. Chen, “Dora: Weight-decomposed low-rank adaptation,” arXiv preprint arXiv:2402.09353, 2024

    2024 arXiv

  17. [25]

    Pissa: Principal singular values and singular vectors adaptation of large language models,

    F. Meng, Z. Wang, and M. Zhang, “Pissa: Principal singular values and singular vectors adaptation of large language models,” arXiv preprint arXiv:2404.02948, 2024

    2024 arXiv

  18. [26]

    Lora+: Efficient low rank adaptation of large models,

    S. Hayou, N. Ghosh, and B. Yu, “Lora+: Efficient low rank adaptation of large models,”arXiv preprint arXiv:2402.12354, 2024

    2024 arXiv

  19. [27]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,

    K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in Proceedings of the IEEE international conference on computer vision, 2015, pp. 1026–1034

    2015

  20. [28]

    Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth,

    T. Nguyen, M. Raghu, and S. Kornblith, “Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth,” in International Conference on Learning Representations , 2021. [Online]. Available: https://openreview.net/forum?...

    2021

  21. [29]

    Similarity of neural network representations revisited,

    S. Kornblith, M. Norouzi, H. Lee, and G. Hinton, “Similarity of neural network representations revisited,” in International conference on machine learning. PMLR, 2019, pp. 3519–3529

    2019

  22. [30]

    Feature selection via dependence maximization,

    L. Song, A. Smola, A. Gretton, J. Bedo, and K. Borgwardt, “Feature selection via dependence maximization,” The Journal of Machine Learning Research, vol. 13, pp. 1393–1434, 2012

    2012

  23. [31]

    Post selection inference with kernels,

    M. Yamada, Y . Umezu, K. Fukumizu, and I. Takeuchi, “Post selection inference with kernels,” in International conference on artificial intelligence and statistics. PMLR, 2018, pp. 152–160

    2018

  24. [32]

    On the variance of the adaptive learning rate and beyond,

    L. Liu, H. Jiang, P. He, W. Chen, X. Liu, J. Gao, and J. Han, “On the variance of the adaptive learning rate and beyond,” arXiv preprint arXiv:1908.03265, 2019

    1908 arXiv

  25. [33]

    Algorithms for the assignment and transportation problems,

    J. Munkres, “Algorithms for the assignment and transportation problems,”Journal of the society for industrial and applied mathematics, vol. 5, no. 1, pp. 32–38, 1957

    1957

  26. [34]

    Boolq: Exploring the surprising difficulty of natural yes/no questions,

    C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova, “Boolq: Exploring the surprising difficulty of natural yes/no questions,”arXiv preprint arXiv:1905.10044, 2019

    1905 arXiv

  27. [35]

    Piqa: Reasoning about physical commonsense in natural language,

    Y . Bisk, R. Zellers, J. Gao, Y . Choiet al., “Piqa: Reasoning about physical commonsense in natural language,” in Proceedings of the AAAI conference on artificial intelligence , vol. 34, no. 05, 2020, pp. 7432–7439

    2020

  28. [36]

    Socialiqa: Commonsense reasoning about social interactions,

    M. Sap, H. Rashkin, D. Chen, R. LeBras, and Y . Choi, “Socialiqa: Commonsense reasoning about social interactions,” arXiv preprint arXiv:1904.09728, 2019

    1904 arXiv

  29. [37]

    Hellaswag: Can a machine really finish your sentence?

    R. Zellers, A. Holtzman, Y . Bisk, A. Farhadi, and Y . Choi, “Hellaswag: Can a machine really finish your sentence?” arXiv preprint arXiv:1905.07830, 2019. 11

    1905 arXiv

  30. [38]

    Winogrande: An adversarial winograd schema challenge at scale,

    K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y . Choi, “Winogrande: An adversarial winograd schema challenge at scale,” Communications of the ACM, vol. 64, no. 9, pp. 99–106, 2021

    2021

  31. [39]

    Think you have solved question answering? try arc, the ai2 reasoning challenge,

    P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, “Think you have solved question answering? try arc, the ai2 reasoning challenge,” arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  32. [40]

    Can a suit of armor conduct electricity? a new dataset for open book question answering,

    T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal, “Can a suit of armor conduct electricity? a new dataset for open book question answering,” arXiv preprint arXiv:1809.02789, 2018

    2018 arXiv

  33. [41]

    Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models,

    Z. Hu, Y . Lan, L. Wang, W. Xu, E.-P. Lim, R. K.-W. Lee, L. Bing, and S. Poria, “Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models,” arXiv preprint arXiv:2304.01933, 2023

    2023 arXiv

  34. [42]

    Program induction by rationale generation: Learning to solve and explain algebraic word problems,

    W. Ling, D. Yogatama, C. Dyer, and P. Blunsom, “Program induction by rationale generation: Learning to solve and explain algebraic word problems,” arXiv preprint arXiv:1705.04146, 2017

    2017 arXiv

  35. [43]

    Training verifiers to solve math word problems,

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano et al., “Training verifiers to solve math word problems,” arXiv preprint arXiv:2110.14168, 2021

    2021 arXiv

  36. [44]

    Mawps: A math word problem repository,

    R. Koncel-Kedziorski, S. Roy, A. Amini, N. Kushman, and H. Hajishirzi, “Mawps: A math word problem repository,” inProceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, 2016, pp. 1152–1157

    2016

  37. [45]

    Are nlp models really able to solve simple math word problems?

    A. Patel, S. Bhattamishra, and N. Goyal, “Are nlp models really able to solve simple math word problems?” arXiv preprint arXiv:2103.07191, 2021

    2021 arXiv

  38. [46]

    Chain-of- thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhouet al., “Chain-of- thought prompting elicits reasoning in large language models,” Advances in neural information processing systems, vol. 35, pp. 24 824–24 837, 2022

    2022

  39. [47]

    Transformer layers as painters,

    Q. Sun, M. Pickett, A. K. Nain, and L. Jones, “Transformer layers as painters,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 24, 2025, pp. 25 219–25 227

    2025

  40. [48]

    Insights on representational similarity in neural networks with canonical correlation,

    A. Morcos, M. Raghu, and S. Bengio, “Insights on representational similarity in neural networks with canonical correlation,” Advances in neural information processing systems, vol. 31, 2018. 12 Algorithm 3 Hungarian Algorithm Require: Cost matrixC of sizem×n Ensure: Optimal as...

    2018

  41. [2025]

    Available: https://ai.meta.com/blog/llama-4-multimodal-intelligence/

    [Online]. Available: https://ai.meta.com/blog/llama-4-multimodal-intelligence/

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.