CLOVER: Cross-Layer Orthogonal Vectors Pruning and Fine-Tuning

Fan Jiang; Fanxu Meng; Muhan Zhang; Pingzhi Tang

REVIEW 4 major objections 5 minor 57 references

CLOVER: Cross-Layer Orthogonal Vectors Pruning and Fine-Tuning

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CLOVER decomposes attention heads' query–key and value–output products with one SVD each, making tiny singular directions prunable and turning singular values into a full-rank fine-tuning update that beats LoRA, DoRA, HiRA, and PiSSA.

desk verdict A clever orthogonalization trick with solid non-RoPE pruning results, but the headline fine-tuning mechanism is not exercised on RoPE models and the trainable 'singular values' are likely dense matrices. read the letter →

arxiv 2411.17426 v3 pith:RNHH35TG submitted 2024-11-26 cs.LG cs.AI

Fanxu Meng , Pingzhi Tang , Fan jiang , Muhan Zhang This is my paper · ORCID

classification cs.LGcs.AI MSC 68T0715A18

keywords cross-layerorthogonalizationSVD-basedpruningattentionheadcompressionparameter-efficientfine-tuningfull-rankupdatesingular-valueadaptationKVcache

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CLOVER claims that the two matrix pairs inside every attention head — query against key, value against output — act as one low-rank bilinear form each, so a single truncated singular value decomposition (SVD) per pair can orthogonalize the head without adding any parameters. Once orthogonalized, most of the head's behavior sits in a few singular directions and the rest is nearly zero, which makes pruning those leftover directions nearly free: the paper reports that pruning 70% of the query–key vectors in GPT-2 XL degrades perplexity about as much as an 8% vanilla prune. The same singular values are also the only trainable parameters during fine-tuning, and because tuning them rescales every basis direction, the paper measures the resulting update to be full-rank, in contrast with LoRA's low-rank updates; on eight commonsense tasks this beats LoRA, DoRA, HiRA, and PiSSA on LLaMA-7B/13B, LLaMA-2-7B, and LLaMA-3-8B at equal or fewer parameters. A sympathetic reader would care because compression and adaptation are usually separate toolchains, and CLOVER offers one orthogonalization that feeds both, with no inference-time overhead.

What carries the argument

The load-bearing object is the merged per-head bilinear form $W^h_{QK}=W^h_Q (W^h_K)^\top\in\mathbb{R}^{D\times D}$, whose rank is bounded by the head dimension $d$, together with its truncated SVD $W^h_{QK}=U^h S^h (V^h)^\top$ keeping the top $r\le d$ singular triples; the same construction on $W^h_{VO}=W^h_V W^h_O$ covers the value–output pair, and the MLP Up layer is handled by treating blocks of 64 consecutive dimensions as heads. The mechanism that carries the argument is treating the Q and K projections as each other's transformation matrices, so orthogonalizing the product creates no extra matrices — the factors simply take the place of the original weights. The singular-value diagonal $S^h$ is then a single small object doing double duty: its magnitudes rank the directions for pruning, and its entries are the only trainable scalars during fine-tuning, giving a per-head parameter count of a dense $d\times d$ diagonal comparable to a rank-32 LoRA. The claimed consequence that makes fine-tuning strong is that updating $S$ rescales every basis direction simultaneously, so the induced update is not confined to a low-rank subspace, and the paper's measured singular-value spectra show it matching full fine-tuning's rank.

What would settle it

Fine-tune LLaMA-2-7B with CLOVER, PiSSA, and LoRA using identical validation-selected checkpoints and matched learning-rate sweeps, then compute the singular-value spectrum of each method's merged update: if CLOVER's accuracy lead over PiSSA disappears, or its update's effective rank is far below the head dimension, the full-rank orthogonalization mechanism is not what produces the reported gains.

Watch

Extended reading notes

Core claim

The paper's central claim is that crossing the layers before decomposing is what makes an attention head compressible without extra machinery. Instead of factorizing each projection matrix individually — the route taken by SVFT, which needs an accompanying transformation matrix and doubles the parameter count — CLOVER forms the merged products $W^h_{QK}=W^h_Q (W^h_K)^\top$ and $W^h_{VO}=W^h_V W^h_O$ inside each head, each a $D\times D$ matrix of rank at most the head dimension $d$, and truncates their SVDs to the top $r\le d$ singular triples. The orthogonal factors $U$ and $V$ replace the original weight slices, so the orthogonalization costs no extra parameters, and the diagonal $S$ is the only new object. In pruning mode, small singular values (and their vectors) are dropped; in fine-tuning mode, $U$ and $V$ are frozen and only $S$ is trained, then merged back at the end. Because $S$ scales every orthogonal basis at once, the update $U\,\Delta S\,V^\top$ can touch all $d$ directions, which the paper calls a full-rank update and shows to be the reason it beats fixed-subspace adapters such as PiSSA. The paper states one clear limit in its conclusion: a nonlinearity between the Q–K or V–O pair, most notably RoPE, makes cross-layer orthogonalization infeasible, so the LLaMA fine-tuning experiments instead orthogonalize the Key layer head-wise.

Load-bearing premise

The load-bearing premise is that an attention head's query–key interaction really is a fixed bilinear product of its Q and K projection matrices, with no position-dependent rotation (such as RoPE, rotary position embeddings) inserted between them, a condition the paper itself says fails for the RoPE-based models it fine-tunes.

Editorial extensions

If this is right

Attention pruning becomes dramatically safer on models whose positional encoding does not sit between Q and K: pruning half of GPT-2 XL's attention parameters raises perplexity 1.38× with CLOVER versus 21.9× with vanilla pruning, and roughly 66 million fine-tuning tokens nearly restore the base perplexity.
Fine-tuning the singular-value diagonal alone matches LoRA rank-32 parameter counts while producing a full-rank update, and outperforms LoRA, DoRA, HiRA, and PiSSA by between 0.6 and 9 percentage points across LLaMA-7B/13B, LLaMA-2-7B, and LLaMA-3-8B on eight commonsense tasks.
Because CLOVER changes only the initialization and merges its factors back into the original weights, it can be combined with any other pruning or quantization technique without altering the model structure or adding inference cost.
Models with pronounced linear redundancy, such as Whisper-Large-v3, can shed 46.42% of their attention parameters with no fine-tuning at all and keep nearly identical output.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because every LLaMA fine-tuning result uses the Key-layer fallback rather than the cross-layer Q–K decomposition, the full-rank-update wins over LoRA and PiSSA should be read as evidence for intra-head orthogonalization, while the cross-layer mechanism is directly validated only by the pruning results on RoPE-free models such as GPT-2, Whisper, ViT, and SDXL.
A natural stress test the paper does not run is continual or long-context learning, where LoRA's intrusive dimensions are known to accumulate; if CLOVER's full-rank updates avoid that accumulation, it becomes a general drop-in adapter rather than a pruning trick.
The same decomposition suggests a quantization policy the paper only gestures at: keep the large singular directions in high precision and drop or aggressively quantize the near-zero directions, which could remove the outliers that KV-cache quantizers currently fight.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper has a real idea, but the main fine-tuning experiments do not actually test it. The cross-layer SVD of W_Q W_K^T is a neat way to expose redundancy, and the pruning results on GPT-2 XL are genuinely impressive. But for LLaMA, which uses RoPE, the Q-K part of the method is abandoned: Section 4.2 says the authors instead perform orthogonal decomposition in the Key layer and fine-tune the transition matrix. So the advertised cross-layer Q-K fine-tuning mechanism is never validated in the model family that matters most. The paper's own limitations section admits this, which is honest, but it means the headline claim outruns the evidence.

What is new: the cross-layer pairing of Q-K and V-O as rank-limited products, then tuning or pruning the middle factor, is not in the cited SVFT, PiSSA, LoRA, or DoRA work. The SVD identity is correct, and the parameter count analysis is transparent. The GPT-2 XL numbers are the strongest part: pruning 50% of attention parameters with CLOVER degrades perplexity far less than vanilla pruning, and fine-tuning recovers most of the gap. For non-RoPE models, this seems like a useful pruning initialization.

The soft spots are substantial. First, the trainable factor is described as singular values, but Appendix A.2 counts 32x128x128 parameters per Q-K pair, which is a dense 128x128 matrix per head, not a diagonal S. If that is right, the method is closer to a per-head full-rank adapter, and the 'fine-tuning singular values' story is misleading. The claimed 'full-rank update' then tells us little beyond the fact that a dense matrix has full rank. Second, the fine-tuning gains over PiSSA are 0.6-1%, and the experiments use a single learning rate without validation. Third, the Whisper pruning demo is one audio sample with no held-out WER. Fourth, pruning is compared only to vanilla L2-norm pruning, not to any state-of-the-art structured pruning method.

Overall verdict: the pruning contribution is worth a serious look, but the fine-tuning claims need major revision. Either test the cross-layer Q-K mechanism on a non-RoPE LLM, or reframe the method as intra-layer orthogonal PEFT and drop the cross-layer fine-tuning language. The dense-matrix issue needs to be resolved before the 'singular value' claims can be trusted. I would send this to peer review, but the reviewers should demand code, multi-seed runs, and an honest split of cross-layer versus intra-layer results.

For a reading group, it is a good case study in overclaiming; I would bring it up for discussion. If I worked on orthogonal PEFT, I would cite the pruning results, not the fine-tuning claims.

Referee Report

4 major / 5 minor

Summary. The paper proposes CLOVER, which applies SVD to the merged per-head matrices W_Q W_K^T and W_V W_O^T, obtaining orthogonal bases and singular values that can guide structured pruning or serve as trainable parameters for parameter-efficient fine-tuning. The authors report large pruning gains on GPT-2 XL and Whisper-Large-v3, and fine-tuning gains over LoRA, DoRA, HiRA, and PiSSA on commonsense reasoning benchmarks for LLaMA-family models. The SVD factorization in Section 3 is mathematically valid, but the main fine-tuning experiments on RoPE-based LLaMA models do not use the cross-layer Q-K decomposition; instead, Section 4.2 switches to an intra-layer Key decomposition with a dense transition matrix. The paper therefore does not test its headline cross-layer fine-tuning mechanism, and the implemented parameterization differs from the derived singular-value tuning.

Significance. If the cross-layer mechanism worked as claimed, it would be a useful contribution to attention-layer compression and parameter-efficient fine-tuning, and the paper covers a broad set of models. The SVD identity in Section 3 is correct and the GPT-2 XL pruning results are a concrete positive data point. However, the significance is substantially weakened because the central fine-tuning claim is not exercised in the main benchmarks: the LLaMA experiments use RoPE, where the paper itself concedes cross-layer Q-K orthogonalization is infeasible, and the trainable middle factor is implemented as a dense matrix rather than as singular values. The paper may contain a viable weaker contribution about intra-layer orthogonalization and pruning, but that is not what the title, abstract, and Section 1 promise.

major comments (4)

[Section 4.2 and Section 5] The paper's own limitation paragraph (Section 5) states that cross-layer orthogonalization of Q-K is not feasible when RoPE is present, and Section 4.2 says that for all LLaMA fine-tuning experiments the authors 'perform orthogonal decomposition in the Key layer and fine-tune the transition matrix' instead of decomposing W_Q W_K^T. Since LLaMA-2-7B and the other fine-tuned models use RoPE, Table 2 never exercises the cross-layer Q-K mechanism that the title, abstract, and Section 3 advertise. The results may support a weaker intra-layer orthogonalization claim, but they do not support the paper's central fine-tuning claim.
[Section 3, Section 4.2, and Appendix A.2] The method described in Section 3 freezes U and V and tunes only the diagonal singular-value matrix S, whose nonzero size is at most d by d. Appendix A.2 counts 32 by 128 by 128 trainable parameters per Q-K pair, which is the size of a dense 128 by 128 matrix, not a diagonal S; Section 4.2 also calls the trainable object a 'transition matrix.' A dense matrix has rank up to 128 and does not correspond to scaling fixed orthogonal bases, so the claim that CLOVER learns 'singular values' and achieves full-rank updates through all orthogonal bases is not what the implementation actually does. This discrepancy changes the effective method relative to the derivation in Section 3.
[Section 4.2 and Table 2] The fine-tuning comparison is not controlled: LoRA and DoRA numbers are taken from the DoRA paper, HiRA from its own paper, and PiSSA is trained with lr=2e-5 while CLOVER uses lr=1e-4; the authors also state that they omit validation every 80 iterations for PiSSA and CLOVER, unlike the DoRA protocol. Under these heterogeneous conditions, the reported gains over SOTA are not attributable to the method alone. A same-pipeline comparison with identical data, hyperparameter choices, and checkpoint selection is needed to support the superiority claims in Table 2 and the abstract.
[Abstract and Table 1] The abstract's quantitative claim that pruning 70% of the Q-K pairs with CLOVER gives perplexity similar to pruning just 8% with vanilla methods is not supported by Table 1: the closest entries are 75% CLOVER perplexity 187.4 versus 12.5% vanilla perplexity 33.76, which are not similar even on a log scale. Figure 1c uses a different 45-versus-5 vector comparison without the experimental details needed to verify it. The stated headline number should be reconciled with the table or removed.

minor comments (5)

[Section 5 and Appendix A.2] The paper repeatedly misspells CLOVER as 'CLVOER' in Section 5 and Appendix A.2; these typos should be corrected.
[Section 4.2 and Appendix A.3] The citation for LLM-Adapters (Hu et al., 2023) used in Appendix A.3 is not included in the reference list, and the unpublished 'Anonymous, 2025' citation for HiRA makes it difficult to verify the reported baseline values.
[Figure 2 and Section 4.3] The 'red dot' and 'red intersection point' mentioned in the caption and text of Figure 2 are not visible in the reproduced figure, so the claim that CLOVER's remaining vectors are consistently less important after that point is hard to check.
[Section 4.4] The pruning thresholds in Section 4.4 (||W_Q|| ||W_K|| <= 5e-3 and ||W_V|| ||W_O^T|| <= 6e-3) are presented without justification or sensitivity analysis, which weakens the training-free pruning claim for Whisper.
[Abstract and Section 1] The statement that parameters are reintegrated 'without increasing its parameter count' is imprecise for the fine-tuning setting, because a dense trainable transition matrix is stored during training before being merged.

Circularity Check

1 steps flagged · score 2.0 of 10

The SVD reparameterization and pruning claims are not circular, but the 'full-rank update' demonstration is a definitional consequence of training a dense S matrix rather than an empirical finding.

self definitional [Section 4.6 (Visualizing Rank Updates) with Appendix A.2]
"To demonstrate CLOVER achieves full-rank updates, we multiply the updated singular values with their corresponding singular vectors and perform SVD on the base model... Figure 5 shows the singular value of the middle layer in LLaMA-2-7B, revealing that CLOVER and Full Fine-tuning achieve full-rank updates..."

CLOVER's fine-tuning uses a trainable matrix S_QK of size 32 x 128 x 128 per Q-K pair, as counted in Appendix A.2, which is dense rather than diagonal. Thus Delta-W = U Delta-S V^T with orthonormal U and V generically has rank up to 128 by construction. The Figure 5 observation that CLOVER produces a full-rank update is therefore a built-in property of the parameterization, not an independently measured result. This does not affect the external pruning and accuracy benchmarks, but it makes the 'full-rank update' evidence self-definitional.

full rationale

The central CLOVER construction is a deterministic SVD reparameterization of existing attention weights: W_QK and W_VO are factorized into orthogonal matrices and singular values, then pruned or fine-tuned. No prediction is derived from fitted constants, and the pruning and commonsense benchmarks are evaluated against external datasets and baselines, so the main empirical claims have independent content. The RoPE limitation in Section 5 is an honest scope concession rather than a circular step, and the PiSSA citation is used as a baseline, not as load-bearing justification. The only notable circularity is the 'full-rank update' demonstration: because the fine-tuned S matrix is dense, the rank of the resulting update is guaranteed by construction, so Figure 5 cannot serve as evidence of a learned advantage. That issue is localized to the interpretability claim and does not invalidate the external performance comparisons, yielding a low overall circularity score.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method rests on standard SVD facts plus three assumptions: no position-dependent rotation for the Q-K bilinear form, concentration of attention weight energy in leading singular directions, and the ad hoc use of a dense d-by-d trainable factor called singular values. Two hand-chosen thresholds and block sizes affect the reported results.

free parameters (3)

Whisper pruning thresholds = ||W_Q|| ||W_K|| <= 5e-3; ||W_V|| ||W_O^T|| <= 6e-3
Chosen by hand to keep a single LibriSpeech transcript readable while removing as many vectors as possible (Section 4.4); no validation set or error metric is used.
MLP Up block size = 64 dimensions per block
The paper groups the 11008-dimensional Up layer into 172 blocks of 64 and applies orthogonal decomposition per block (Section 4.2, Appendix A.2); block size is an implementation choice that sets the parameter count.
GPT-2 pruning ratio per layer = 12.5% to 75% grid
The same pruning ratio is applied to every layer without per-layer selection (Section 4.1); results are reported across a manually chosen grid.

assumptions (4)

standard math SVD of the rank-at-most-d matrix W_Q W_K^T and W_V W_O^T reproduces the exact attention logits when no position-dependent rotation is present.
Invoked in Section 3 and Appendix A.1.
domain assumption Attention behavior is concentrated in the leading singular vectors of W_QK and W_VO, so deleting small singular directions is safe.
The central pruning premise; tested empirically in Sections 4.3-4.4 but not proven for general models.
domain assumption RoPE can be bypassed for Q-K by decomposing the Key projection alone while still calling the method cross-layer.
Used in Section 4.2 to adapt CLOVER to LLaMA; this replaces the cross-layer construction with an intra-layer one.
ad hoc to paper The middle factor S is allowed to become a dense d-by-d matrix during training, contradicting the word 'singular values'.
Appendix A.2 parameter counts show dense 128-by-128 and 64-by-64 blocks (1,753,088 parameters total), not diagonal matrices. This is an ad hoc modeling decision central to the parameter-efficiency and full-rank claims.

how reviews work

0 comments

Cite this review

Pith. "Pith review of CLOVER: Cross-Layer Orthogonal Vectors Pruning and Fine-Tuning." pith.science (2026). https://pith.science/paper/RNHH35TG

@misc{pith2026241117426,
  author       = {Pith},
  title        = {Pith review of: CLOVER: Cross-Layer Orthogonal Vectors Pruning and Fine-Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RNHH35TG}},
  note         = {Machine review of arXiv:2411.17426}
}

abstract

Decoder-only models generate tokens autoregressively by caching key/value vectors, but as the cache grows, inference becomes memory-bound. To address this issue, we introduce CLOVER (Cross-Layer Orthogonal Vectors), a novel approach that treats pairs of attention layers as a set of low-rank decompositions. CLOVER applies Singular Value Decomposition (SVD) to the $ Q $-$ K $ and $ V $-$ O $ pairs within each attention head. The resulting singular values can either guide pruning or serve as trainable parameters for efficient fine-tuning of all orthogonal vectors. After pruning or fine-tuning, these values are reintegrated into the model without increasing its parameter count. We apply CLOVER to various models, including GPT-2 XL, DeepSeek-V2-Lite, Whisper-Large-v3, Stable Diffusion XL, and LLaMA-3.2-11B-Vision. Our results demonstrate that CLOVER significantly improves pruning efficiency. For instance, the perplexity of pruning 70\% of the $ Q $-$ K $ pairs in GPT-2 XL is similar to that of pruning just 8\% with vanilla methods. Fine-tuning the singular values further results in a full-rank update, outperforming state-of-the-art methods (LoRA, DoRA, HiRA, and PiSSA) by 7.6\%, 5.5\%, 3.8\%, and 0.7\%, respectively, on eight commonsense tasks for LLaMA-2 7B.

Figures

Figures reproduced from arXiv: 2411.17426 by the authors.

**Figure 1.** (a) We treat the Query-Key and Value-Output layers within a single attention head as a unified structure. (b) Apply SVD to obtain two sets of singular vectors for initializing the Q-K and V-O layers, along with singular values that guide pruning or enable efficient full-rank fine-tuning. (c) This cross-layer orthogonalization strategy allows for higher pruning rates. (d) The pruned model maintains strong performanc… view at source ↗

**Figure 2.** CLOVER (orange) uses fewer orthogonal basis vectors than Vanilla Pruning (blue) to span the attention head space. The first row shows the importance of Q-K dimensions, and the second row shows V-O dimensions. After the red dot, CLOVER’s importance is lower, and pruning these vectors results in less performance loss. 4.3. CLOVER Removal Redundant Vectors CLOVER achieves a higher pruning ratio due to the significant … view at source ↗

**Figure 3.** An audio waveform from the librispeech dataset. Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel. Nor is Mr. Quilter’s manner less interesting than his matter. He tells us that at this festive season of the year, with Christmas and roast beef looming before us, similes drawn from eating and its results occur most readily to the mind. He has grave doubts whether Sir Frederick La… view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Proportion of data projections across different components in random directions (LoRA) versus orthogonal directions (PiSSA), as well as all orthogonal directions (CLOVER). on the principal singular values and vectors, leading to improved training performance. In cont…

**Figure 5.** Figure 5: ∆W is low rank in LoRA, while full rank for Full-FineTuning and CLOVER. 4.7. CLOVER Avoids Intrusive Dimensions Recent research (Shuttleworth et al., 2024) has highlighted an issue with LoRA, referred to as the “intrusive dimensions” phenomenon. As illustrated in Figu…

**Figure 6.** Figure 6: Intruder dimensions phenomenal in LoRA, which does not exist in Full Fine-Tuning and CLOVER. 5. Conclusion and Limitations In this paper, we introduce Cross-Layer Orthogonal Vectors (CLVOER), a method that orthogonalizes vectors within attention heads without requiring…

**Figure 7.** Figure 7: The L2-norm for the 0-th, 15-th, and 31-st attention layers in the Whisper-large-v3 encoder. The blue line represents the results after redundancy removal using the CLOVER method, while the orange line depicts the L2-norm directly computed for each dimension. 15 [PITH…

**Figure 8.** Figure 8: The L2-norm for the 0-th, 15-th, and 31-st attention layers in the ViT-bigG. The blue line represents the results after redundancy removal using the CLOVER method, while the orange line depicts the L2-norm directly computed for each dimension. 16 [PITH_FULL_IMAGE:figu…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 2 canonical work pages

[1]

J., Javaheripi, M., Kauffmann, P., et al

Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gunasekar, S., Harrison, M., Hewett, R. J., Javaheripi, M., Kauffmann, P., et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905,

arXiv
[4]

Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y ., Lebr´on, F., and Sanghai, S

URL https://github.com/meta-llama/llama3/ blob/main/MODEL_CARD.md. Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y ., Lebr´on, F., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head check- points. arXiv preprint arXiv:2305.13245,

arXiv
[6]

anthropic.com/news/claude-3-5-sonnet

URL https://www. anthropic.com/news/claude-3-5-sonnet . Asai, A., Salehi, M., Peters, M. E., and Hajishirzi, H. Attempt: Parameter-efficient multi-task tuning via at- tentional mixtures of soft prompts. arXiv preprint arXiv:2205.11961,

arXiv
[9]

D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901,

1901
[10]

URL https://doi.org/ 10.48550/arXiv.2307.09288

48550/arXiv.2307.09288. URL https://doi.org/ 10.48550/arXiv.2307.09288. AI@Meta. Llama 3 model card,

arXiv doi:10.48550/arxiv.2307.09288
[11]

Reproducible scaling laws for contrastive language-image learning

Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., and Jitsev, J. Reproducible scaling laws for contrastive language-image learning. arXiv preprint arXiv:2212.07143,

arXiv
[12]

Boolq: Exploring the surpris- ing difficulty of natural yes/no questions

Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surpris- ing difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044,

arXiv 1905
[15]

48550/arXiv.2405.04434

URL https://doi.org/10. 48550/arXiv.2405.04434. Dettmers, T., Lewis, M., Belkada, Y ., and Zettlemoyer, L. Gpt3. int8 (): 8-bit matrix multiplication for transform- ers at scale. Advances in Neural Information Processing Systems, 35:30318–30332,

arXiv doi:10.48550/arxiv.2405.04434

Show all 57 references

[16]

Gptq: Accurate post-training quantization for generative pre- trained transformers

Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre- trained transformers. arXiv preprint arXiv:2210.17323,

arXiv
[17]

Lazyllm: Dynamic token pruning for efficient long context llm inference

Fu, Q., Cho, M., Merth, T., Mehta, S., Rastegari, M., and Najibi, M. Lazyllm: Dynamic token pruning for efficient long context llm inference. arXiv preprint arXiv:2407.14057,

arXiv
[19]

and Dao, T

Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752,

arXiv
[20]

M., and Kim, Y

Guo, D., Rush, A. M., and Kim, Y . Parameter-efficient transfer learning with diff pruning. arXiv preprint arXiv:2012.07463,

2012 arXiv
[21]

L., and Yang, M

Guo, S., Xu, J., Zhang, L. L., and Yang, M. Com- presso: Structured pruning with collaborative prompting learns compact large language models. arXiv preprint arXiv:2310.05015,

arXiv
[22]

Warp: Word-level adversarial reprogramming

Hambardzumyan, K., Khachatrian, H., and May, J. Warp: Word-level adversarial reprogramming. arXiv preprint arXiv:2101.00121,

arXiv
[24]

J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W

Hu, E. J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685,

arXiv
[25]

Mora: High-rank updating for parameter-efficient fine-tuning

Jiang, T., Huang, S., Luo, S., Zhang, Z., Huang, H., Wei, F., Deng, W., Sun, F., Zhang, Q., Wang, D., et al. Mora: High-rank updating for parameter-efficient fine-tuning. arXiv preprint arXiv:2405.12130,

arXiv
[26]

and Shin, D

Jo, H.-r. and Shin, D. A2sf: Accumulative attention scoring with forgetting factor for token pruning in transformer decoder. arXiv preprint arXiv:2407.20485,

arXiv
[27]

Lawton, N., Kumar, A., Thattai, G., Galstyan, A., and Steeg, G. V . Neural architecture search for parameter-efficient fine-tuning of large pre-trained language models. arXiv preprint arXiv:2305.16597,

arXiv
[28]

The power of scale for parameter-efficient prompt tuning

Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691,

arXiv
[29]

Svdqunat: Absorb- ing outliers by low-rank components for 4-bit diffusion models

Li, M., Lin, Y ., Zhang, Z., Cai, T., Li, X., Guo, J., Xie, E., Meng, C., Zhu, J.-Y ., and Han, S. Svdqunat: Absorb- ing outliers by low-rank components for 4-bit diffusion models. arXiv preprint arXiv:2411.05007, 2024a. Li, X. L. and Liang, P. Prefix-tuning: Optimizing contin...
[30]

Snapkv: Llm knows what you are looking for before generation

Li, Y ., Huang, Y ., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P., and Chen, D. Snapkv: Llm knows what you are looking for before generation. arXiv preprint arXiv:2404.14469, 2024b. Lingam, V ., Tejaswi, A., Vavre, A., Shetty, A., Gudur, G. K., Ghosh, J.,...

arXiv
[31]

Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., Guo, D., et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024a. Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu,...

arXiv
[32]

Pissa: Principal singular values and singular vectors adaptation of large language models

Meng, F., Wang, Z., and Zhang, M. Pissa: Principal singular values and singular vectors adaptation of large language models. arXiv preprint arXiv:2404.02948,

arXiv
[34]

com/index/hello-gpt-4o/

URL https://openai. com/index/hello-gpt-4o/. Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Grella, M., et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048,

arXiv
[35]

Sdxl: Im- proving latent diffusion models for high-resolution image synthesis

Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., M ¨uller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952,

arXiv
[36]

A., and Lewis, M

Press, O., Smith, N. A., and Lewis, M. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409,

arXiv
[37]

Socialiqa: Commonsense reasoning about social interac- tions

Sap, M., Rashkin, H., Chen, D., LeBras, R., and Choi, Y . Socialiqa: Commonsense reasoning about social interac- tions. arXiv preprint arXiv:1904.09728,

1904 arXiv
[38]

Fast transformer decoding: One write-head is all you need

Shazeer, N. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150,

1911 arXiv
[39]

Lora vs full fine-tuning: An illusion of equivalence

Shuttleworth, R., Andreas, J., Torralba, A., and Sharma, P. Lora vs full fine-tuning: An illusion of equivalence. arXiv preprint arXiv:2410.21228,
[40]

Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695,

arXiv
[41]

You only cache once: Decoder-decoder architectures for language models

Sun, Y ., Dong, L., Zhu, Y ., Huang, S., Wang, W., Ma, S., Zhang, Q., Wang, J., and Wei, F. You only cache once: Decoder-decoder architectures for language models. arXiv preprint arXiv:2405.05254,

arXiv
[42]

I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al

Team, G., Georgiev, P., Lei, V . I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024a. Team, G., Riviere, M., Patha...

arXiv
[43]

Spot: Better frozen model adaptation through soft prompt transfer

Vu, T., Lester, B., Constant, N., Al-Rfou, R., and Cer, D. Spot: Better frozen model adaptation through soft prompt transfer. arXiv preprint arXiv:2110.07904,

arXiv
[44]

Z., Khabsa, M., Fang, H., and Ma, H

Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768,

2006 arXiv
[45]

Lora-ga: Low-rank adap- tation with gradient approximation

11 CLOVER: Cross-Layer Orthogonal Vectors Wang, S., Yu, L., and Li, J. Lora-ga: Low-rank adap- tation with gradient approximation. arXiv preprint arXiv:2407.05000, 2024a. Wang, Z., Panda, R., Karlinsky, L., Feris, R., Sun, H., and Kim, Y . Multitask prompt tuning enables param...

arXiv
[46]

Lora- pro: Are low-rank adapters properly optimized? arXiv preprint arXiv:2407.18242, 2024b

Wang, Z., Liang, J., He, R., Wang, Z., and Tan, T. Lora- pro: Are low-rank adapters properly optimized? arXiv preprint arXiv:2407.18242, 2024b. Xia, M., Gao, T., Zeng, Z., and Chen, D. Sheared llama: Accelerating language model pre-training via structured pruning. arXiv prepri...

arXiv
[47]

Raise a child in large language model: Towards effective and generalizable fine-tuning

Xu, R., Luo, F., Zhang, Z., Tan, C., Chang, B., Huang, S., and Huang, F. Raise a child in large language model: Towards effective and generalizable fine-tuning. arXiv preprint arXiv:2109.05687,

arXiv
[48]

Effectively com- press kv heads for llm

Yu, H., Yang, Z., Li, S., Li, Y ., and Wu, J. Effectively com- press kv heads for llm. arXiv preprint arXiv:2406.07056,

arXiv
[49]

Bridging the gap between low- rank and orthogonal adaptation via householder reflection adaptation

Yuan, S., Liu, H., and Xu, H. Bridging the gap between low- rank and orthogonal adaptation via householder reflection adaptation. arXiv preprint arXiv:2405.17484,

arXiv
[50]

B., Ravfogel, S., and Goldberg, Y

Zaken, E. B., Ravfogel, S., and Goldberg, Y . Bitfit: Simple parameter-efficient fine-tuning for transformer- based masked language-models. arXiv preprint arXiv:2106.10199,
[51]

Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,

Zellers, R., Holtzman, A., Bisk, Y ., Farhadi, A., and Choi, Y . Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,

1905 arXiv
[52]

Adalora: Adaptive budget allocation for parameter-efficient fine- tuning

Zhang, Q., Chen, M., Bukharin, A., Karampatziakis, N., He, P., Cheng, Y ., Chen, W., and Zhao, T. Adalora: Adaptive budget allocation for parameter-efficient fine- tuning. arXiv preprint arXiv:2303.10512,

arXiv
[53]

Galore: Memory-efficient llm train- ing by gradient low-rank projection

Zhao, J., Zhang, Z., Chen, B., Wang, Z., Anandkumar, A., and Tian, Y . Galore: Memory-efficient llm train- ing by gradient low-rank projection. arXiv preprint arXiv:2403.03507,

arXiv
[54]

Mask- ing as an efficient alternative to finetuning for pretrained language models

Zhao, M., Lin, T., Mi, F., Jaggi, M., and Sch¨utze, H. Mask- ing as an efficient alternative to finetuning for pretrained language models. arXiv preprint arXiv:2004.12406,

2004 arXiv
[55]

Delta-lora: Fine-tuning high-rank parame- ters with the delta of low-rank matrices

Zi, B., Qi, X., Wang, L., Wang, J., Wong, K.-F., and Zhang, L. Delta-lora: Fine-tuning high-rank parame- ters with the delta of low-rank matrices. arXiv preprint arXiv:2309.02411,

arXiv
[56]

Zuhri, Z. M. K., Adilazuarda, M. F., Purwarianti, A., and Aji, A. F. Mlkv: Multi-layer key-value heads for memory efficient transformer decoding. arXiv preprint arXiv:2406.09297,

arXiv
[57]

Both PiSSA and CLOVER exhibit stable training performance

To adjust the number of learnable parameters, CLOVER can either span multiple heads or split a single head into multiple blocks. Both PiSSA and CLOVER exhibit stable training performance. Therefore, instead of validating every 80 steps, we omit frequent validation, improving t...

2023
[58]

WinoGrande (Sakaguchi et al., 2021)40,398 1,267 Fill-in-the-blank task with binary options

39,905 10,042 Commonsense NLI questions with context and endings. WinoGrande (Sakaguchi et al., 2021)40,398 1,267 Fill-in-the-blank task with binary options. ARC-e (Clark et al.,

2021
[2016]

Can a suit of armor conduct electricity? a new dataset for open book question answering

Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789,

arXiv
[2018]

L., Fernando, A., Botev, A., Cristian- Muraru, G., Gu, A., Haroun, R., Berrada, L., Chen, Y ., Srinivasan, S., et al

De, S., Smith, S. L., Fernando, A., Botev, A., Cristian- Muraru, G., Gu, A., Haroun, R., Berrada, L., Chen, Y ., Srinivasan, S., et al. Griffin: Mixing gated linear recur- rences with local attention for efficient language models. arXiv preprint arXiv:2402.19427,

arXiv
[2019]

Think you have solved question answering? try arc, the ai2 reasoning challenge

Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457,

arXiv
[2020]

Brandon, W., Mishra, M., Nrusimha, A., Panda, R., and Kelly, J. R. Reducing transformer key-value cache size with cross-layer attention. arXiv preprint arXiv:2405.12981,

arXiv
[2021]

W., Shao, Y

Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, Y . S., Keutzer, K., and Gholami, A. Kvquant: Towards 10 million context length llm in- ference with kv cache quantization. arXiv preprint arXiv:2401.18079,

arXiv
[2022]

L., Nascimento, M

Ashkboos, S., Croci, M. L., Nascimento, M. G. d., Hoefler, T., and Hensman, J. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401.15024,

arXiv
[2023]

9 CLOVER: Cross-Layer Orthogonal Vectors Gandhi, S., von Platen, P., and Rush, A. M. Distil-whisper: Robust knowledge distillation via large-scale pseudo la- belling. arXiv preprint arXiv:2311.00430,

arXiv
[2024]

Llama 2: Open foundation and fine-tuned chat models

AI@Meta. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288,

arXiv
[2025]

net/forum?id=TwJrTz9cRS

URL https://openreview. net/forum?id=TwJrTz9cRS. Ansell, A., Ponti, E. M., Korhonen, A., and Vuli ´c, I. Composable sparse fine-tuning for cross-lingual trans- fer. arXiv preprint arXiv:2110.07560,

arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

J., Javaheripi, M., Kauffmann, P., et al

Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gunasekar, S., Harrison, M., Hewett, R. J., Javaheripi, M., Kauffmann, P., et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905,

arXiv

[2] [4]

Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y ., Lebr´on, F., and Sanghai, S

URL https://github.com/meta-llama/llama3/ blob/main/MODEL_CARD.md. Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y ., Lebr´on, F., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head check- points. arXiv preprint arXiv:2305.13245,

arXiv

[3] [6]

anthropic.com/news/claude-3-5-sonnet

URL https://www. anthropic.com/news/claude-3-5-sonnet . Asai, A., Salehi, M., Peters, M. E., and Hajishirzi, H. Attempt: Parameter-efficient multi-task tuning via at- tentional mixtures of soft prompts. arXiv preprint arXiv:2205.11961,

arXiv

[4] [9]

D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901,

1901

[5] [10]

URL https://doi.org/ 10.48550/arXiv.2307.09288

48550/arXiv.2307.09288. URL https://doi.org/ 10.48550/arXiv.2307.09288. AI@Meta. Llama 3 model card,

arXiv doi:10.48550/arxiv.2307.09288

[6] [11]

Reproducible scaling laws for contrastive language-image learning

Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., and Jitsev, J. Reproducible scaling laws for contrastive language-image learning. arXiv preprint arXiv:2212.07143,

arXiv

[7] [12]

Boolq: Exploring the surpris- ing difficulty of natural yes/no questions

Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surpris- ing difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044,

arXiv 1905

[8] [15]

48550/arXiv.2405.04434

URL https://doi.org/10. 48550/arXiv.2405.04434. Dettmers, T., Lewis, M., Belkada, Y ., and Zettlemoyer, L. Gpt3. int8 (): 8-bit matrix multiplication for transform- ers at scale. Advances in Neural Information Processing Systems, 35:30318–30332,

arXiv doi:10.48550/arxiv.2405.04434