Pith. sign in

REVIEW 4 major objections 6 minor 31 references

Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQA

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that aligning attention heads with Procrustes rotations before merging them lets multi-head attention be converted to grouped-query attention at up to 87.5 percent KV-head compression with acceptable accuracy loss.

desk verdict Procrustes alignment before L0 head merging is a real, mathematically sound trick with promising results, but the missing comparison to the closest prior work and untested calibration sensitivity keep me from fully trusting the effect size. read the letter →

arxiv 2412.20677 v2 pith:WQ4Z65EX submitted 2024-12-30 cs.CL

Qingyun Jin , Xiaohui Song , Feng Zhou , Zengchang Qin This is my paper · ORCID
classification cs.CL
keywords grouped-queryattentionKVcachecompressionProcrustesanalysisL0regularizationheadmergingmulti-headorthogonaltransformationLLMinferenceefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to establish that a pretrained multi-head attention (MHA) transformer can be converted into a grouped-query attention (GQA) transformer at almost any key-value (KV) head compression ratio through a two-stage procedure: a cheap, output-preserving alignment of attention heads followed by sparsifying fine-tuning. The alignment stage uses generalized Procrustes analysis to find orthogonal rotations that make the key and value caches of heads within the same group as similar as possible, and fuses those rotations into the projection matrices so the model's output is unchanged. The fine-tuning stage then uses L0-regularized masks to transfer the original heads into shared KV heads, producing a model that fits the standard GQA framework. On LLaMA2-7B the method reaches 87.5 percent KV-head compression and on Sheared-LLaMA-1.3B 75 percent, with what the authors describe as acceptable average accuracy loss relative to the MHA teacher. The practical stake is that KV-cache memory and inference cost can be cut sharply without the expensive uptraining that earlier MHA-to-GQA conversions required.

What carries the argument

The load-bearing mechanism is the orthogonal Procrustes rotation fused into projection matrices, paired with L0 masks for head transfer. Generalized Procrustes analysis rotates each head's KV cache toward the group mean; fusing the resulting orthogonal matrix into the value/output pair, or in RoPE-compatible block-diagonal form (2D rotations per coordinate pair) into the key/query pair, preserves the attention output exactly. This changes the geometry of the KV caches so that heads in a group are nearly parallel, turning what would otherwise be a large pruning step into a small perturbation. A simulated-annealing search over head groupings, scored by post-alignment cosine similarity or Euclidean distance, determines which heads should share a KV head.

What would settle it

Run the same MHA-to-GQA conversion twice, once with rotations computed from the paper's C4 calibration caches and once with rotations computed from an out-of-domain corpus such as code or mathematics. If the out-of-domain rotations erase or reverse the accuracy gain over the no-alignment baseline, the central claim that the alignment itself causes the improvement is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that the near-orthogonality of KV caches across attention heads is not fixed: because the caches are low-rank, a per-head orthogonal rotation can make them nearly parallel without changing the attention computation. For value heads, the paper computes the optimal rotation aligning one cache to another, or iteratively to a group mean, from the SVD of their cross-covariance, and fuses the rotation into the value and output projection matrices. For key and query heads, the rotation is restricted to a block-diagonal form of 2D rotations so that it commutes with rotary position embeddings, and it is fused into the key and query projections. The transformed model is mathematically identical to the original MHA, but its heads within each group are much more similar, so mean-pooling initialization plus L0 masks can transfer all original heads into shared KV heads with less damage. The paper reports that transformed and regrouped models consistently beat direct-pruning baselines on the tested benchmarks, and that the gap grows as the compression ratio increases.

Load-bearing premise

The whole alignment step rests on the 128 C4 sequences of 2048 tokens used for calibration being representative of the model's deployment distribution; if those caches are not representative, the computed rotations are miscalibrated and the advantage over direct pruning disappears.

Editorial extensions

If this is right

  • The conversion is not locked to preset GQA group sizes: L0 masks can transfer original heads to shared heads at any target ratio, so practitioners can choose a KV-head budget freely.
  • The resulting model is a standard GQA model, meaning it can be served by existing GQA inference kernels and frameworks without custom attention code.
  • The alignment stage is cheap relative to training: calibration and transformation take at most about an hour on one A100 GPU for the tested models, according to the paper's appendix.
  • Because the rotation is output-invariant before pruning, the same alignment can be combined with other fine-tuning or distillation objectives, not only the L0-plus-BiLD setup used here.
  • At 87.5 percent KV-head compression on LLaMA2-7B and 75 percent on Sheared-LLaMA-1.3B, converted models retain average accuracy close to the MHA teacher on the eight commonsense benchmarks reported.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the alignment is doing the work the paper attributes to it, the same rotations should transfer across fine-tuned variants of a base model, since they are computed only from KV caches; that would make the conversion a one-time per-architecture transformation rather than a per-checkpoint procedure.
  • The comparable results for cosine-similarity and Euclidean-distance criteria suggest the exact objective matters less than the fact of rotating heads into a common frame; a cheaper heuristic such as aligning each head to the group centroid once might capture most of the gain.
  • A direct extension is to test alignment on out-of-domain calibration data; persistent gains would indicate robustness, while vanishing gains would reveal calibration distribution as a hidden hyperparameter.
  • Because the pre-pruning transformation is output-invariant, the same head-alignment trick could benefit other head-merging schemes, such as per-layer asymmetric key/value sharing, independent of L0 pruning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a two-stage method for converting a pre-trained multi-head attention (MHA) model into a grouped-query attention (GQA) model. In the first stage, the authors collect KV caches from a small calibration set (128 C4 sequences of 2048 tokens), compute pairwise similarity scores between key heads and between value heads under an orthogonal Procrustes alignment, group heads by simulated-annealing search to maximize intra-group similarity, and then fuse the resulting orthogonal rotations into the query, key, and value projection matrices so that the model output is exactly preserved. In the second stage, L0 regularization with a hard-concrete mask is used to gradually transfer each original KV head to a new shared head initialized by mean-pooling within each group, with logits distillation from the fine-tuned MHA teacher. The final model is a standard GQA model. Experiments are reported on LLaMA2-7B and Sheared-LLaMA-1.3B for GQA-16, GQA-8, and GQA-4 configurations, evaluated on eight commonsense QA datasets. The authors claim the method compresses up to 87.5% of LLaMA2-7B KV heads and 75% of Sheared-LLaMA-1.3B KV heads with acceptable performance degradation.

Significance. If the central claim is correct, the paper offers a practical recipe for converting MHA checkpoints to GQA at arbitrary (divisor-based) compression ratios, with a computationally cheap alignment step and a training procedure based on L0 pruning and distillation. The output-invariance argument for the value-side transformation (Eqs. 12-13) and for the RoPE-compatible key-side transformation (Eqs. 14-17) is mathematically sound, and the release of code is a strength. The generalized Procrustes formulation and the use of calibration statistics to guide grouping are reasonable. However, the current evidence is insufficient to establish that the proposed alignment step is the cause of the reported gains: there are no comparisons against existing MHA-to-GQA conversion methods, no sensitivity analysis for the calibration set, and no multi-seed or error-bar information on a very narrow evaluation suite. The contribution is potentially valuable, but its robustness and incremental value over prior work remain unproven.

major comments (4)
  1. [4.1, Figure 2, Tables 1-2] The entire alignment step depends on rotations and groupings computed from a single calibration sample of 128 C4 sequences (262K tokens), yet the paper provides no sensitivity analysis over calibration sets, no held-out validation of the alignment, and Figure 2 reports cosine-similarity gains only on the calibration set itself. If the learned rotations do not transfer to the deployment distribution (for example, after fine-tuning on the target tasks), the advantage over direct pruning could disappear. This risk is not merely hypothetical: in Table 1, GQA-8 with grouping-by-value cos reaches 80.32 average versus 81.65 for the no-transformation baseline, and Section 4.4 concedes that the transformation can hurt due to cumulative errors. The limitations paragraph stating that the method relies on statistical mathematical features of attention heads does not address calibration robustness. I request a sensitivity study with different calibration subsamples and a verification that the computed rotations improve mergeability on held-out data, not only on the calibration set.
  2. [4 (Main results) and 2.2 (Related Works)] The experiments compare only against the authors' own baseline of L0 pruning without transformation. No comparison is made to established MHA-to-GQA conversion methods, such as mean-pooling followed by uptraining (Ainslie et al., 2023), the principal-component-based method of Yu et al. (2024), or the grouped-query conversion methods cited in Section 2.2 (Chen et al., 2024; Chen et al., 2024a). As a result, the paper cannot support its central claim that the proposed alignment step is an effective way to convert MHA to GQA or that the overall pipeline is cost-effective; a reader cannot tell whether the reported gains are due to the Procrustes alignment, the L0-plus-distillation training budget, or other hyperparameters. I request at least one standard MHA-to-GQA baseline at matched token budgets and evaluation conditions.
  3. [Tables 1-2 and 4.1] The empirical evidence is too narrow to support the abstract's claims of acceptable performance degradation at 87.5% and 75% KV-head compression. All results are single runs on eight small commonsense QA datasets, with no error bars, no seeds, no language-modeling perplexity, and no generation or multitask benchmarks such as MMLU or GSM8K. Given the test-set sizes in Appendix B (e.g., ARC-C 299, OpenbookQA 500, WinoGrande 1267), average-accuracy differences of 0.5-1.5 points are within noise. The paper should report multiple seeds or confidence intervals and ideally include a broader evaluation, including held-out perplexity and a larger multitask benchmark, before claims about acceptable degradation can be accepted.
  4. [Eq. (14)-(17), Section 3.3] The key-side alignment is under-specified. The paper states that due to RoPE, the orthogonal matrix should be block-diagonal with 2D rotation blocks, and that one can apply Procrustes analysis in every two dimensions just like RoPE. It does not derive the constrained Procrustes solution or prove that solving independent per-subspace problems minimizes the Frobenius loss over block-diagonal rotations. The output-invariance derivation in Eq. (17) is correct once such an R is chosen, but it does not address optimality of the chosen R for the alignment objective. Please provide the constrained optimization (objective and SVD per 2D block) or a reference that justifies this step.
minor comments (6)
  1. [3.4, Eq. (23)-(24)] The method as described assumes equal-size groups with D = H/G, which requires G to divide H; the paper tests only G = 16, 8, 4 for LLaMA2-7B. The abstract's claim of 'any compression ratio' should be qualified to avoid overclaiming.
  2. [3.5, Eq. (27)-(28)] The paper does not specify how the continuous hard-concrete masks are discretized after training, nor what happens if some masks remain nonzero when the training ends; a precise post-processing step is needed for reproducibility.
  3. [3.5, Eq. (29)] The target size T is described both as a target size and as something that equals zero after sparsity warm-up steps; the annealing schedule is only sketched in Appendix A. Please clarify the relationship between T, the number of retained heads, and the actual mask values during and after training.
  4. [Algorithm 2] The algorithm only accepts strict improvements and has no temperature-based acceptance probability, so it is random-restart hill climbing rather than simulated annealing; the nomenclature should be adjusted or the algorithm should be changed to match the name.
  5. [Throughout] There are several typographical and formatting issues: 'theL0 method' in Section 2.1, 'SIQAAvg.' in the Table 1 header, 'Sheared-llama-1.3B' in Table 6, and inconsistent capitalization of 'Sheared-LLaMA' elsewhere. These should be corrected in a revision.
  6. [References] The generalized Procrustes analysis is cited via a Wikipedia article; please cite the original methodology literature (e.g., Gower's work on generalized Procrustes analysis) for a citable and verifiable source.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the alignment and pruning pipeline is evaluated on held-out benchmarks, and the only self-citation is an auxiliary loss that is not load-bearing.

full rationale

The derivation chain is self-contained. The orthogonal rotations are computed from unlabeled C4 calibration KV caches via Procrustes analysis (Eqs. 9-16), the grouping objective is the sum of pairwise cache similarities (Eqs. 23-24), and the L0 masks transfer original heads to mean-pooled group heads (Eqs. 25-28) with a user-specified target size T in Eq. 29. None of these fitted quantities is renamed as a prediction: downstream claims are measured on held-out benchmark accuracy in Tables 1 and 2, not on the calibration similarity objective. The only same-group citation is BiLD (Li et al., 2024), used as an auxiliary distillation loss alongside KL loss; it is not load-bearing and no uniqueness theorem or ansatz is imported from it. The absence of calibration-set sensitivity analysis is a robustness concern, not circularity, because the method is not statistically forced by any evaluation-label fit.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The method introduces no new physical or architectural entities. Its free parameters are experimental choices, and its main conceptual commitments are the representativeness of calibration data and the validity of the similarity-score proxy, plus an under-specified constrained Procrustes step for key heads.

free parameters (3)
  • Target KV-head retention T = 0.5, 0.25, 0.125 (LLaMA2-7B); 0.5, 0.25 (Sheared-LLaMA-1.3B)
    User-specified compression ratio used in the L0 loss (Eq. 29); chosen per experiment, not fitted to evaluation labels.
  • Simulated annealing iterations (maxIter, epoch) = not reported
    Controls grouping search quality (Algorithm 2); no sensitivity analysis is provided.
  • Mask learning rate = 1e-2
    Initial learning rate for L0 masks (Appendix A); training hyperparameter, not fitted to test performance.
assumptions (6)
  • standard math The orthogonal Procrustes problem is solved by an SVD of the cross-covariance matrix (Eq. 4-6).
    Classical result from Schönemann (1966), used in Section 3.2.
  • standard math For 2D rotations in the same subspace, R_a^T R_b R_a = R_b, so RoPE inner products are preserved under block-diagonal rotation (Eq. 17).
    Required for the key-side invariance argument in Section 3.3.
  • domain assumption KV caches are low-rank, so orthogonal alignment can increase the similarity of heads without losing information.
    Motivation in Section 3.1, citing Yu et al. (2024).
  • domain assumption 128 C4 sequences of 2048 tokens are a representative calibration set for computing head alignments.
    Used in Section 4.1; no robustness check is reported.
  • domain assumption Maximizing pairwise cosine similarity or negative Euclidean distance among KV caches within a group is a valid proxy for post-pruning model quality.
    The paper's own footnote 1 concedes this is only 'acceptable when considering computation and time costs'; the proxy is not shown to track final accuracy.
  • ad hoc to paper For keys, the Procrustes solution can be restricted to RoPE-compatible block-diagonal rotations without derivation of the constrained optimum.
    Algorithm 1 outputs general orthogonal matrices, while Eq. 14 requires block-diagonal 2D rotations; the restriction is asserted, not derived.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQA." pith.science (2026). https://pith.science/paper/WQ4Z65EX

@misc{pith2026241220677,
  author       = {Pith},
  title        = {Pith review of: Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQA},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WQ4Z65EX}},
  note         = {Machine review of arXiv:2412.20677}
}
abstract

Large language models (LLMs) have demonstrated exceptional performance across diverse natural language processing tasks. However, as the model size and the input sequence's length increase, the linearly increasing key-value (KV) cache significantly degrades inference throughput. Therefore, grouped-query attention (GQA), as an alternative to multi-head attention (MHA), has been widely introduced into LLMs. In this work, we propose a cost-effective method for converting MHA into GQA with any compression ratio of KV heads. The key point of our method lies in the application of Procrustes analysis to the attention heads, which enhances the similarity among attention heads while preserving computational invariance, thereby improving the model's post-training performance. Subsequently, we employ $\mathit{L_0}$ regularization to prune redundant parameters. The model after pruning can be adapted to the standard GQA framework. Experimental results show that our strategy can compress up to 87.5\% KV heads of LLaMA2-7B model and 75\% KV heads of Sheared-LLaMA-1.3B with acceptable performance degradation. Our code is released at https://github.com/fpcsong/mha2gqa.

Figures

Figures reproduced from arXiv: 2412.20677 by the authors.

Figure 1
Figure 1. An illustration of our method. 1. In each attention layer of LLM, after the average similarity score [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. This figure shows the average cosine similarity of key and value caches between any two heads before and [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The shared sparsity of L0 masks across blocks allows different pruning speeds for different blocks in LLaMA2-7B, leading to a more stable training process. Attention heads closer to the input layer are pruned last, as pruning these heads significantly impacts all subsequent layers. 5 Conclusion In this paper, we propose a general method for con￾verting an MHA model into a GQA model with any compression ratio of KV h… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 5 canonical work pages

  1. [5]

    Boolq: Exploring the surprising diffi- culty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising diffi- culty of natural yes/no questions. arXiv preprint arXiv:1905.10044,

    arXiv 1905

  2. [6]

    datasets train test BoolQ 9427 3270 PIQA 16113 1838 HellaSwag 39905 10042 WinoGrande 40398 1267 ARC-C 1119 299 ARC-E 2251 570 OpenbookQA 4957 500 SIQA 33410 1954 total 147580 19740 Table 4: Sizes of different datasets The template of each dataset can be seen in Table

    work page 1954

  3. [9]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    URL https://openreview.net/ forum?id=0LXotew9Du. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685,

    arXiv

  4. [10]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825,

    arXiv

  5. [11]

    Bild: Bi- directional logits difference loss for large language model distillation

    9 Minchong Li, Feng Zhou, and Xiaohui Song. Bild: Bi- directional logits difference loss for large language model distillation. arXiv preprint arXiv:2406.13555,

    arXiv

  6. [12]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024a. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zh...

    arXiv

  7. [13]

    Learning sparse neural networks through l_0 regular- ization

    Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through l_0 regular- ization. arXiv preprint arXiv:1712.01312,

    arXiv

  8. [15]

    Socialiqa: Commonsense reasoning about social interactions

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728,

    arXiv 1904

Show all 31 references
  1. [17]

    Recursive deep models for se- mantic compositionality over a sentiment treebank

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for se- mantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empiri- cal methods in natural language p...

    2013

  2. [18]

    A simple and effective pruning approach for large language models

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695,

    arXiv

  3. [19]

    Razoratten- tion: Efficient kv cache compression through retrieval heads

    Hanlin Tang, Yang Lin, Jing Lin, Qingsen Han, Shikuan Hong, Yiwu Yao, and Gongyi Wang. Razoratten- tion: Efficient kv cache compression through retrieval heads. arXiv preprint arXiv:2407.15891,

    arXiv

  4. [20]

    Llama 2: Open foundation and fine- tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti 10 Bhosale, et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288,

    arXiv

  5. [21]

    neurips.cc/paper_files/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper

    URL https://proceedings. neurips.cc/paper_files/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper. pdf. Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Struc- tured pruning of large language models. arXiv preprint arXiv:1910.04732,

    2017 arXiv

  6. [23]

    Sheared llama: Accelerating language model pre-training via structured pruning

    Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared llama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.06694,

    arXiv

  7. [24]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024a. June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Eunho Yang, S...

    arXiv

  8. [25]

    Effectively compress kv heads for llm

    Hao Yu, Zelan Yang, Shen Li, Yong Li, and Jianxin Wu. Effectively compress kv heads for llm. arXiv preprint arXiv:2406.07056,

    arXiv

  9. [26]

    Wkvquant: Quantizing weight and key/value cache for large language models gains more

    Yuxuan Yue, Zhihang Yuan, Haojie Duanmu, Sifan Zhou, Jianlong Wu, and Liqiang Nie. Wkvquant: Quantizing weight and key/value cache for large language models gains more. arXiv preprint arXiv:2402.12065,

    arXiv

  10. [27]

    Hellaswag: Can a ma- chine really finish your sentence? arXiv preprint arXiv:1905.07830,

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence? arXiv preprint arXiv:1905.07830,

    1905 arXiv

  11. [28]

    Tinyllama: An open-source small language model

    Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385,

    arXiv

  12. [29]

    During the pruning training process, the sparsity warm-up steps account for 30% of the total steps, during which the target size of the L0 masks decreases linearly to zero

    11 A Hyperparameter settings To reduce memory usage, we employ DeepSpeed during both SFT and pruning training, we set k=16 for BiLD loss (Li et al., 2024). During the pruning training process, the sparsity warm-up steps account for 30% of the total steps, during which the targ...

    2024

  13. [31]

    12 Dataset Template Arc-C Arc-E OpenbookQA Which color shirt will reflect the most light on a hot, sunny day? Choices: [’black’, ’blue’, ’red’, ’white’] Answer: HellaSwag Please choose the most appropriate text to complete the passage below: Passage: A male athlete puts powder...

    2017

  14. [1966]

    Fast transformer decoding: One write-head is all you need

    Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150,

    1911 arXiv

  15. [2013]

    Dha: Learning decoupled-head attention from trans- former checkpoints via adaptive heads fusion

    Yilong Chen, Linhao Zhang, Junyuan Shang, Zhenyu Zhang, Tingwen Liu, Shuohuan Wang, and Yu Sun. Dha: Learning decoupled-head attention from trans- former checkpoints via adaptive heads fusion. arXiv preprint arXiv:2406.06567, 2024a. Yuang Chen, Cheng Zhang, Xitong Gao, Robert ...

    arXiv

  16. [2017]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789,

    arXiv

  17. [2018]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    arXiv

  18. [2019]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457,

    arXiv

  19. [2020]

    Lan- guage models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey ...

    1901

  20. [2021]

    Measuring massive multitask language understand- ing

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understand- ing. arXiv preprint arXiv:2009.03300,

    2009 arXiv

  21. [2022]

    [Online; accessed 24-October-2024]

    URL https://en.wikipedia.org/w/ index.php?title=Generalized_Procrustes_ analysis&oldid=1126373270. [Online; accessed 24-October-2024]. Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Struc- tured pruning learns compact and accurate models. arXiv preprint arXiv:2204.00408,

    2024 arXiv

  22. [2023]

    Slicegpt: Compress large language mod- els by deleting rows and columns

    Saleh Ashkboos, Maximilian L Croci, Marcelo Gen- nari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language mod- els by deleting rows and columns. arXiv preprint arXiv:2401.15024,

    arXiv

  23. [2024]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245,

    arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.