Merging Feed-Forward Sublayers for Compressed Transformers

Kenton Murray; Kevin Duh; Neha Verma

REVIEW 2 major objections 6 minor 55 references

Merging Feed-Forward Sublayers for Compressed Transformers

T0 review · 2 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adjacent feed-forward sublayers in trained Transformers can be aligned, averaged, and tied into one shared sublayer, removing over a third of feed-forward layers while keeping performance after recovery fine-tuning.

desk verdict A clean post-training compression method that merges feed-forward sublayers via permutation alignment; the main weakness is uneven baselines for two of three tasks, but the core result holds. read the letter →

arxiv 2501.06126 v2 pith:HJDTGOYF submitted 2025-01-10 cs.CL cs.LG

Neha Verma , Kenton Murray , Kevin Duh This is my paper · ORCID

classification cs.CLcs.LG

keywords modelcompressionweighttyingfeed-forwardsublayerspermutationalignmenttransformersvisionlayerpruningquantization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a large part of a trained Transformer is redundant at the level of whole feed-forward sublayers, and that redundancy can be exploited by merging instead of pruning. The method aligns the neurons of adjacent sublayers with permutations, averages the aligned weights, and ties them so that several sublayers occupy the memory of one. Across language modeling, image classification, and machine translation, the paper reports performance close to the original models after merging more than a third of feed-forward sublayers, and better performance than a strong layer-pruning baseline at matched compression ratios. It also reports that feed-forward sublayer activations show clear blocks of high similarity, which it offers as evidence that the mergeability is grounded in the models' actual behavior.

What carries the argument

The load-bearing device is the permutation-aligned sublayer merge. For a window of $k$ adjacent feed-forward sublayers, the first sublayer is the anchor; the paper collects activations just before the nonlinearity, computes a cross-correlation matrix between the anchor and each other sublayer, solves the resulting linear assignment problem for the best neuron permutation, applies that permutation to the input weights and bias and its inverse to the output weights of each non-anchor sublayer, then averages the $k$ parameter sets and ties them so they are stored once. A sliding window over all possible contiguous groups, evaluated on validation data, chooses which sublayers to merge, and a short recovery fine-tuning step restores task performance. For SwiGLU feed-forward blocks the same recipe operates on the gated-product features.

What would settle it

Fine-tune the original ViT and OPUS-MT models on the same data used for recovery fine-tuning and measure their accuracy and BLEU against the merged-and-fine-tuned models at the same compression ratio; if the fine-tuned originals outperform the merged models by more than the reported 1 accuracy-point or 2 BLEU-point gaps, the claim of comparable performance to the original models does not hold for those settings.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that post-training merge-and-tie compression of feed-forward sublayers works across decoder-only, encoder, and encoder-decoder Transformers. For example, merging four of twelve ViT feed-forward sublayers and fine-tuning removes over 21% of total parameters while keeping 99% of original ImageNet accuracy; merging twelve of thirty-six GPT-2 sublayers raises test perplexity by roughly one point, and merging four of twelve OPUS-MT sublayers drops BLEU by about two points. In each case the permutation-aligned merge beats the no-permutation version and matches or beats a layer-dropping baseline. The paper further shows that feed-forward sublayer outputs are often highly similar under CKA even when isolated from residual connections, a pattern it does not find in attention sublayers.

Load-bearing premise

The comparison for ViT and machine translation assumes the original, uncompressed models are a fair baseline even though only the compressed models receive recovery fine-tuning; if the originals were fine-tuned on the same data, the performance gap could be larger.

Editorial extensions

If this is right

Merging can be applied to an already-trained model without retraining from scratch, so it fits into existing deployment pipelines as a post-training step.
Because merged weights are tied rather than zeroed, the compression is realized immediately in memory and on disk, and can be combined with quantization for roughly doubled savings at fixed performance.
A validation-selected layer-dropping baseline is weaker at the same compression ratios, so layer redundancy may be better exploited by merging than by deletion.
The observed high similarity between feed-forward sublayer activations suggests mergeability is not an artifact of residual connections and may be a general property of trained Transformers.
The method carries over to SwiGLU-based models, so it can shrink current large language models that use this nonlinearity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the reported CKA similarity forms contiguous blocks, a cheap selection rule that scores windows by average activation similarity could replace exhaustive validation, extending the method to much deeper models.
Combining the merge step with quantization changes the error surface in a way the experiments do not isolate; comparing merged-then-quantized models against pruned-then-quantized models at the same storage budget would separate the two sources of error drift.
If feed-forward sublayer similarity is a general property of trained Transformers rather than a quirk of these three architectures, the align-average-tie recipe could transfer to other architectures with repeated sublayers, though that transfer is outside the paper's experiments.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The core result: you can take a pre-trained Transformer, align the neurons of adjacent feed-forward sublayers with permutations, average their weights, and tie them so they share one parameter set. Doing this for a third of the FF sublayers costs 1 PPL point on GPT-2, 1 accuracy point on ViT, and 2 BLEU on zh-en MT. That is the real result, and it is a decent one. The method is simple, the equations in Section 3.3 are correct, and the paper makes a fair comparison to a strong layer-pruning baseline, which it matches or beats.

What is new is applying model-merging permutation alignment inside a single model, not across models. That is a genuine extension, and the paper does a careful job across decoder-only, encoder, and encoder-decoder architectures. The QLoRA and quantization extensions are useful, and the CKA analysis showing structured similarity among FF sublayers is a nice empirical observation.

The main soft spot is the baselines. For ViT and OPUS-MT, the original uncompressed models are not fine-tuned on the task data, while the merged models receive substantial recovery fine-tuning (up to 50k and 100k steps). The paper discloses this in Section 4.1, but it means the 'comparable to original' claim is only properly tested for GPT-2, where the fine-tuned baseline reaches 16.16 PPL versus 17.27 after merging a third of the FFs. For the other two tasks, the gap could widen if the originals got the same treatment. This does not sink the method—the comparison to the layer-pruning baseline is fair because that baseline is also fine-tuned—but the abstract's phrasing overstates the evidence.

Other issues are minor: no error bars or multiple seeds, and the code link appears truncated. The sliding-window selection of which layers to merge is a form of validation-based model selection; it is standard, though the reported numbers are for the best window.

Overall, this is a solid empirical paper. A serious referee would ask for fine-tuned baselines on ViT and MT and some variance estimates, but the method is well-motivated, the experiments are broad, and the core claim is convincing. I would send it to review.

Referee Report

2 major / 6 minor

Summary. The paper proposes a post-training compression method for Transformer models that merges adjacent feed-forward (FF) sublayers. The method computes activation-based neuron permutations to align FF sublayers within a window, averages the aligned parameters, and ties the merged weights so that k sublayers occupy the memory of one. The authors evaluate on GPT-2 (language modeling), ViT (image classification), OPUS-MT (machine translation), and OLMo-7B with QLoRA (summarization), comparing against layer pruning and showing compatibility with quantization. The central claim is that more than a third of FF sublayers can be merged with performance comparable to the original models and better than a layer-pruning baseline.

Significance. If the results hold, this is a useful and simple post-training compression technique that is orthogonal to quantization and pruning. The merging equations in Section 3.3 are transparent and mathematically sound. The paper includes a fine-tuned baseline for GPT-2, releases a toolkit, and tests across three Transformer architectures and modalities, with additional robustness checks on window selection and anchor choice. The main weakness is that, for two of the three primary tasks, the comparison against the original model is uneven because the compressed models receive recovery fine-tuning while the original models do not; this is a load-bearing issue for the 'comparable performance' claim.

major comments (2)

[Section 4.1 and Tables 10, 11] The 'comparable to original' claim for ViT and OPUS-MT is evaluated against non-fine-tuned baselines. Section 4.1 states that no fine-tuned baseline is provided for these models because the recovery data already appears in their original training data, but the compressed models are still fine-tuned for up to 50k steps (ViT) and 100k steps (MT) on that same data (Appendix C). The GPT-2 experiment shows that such fine-tuning can improve the uncompressed model: the fine-tuned GPT-2 baseline is 16.16 PPL, and this is the reference used in Table 10. For ViT and OPUS-MT, the reference values 80.3 accuracy and 35.8 BLEU are from the original models without this additional recovery fine-tuning. If the original models were fine-tuned under the same protocol, the gap could widen. This directly affects the validity of the central claim for two of the three main tasks. The authors should either run the fine-tuned baselines for ViT and OPUS-MT or explicitly restrict the claim and justify why fine-tuning would not change the reference numbers.
[Section 5.1, Figure 2] The claim that 'performance is almost identical to the original model' at 1/3 FFs removed is unevenly supported by the reported metrics. In Table 10, at 4/12 FFs removed, ViT accuracy drops from 80.3 to 79.2 (about 1.1% relative), GPT-2 PPL increases from 16.16 to 17.27 (about 6.9% relative increase, where lower is better), and OPUS-MT BLEU drops from 35.8 to 33.5 (about 6.4% relative). Given the baseline fairness issue in the first comment, the 'almost identical' phrasing is too strong, especially for GPT-2 and OPUS-MT. The authors should report relative changes and qualify the comparison to account for the fact that the ViT and MT references are not recovery-fine-tuned.

minor comments (6)

[Appendix A] The text says 'for simplicty' and should be 'for simplicity'.
[References] The reference for Dettmers et al. (2022) is formatted as 'Gllm.int8 ()' and should be 'LLM.int8()'; the reference for Voita et al. is printed as 'V oita'.
[Figure 4] The y-axis label reads 'BLEU on T atoeba' and should be 'BLEU on Tatoeba'.
[Section 4.5] The phrase 'apply our method alongsize 4-bit QLoRA' should be 'alongside 4-bit QLoRA'.
[Table 3] The column label 'Compression' is ambiguous: it is not clear whether the values (e.g., 78%, 20%) represent the percentage of original model size remaining or the percentage reduction. Please clarify the definition in the caption or text.
[Algorithm 1] In Algorithm 1, the loop variable i is not passed to the COMPRESS function, so the pseudocode does not specify which window of k adjacent layers is being compressed in each iteration. This should be corrected for reproducibility.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the compression results are empirically self-contained; the only self-citation is a non-load-bearing design note, and the uneven fine-tuning baselines are a fairness caveat, not a circular step.

full rationale

The paper's central claim is an empirical one: merging and tying feed-forward sublayers, followed by recovery fine-tuning, yields models whose measured accuracy, perplexity, and BLEU are close to the original models. Section 5.1 reports these measurements directly, and the two data-dependent choices in the pipeline (activation-based permutation matrices and sliding-window selection via validation performance) are standard model-selection steps rather than fitted parameters disguised as predictions. The reported numbers are evaluated on held-out test data for GPT-2 and OPUS-MT, with ViT results reported transparently on the validation set. No equation in the paper defines the target result in terms of an input that is itself the target result. The only self-citation appears in Section 3.3: "We consider only the neurons just after W_in because prior work has shown that to reorder the input to W_in and output of W_out requires permuting many additional weights due to the residual connections in order to maintain functional equivalence (Verma and Elbayad, 2024)." This is a non-load-bearing design justification: the empirical compression results are established by the paper's own experiments and do not reduce to this cited claim, and there is no imported uniqueness theorem or ansatz smuggled in through the citation. The paper's Section 4.1 states that no fine-tuned baseline is provided for ViT and OPUS-MT "as the data we use already appears in their original training data," which is a genuine evaluation-fairness caveat but not a circular derivation; the compressed models receive recovery fine-tuning while the original models do not, so the 'comparable to original' claim may be optimistic, but this does not make the method's output equivalent to its inputs. The limitations stated in Section 7 (no inference speed-up focus, Transformer-only testing) are scope restrictions, not circular steps. Overall, the derivation chain is self-contained; the score of 2 reflects only the presence of a minor, non-load-bearing self-citation.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The method introduces no new theoretical entities. It relies on standard permutation equivariance, an optimal assignment solver, and several empirical assumptions about sample size and fine-tuning sufficiency. The only hand-chosen quantities are compression targets and fine-tuning budgets.

free parameters (4)

k (number of FF sublayers merged) = 4, 6, 11 (ViT/OPUS-MT); 12, 18, 35 (GPT-2)
Chosen by hand to hit 1/3, 1/2, and (n-1)/n compression targets; directly determines parameter reduction and performance.
Exemplar data size for alignment = ~10k tokens or patches
Chosen by hand; the permutation matrices depend on the statistics of this sample and on the validation set used.
Recovery fine-tuning steps = 100k (GPT-2, OPUS-MT), 50k (ViT), 3k (OLMo)
Chosen by hand; fine-tuning is required to restore performance after merging and is part of the method.
Anchor layer = first FF sublayer in the window
Design choice; the paper tests middle and last anchors and finds robustness, so this is not a critical fitted value.

assumptions (5)

standard math A feed-forward sublayer W_out phi(W_in x + b) remains functionally identical under a neuron permutation P: (P W_in, P b, W_out P^T).
Used in Section 3.3 to justify applying Pi to the weights before averaging.
standard math The Jonker-Volgenant algorithm returns an optimal solution to the linear assignment problem in Eq. (2).
Invoked in Section 3.2 to compute the best permutation from the correlation matrix.
domain assumption Cross-correlation of pre-activation features across sublayers is a valid proxy for neuron correspondence.
Used in Section 3.2; the whole alignment step rests on this correlation criterion.
domain assumption A sample of ~10k tokens or patches yields activation statistics representative of the full data distribution.
Used in Section 4 to compute alignments and CKA; unstable statistics would undermine the permutations.
domain assumption Short recovery fine-tuning restores most performance lost by merging, and the same tuning recipe is fair across methods.
Used in Section 3.4 and 5; if fine-tuning were not sufficient, the central claim would fail.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Merging Feed-Forward Sublayers for Compressed Transformers." pith.science (2026). https://pith.science/paper/HJDTGOYF

@misc{pith2026250106126,
  author       = {Pith},
  title        = {Pith review of: Merging Feed-Forward Sublayers for Compressed Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HJDTGOYF}},
  note         = {Machine review of arXiv:2501.06126}
}

read the original abstract

With the rise and ubiquity of larger deep learning models, the need for high-quality compression techniques is growing in order to deploy these models widely. The sheer parameter count of these models makes it difficult to fit them into the memory constraints of different hardware. In this work, we present a novel approach to model compression by merging similar parameter groups within a model, rather than pruning away less important parameters. Specifically, we select, align, and merge separate feed-forward sublayers in Transformer models, and test our method on language modeling, image classification, and machine translation. With our method, we demonstrate performance comparable to the original models while combining more than a third of model feed-forward sublayers, and demonstrate improved performance over a strong layer-pruning baseline. For instance, we can remove over 21% of total parameters from a Vision Transformer, while maintaining 99% of its original performance. Additionally, we observe that some groups of feed-forward sublayers exhibit high activation similarity, which may help explain their surprising mergeability.

Figures

Figures reproduced from arXiv: 2501.06126 by the authors.

**Figure 2.** Results across all three tasks depicting compression versus performance results. We include results from [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Results across all three tasks depicting compression versus performance for our method and a strong [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Performance curves over different ranges of merged feed-forward sublayers representing 1/3 FFs removed. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: CKA plots of feed-forward sublayer hidden states across three different models. In all three settings, we [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: CKA plots of multi-headed self-attention sublayer activations across three different trained models. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 20 canonical work pages

[1]

Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. 2023. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Representations

work page 2023
[2]

Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. 2022. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis...

work page 2022
[3]

David F. Crouse. 2016. https://doi.org/10.1109/TAES.2016.140952 On implementing 2d rectangular assignment algorithms . IEEE Transactions on Aerospace and Electronic Systems, 52(4):1679--1696

arXiv 2016
[4]

Fahim Dalvi, Hassan Sajjad, Nadir Durrani, and Yonatan Belinkov. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.398 Analyzing redundancy in pretrained transformer models . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4908--4926, Online. Association for Computational Linguistics

doi:10.18653/v1/2020.emnlp-main.398 2020
[5]

Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. 2019. Universal transformers. In International Conference on Learning Representations

work page 2019
[6]

Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Gllm.int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:30318--30332

work page 2022
[7]

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088--10115

2023
[8]

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations

2020

Show all 55 references

[9]

Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. 2022. The role of permutation invariance in linear mode connectivity of neural networks. In International Conference on Learning Representations

2022
[10]

Emile Fiesler, Amar Choudry, and H John Caulfield. 1990. Weight discretization paradigm for optical neural networks. In Optical interconnections and networks, volume 1281, pages 164--173. SPIE

1990
[11]

Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. https://doi.org/10.18653/v1/D19-5409 SAMS um corpus: A human-annotated dialogue dataset for abstractive summarization . In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70--79, Ho...

2019 doi
[12]

Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. 2024. Arcee's mergekit: A toolkit for merging large language models. arXiv preprint arXiv:2403.13257

2024 arXiv
[13]

Thamme Gowda, Roman Grundkiewicz, Elijah Rippeth, Matt Post, and Marcin Junczys-Dowmunt. 2024. Pymarian: Fast neural machine translation and evaluation in python. arXiv preprint arXiv:2408.11853

2024 arXiv
[14]

Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. 2024. Olmo: Accelerating the science of language models. arXiv preprint arXiv:2402.00838

2024 arXiv
[15]

Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A Roberts. 2024. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403.17887

2024 arXiv
[16]

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. stat, 1050:9

2015
[17]

Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations

2022
[18]

Hakan Inan, Khashayar Khosravi, and Richard Socher. 2017. Tying word vectors and word classifiers: A loss framework for language modeling. In International Conference on Learning Representations

2017
[19]

Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2024. Analyzing feed-forward blocks in transformers through the lens of attention maps. In The Twelfth International Conference on Learning Representations

2024
[20]

Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019. Similarity of neural network representations revisited. In International conference on machine learning, pages 3519--3529. PMLR

2019
[21]

Fran c ois Lagunas, Ella Charlaix, Victor Sanh, and Alexander M Rush. 2021. Block pruning for faster transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10619--10629

2021
[22]

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. https://openreview.net/forum?id=H1eA7AEtvS Albert: A lite bert for self-supervised learning of language representations . In International Conference on Learning Representations

2020
[23]

Yann LeCun, John Denker, and Sara Solla. 1989. Optimal brain damage. Advances in neural information processing systems, 2

1989
[24]

Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. 2024. Merge, then compress: Demystify efficient smoe with hints from its routing policy. In The Twelfth International Conference on Learning Representations

2024
[25]

Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. 2015. https://proceedings.mlr.press/v44/li15convergent.html Convergent learning: Do different neural networks learn the same representations? In Proceedings of the 1st International Workshop on Feature Extra...

2015
[26]

Zonglin Li, Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank J Reddi, Ke Ye, Felix Chern, Felix Yu, Ruiqi Guo, et al. 2023. The lazy neuron phenomenon: On emergence of activation sparsity in transformers. In The Eleventh International Conference on Learn...

2023
[27]

Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013/ ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

2004
[28]

Zechun Liu, Changsheng Zhao, Forrest Iandola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, et al. 2024. Mobilellm: Optimizing sub-billion parameter language models for on-device use cases. In Forty-first Internatio...

2024
[29]

Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702--21720

2023
[30]

Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. 2024. https://arxiv.org/abs/2403.03853 Shortgpt: Layers in large language models are more redundant than you expect . Preprint, arXiv:2403.03853

2024 arXiv
[31]

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In International Conference on Learning Representations

2017
[32]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 B leu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--3...

2002
[33]

Telmo Pires, Ant \'o nio Vilarinho Lopes, Yannick Assogba, and Hendra Setiawan. 2023. https://doi.org/10.18653/v1/2023.wmt-1.98 One wide feedforward is all you need . In Proceedings of the Eighth Conference on Machine Translation, pages 1031--1044, Singapore. Association for C...

2023 doi
[34]

Matt Post. 2018. https://doi.org/10.18653/v1/W18-6319 A call for clarity in reporting BLEU scores . In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186--191, Brussels, Belgium. Association for Computational Linguistics

2018 doi
[35]

Ofir Press and Lior Wolf. 2017. Using the output embedding to improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 157--163

2017
[36]

Bharadwaj Pudipeddi, Maral Mesmakhosroshahi, Jinwen Xi, and Sujeeth Bharadwaj. 2020. Training large neural networks with constant memory using a new execution algorithm. arXiv preprint arXiv:2002.05645

2020 arXiv
[37]

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners

2019
[38]

Ricardo Rei, Jos \'e G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and Andr \'e F. T. Martins. 2022. https://aclanthology.org/2022.wmt-1.52/ COMET -22: Unbabel- IST 2022 submission for the metrics shared task . In Pr...

2022
[39]

Machel Reid, Edison Marrese-Taylor, and Yutaka Matsuo. 2021. https://doi.org/10.18653/v1/2021.findings-emnlp.344 Subformer: Exploring weight sharing for parameter efficiency in generative transformers . In Findings of the Association for Computational Linguistics: EMNLP 2021, ...

2021 doi
[40]

Berg, and Li Fei-Fei

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. https://doi.org/10.1007/s11263-015-0816-y ImageNet Large Scale Visual Recognition Challeng...

2015 doi
[41]

Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. 2023. On the effect of dropping layers of pre-trained transformer models. Computer Speech & Language, 77:101429

2023
[42]

Noam Shazeer. 2020. Glu variants improve transformer. arXiv preprint arXiv:2002.05202

2020 arXiv
[43]

Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennigh...

2024
[44]

Sho Takase and Shun Kiyono. 2023. https://doi.org/10.18653/v1/2023.sustainlp-1.5 Lessons on parameter sharing across layers in transformers . In Proceedings of The Fourth Workshop on Simple and Efficient Natural Language Processing (SustaiNLP), pages 78--90, Toronto, Canada (H...

2023 doi
[45]

Norman Tatro, Pin-Yu Chen, Payel Das, Igor Melnyk, Prasanna Sattigeri, and Rongjie Lai. 2020. Optimizing mode connectivity via neuron alignment. Advances in Neural Information Processing Systems, 33:15300--15311

2020
[46]

J \"o rg Tiedemann. 2012. http://www.lrec-conf.org/proceedings/lrec2012/pdf/463_Paper.pdf Parallel data, tools and interfaces in OPUS . In Proceedings of the Eighth International Conference on Language Resources and Evaluation ( LREC '12) , pages 2214--2218, Istanbul, Turkey. ...

2012
[47]

J \"o rg Tiedemann. 2020. https://www.aclweb.org/anthology/2020.wmt-1.139 The T atoeba T ranslation C hallenge -- R ealistic data sets for low resource and multilingual MT . In Proceedings of the Fifth Conference on Machine Translation, pages 1174--1182, Online. Association fo...

2020
[48]

J \"o rg Tiedemann and Santhosh Thottingal. 2020. https://aclanthology.org/2020.eamt-1.61 OPUS - MT -- building open translation services for the world . In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 479--480, Lisboa, P...

2020
[49]

Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N

Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. https://api.semanticscholar.org/CorpusID:13756489 Attention is all you need . In Neural Information Processing Systems

2017
[50]

Neha Verma and Maha Elbayad. 2024. https://arxiv.org/abs/2403.00986 Merging text transformer models from different initializations . Preprint, arXiv:2403.00986

2024 arXiv
[51]

Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. https://doi.org/10.18653/v1/P19-1580 Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned . In Proceedings of the 57th Annual Meeting of the Associatio...

2019 doi
[52]

Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. 2024 a . https://arxiv.org/abs/2408.07666 Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities . Preprint, arXiv:2408.07666

2024 arXiv
[53]

Yifei Yang, Zouying Cao, and Hai Zhao. 2024 b . Laco: Large language model pruning via layer collapse. arXiv preprint arXiv:2402.11187

2024 arXiv
[54]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[55]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. 2023. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Representations

work page 2023

[2] [2]

Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. 2022. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis...

work page 2022

[3] [3]

David F. Crouse. 2016. https://doi.org/10.1109/TAES.2016.140952 On implementing 2d rectangular assignment algorithms . IEEE Transactions on Aerospace and Electronic Systems, 52(4):1679--1696

arXiv 2016

[4] [4]

Fahim Dalvi, Hassan Sajjad, Nadir Durrani, and Yonatan Belinkov. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.398 Analyzing redundancy in pretrained transformer models . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4908--4926, Online. Association for Computational Linguistics

doi:10.18653/v1/2020.emnlp-main.398 2020

[5] [5]

Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. 2019. Universal transformers. In International Conference on Learning Representations

work page 2019

[6] [6]

Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Gllm.int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:30318--30332

work page 2022

[7] [7]

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088--10115

2023

[8] [8]

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations

2020