Pith. sign in

REVIEW 4 major objections 6 minor 89 references

ElastiFormer: Learned Redundancy Reduction in Transformer via Self-Distillation

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ElastiFormer adds tiny learned routers to frozen pretrained Transformers that cut active parameters or tokens by 20-50% across language, vision, and multimodal models, trained by self-distillation.

desk verdict A clean post-training routing recipe with useful scaling curves, but the headline compute-saving claim rests on teacher-fidelity proxies for two of three modalities, not task accuracy. read the letter →

arxiv 2411.15281 v1 pith:5T3KF2IY submitted 2024-11-22 cs.LG cs.AI

Junzhang Liu , Tingkai Liu , Yueyuan Sui , Stephen Xia This is my paper · ORCID
classification cs.LGcs.AI
keywords transformerefficiencyself-distillationlearnedroutingmixture-of-expertstokenpruningattentionheadselectionvariablecomputepost-trainingadaptation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ElastiFormer claims that pretrained Transformer models of any modality contain input-dependent redundancy that can be exploited after training, without retraining the backbone. The paper introduces tiny routing modules, as small as 0.00006% of the model's parameters, that learn to skip attention heads, MLP experts, and whole tokens per input. Trained only by self-distillation against the frozen teacher model, these routers preserve teacher output fidelity while cutting active parameters or tokens by 20-50% across language, vision, and vision-language models. If the claim holds, it offers a cheap, post-training path to variable inference-time compute and a tool for studying where redundancy lives in Transformers.

What carries the argument

The load-bearing object is the routing module: a linear (or one-hidden-layer MLP) projector that, for each layer and input, produces either a set of top-$k$ scores over sub-networks or a set of top-$k$ scores over tokens. Parameter subset selection picks attention heads, or experts obtained by losslessly factorizing a dense MLP into block matrices; an input subset selection picks which tokens continue through MHA or MLP. The router weights are normalized so that selecting every sub-network with uniform weight reproduces the pretrained model exactly, which anchors the method to the teacher. Training is driven by self-distillation: forward KL divergence over the teacher's top-50 tokens for language and vision-language outputs, and cosine distance between encoder token embeddings for ViT-MAE, plus a load-balancing loss for expert routers and a binary cross-entropy loss that keeps causal-LM token selection consistent during autoregressive inference.

What would settle it

Run the exact Elasti-LLM configuration that matches the teacher's LM loss at 38% attention heads and 56% MLP experts on GSM8K and HumanEval and compare exact-match or pass@1 accuracy against the full teacher; if accuracy drops substantially while LM loss stays matched, the fidelity proxy is not sufficient. Similarly, fine-tune a linear probe on top of the Elasti-ViT encoder at 50% head capacity and check ImageNet top-1 accuracy against the teacher encoder.

Watch

Extended reading notes

Core claim

The central discovery is that a frozen pretrained Transformer can be converted into an 'elastic' model whose per-input compute budget is controlled by learned routers, and that this pays off across modalities. In language modeling, ElastiFormer matches the teacher's LM loss with 38% of attention heads and 56% of MLP parameters active, and with rank-1 LoRA adapters it can also drop 20% of tokens from attention. For a ViT-MAE backbone, routing only even layers lets 50% heads, 31% MLP parameters, 70% MHA tokens, and 10% MLP tokens reproduce the teacher's decoder output with cosine similarity above 0.95. For LLaVA, dropping 40% of image tokens before the language decoder preserves LLava-Bench performance, though the paper notes no clear advantage on detail-oriented benchmarks. The paper frames this as evidence that redundancy in pretrained Transformers is data-dependent and learnable rather than static.

Load-bearing premise

The load-bearing assumption is that matching the frozen teacher's outputs (LM loss, decoder cosine similarity, or benchmark win rate) is a faithful proxy for real-task performance; the paper reports no downstream accuracy numbers, so if teacher-fidelity survives while task accuracy drops, the 'no noticeable impact on performance' claim breaks.

Editorial extensions

If this is right

  • Post-training compute reduction applies to frozen pretrained models without architectural retraining, so existing served models can be made elastic on top of techniques like QLoRA.
  • A single checkpoint can be deployed at multiple compute budgets: capacity factors set the fraction of heads, experts, or tokens used per input.
  • The learned routers generalize across data distributions: ViT routers trained on different ImageNet subsets produce highly similar selections on held-out images.
  • Attention-head and MLP-expert redundancy is substantially larger than static pruning suggests: ElastiFormer skips 640 attention heads across layers in Phi-3.5-mini while matching teacher LM loss.
  • Very low-rank LoRA adapters (rank 1) can restore performance lost by token-level attention skipping, pointing to a cheap repair for input-selection weaknesses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the fidelity proxies (LM loss, cosine similarity, win rate) are not task accuracy; a natural next test is whether routers trained to match teacher outputs also preserve GSM8K accuracy, ImageNet classification, or code pass rates at the claimed compute reductions.
  • Beyond the paper: the robustness result suggests routers could serve as lightweight probes for interpretability, e.g., mapping which attention heads or MLP experts are consistently dispensable for a given input class or domain.
  • Beyond the paper: the method's modality-agnostic formulation implies it could be applied to encoder-decoder transformers, diffusion backbones, or speech models, where the same distillation objective would need a suitable output-space metric.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces ElastiFormer, a post-training method that inserts lightweight routing modules into pretrained Transformers to select, per input, a subset of attention heads, MLP experts, and tokens. The routers are trained by self-distillation losses so that the elastic model's outputs match the frozen teacher model. The method is applied to causal LLMs (Phi-3.5-mini-instruct, Gemma-2-2b-it), a ViT-MAE encoder, and LLaVA-1.5-7b. The authors report scaling curves showing that roughly 38% of attention heads and 56% of MLP parameters suffice to match teacher LM loss, that 60–80% of tokens suffice in several settings, and that adding rank-1 LoRA to Q/V projections further reduces required capacity. They also report a router-robustness study across ImageNet subsets and VLM evaluations on LLaVA-Bench and OpenChair.

Significance. If the central claim were fully supported, ElastiFormer would be a practically useful and cheap way to obtain input-dependent compute savings from off-the-shelf pretrained models across modalities. The method's strengths are its simplicity, the very small number of added parameters (as low as 0.00006% in the VLM token router), the systematic scaling analysis for several routing schemes, and the explicit robustness study of ViT routers across ImageNet subsets. The paper is also honest about one limitation: on detail-oriented VLM tasks it reports no clear advantage. However, the headline claim that compute can be reduced by 20–50% 'without noticeable impact on performance' is currently supported mainly by teacher-fidelity proxies—LM loss for LLMs and cosine similarity of MAE decoder outputs for ViT—rather than by task-level accuracy. The significance of the contribution therefore depends on whether those proxies are validated against downstream task metrics.

major comments (4)
  1. [§5.1, Figs. 5–6] The Elasti-LLM scaling claim is supported only by LM loss, and the evaluation set for those curves is not specified. Since the elastic models are trained on the GSM8K training set via self-distillation, if the LM-loss curves are computed on that same set they would reflect training-set teacher matching rather than generalization. Section 3 motivates redundancy using GSM8K and HumanEval task performance, yet no GSM8K accuracy or HumanEval pass@1 is reported for Elasti-LLM. Without a stated evaluation distribution and task-level metrics, the statement that 38% of heads and 56% of MLP parameters yield no noticeable performance impact is unverified.
  2. [§5.2, Fig. 7] Elasti-ViT is evaluated only by cosine similarity between the teacher and student MAE decoder outputs. The 0.95 threshold used in Figure 7 is an arbitrary fidelity cutoff; there is no evidence that exceeding it corresponds to no noticeable impact on an actual vision task such as ImageNet classification or linear probing. Consequently, the claimed 20–50% compute savings for image modeling rest on an unvalidated proxy. Reporting a downstream task metric, or at least a linear-probe accuracy curve, would make the claim load-bearing.
  3. [§5.3, Fig. 9] The paper itself concedes that on detail-oriented VLM tasks (LLaVA-Bench detail and OpenChair), Elasti-VLM offers no clear advantage over the base model. This is an internal admission that teacher-fidelity transfer can fail precisely when detailed visual information is needed, and it directly qualifies the umbrella claim of '20% to 50% reduction for all modalities without noticeable impact on performance.' The authors should either restrict the claim to the settings where task benchmarks support it or analyze why fidelity transfer fails on detail-oriented tasks.
  4. [Appendix B.1, §4.1] The compute-savings claims are expressed in terms of nominal capacity factors, but during inference the token router uses a fixed decision threshold of 0.5 rather than the top-k selection used in training. The paper does not report the realized fraction of tokens or experts actually selected at inference. If the threshold-based router selects more than the nominal capacity, the reported savings are overstated. Reporting measured active-token/active-parameter counts at inference would make the compute claims precise.
minor comments (6)
  1. [Abstract vs. §1] The Abstract states that added parameters can be 'as low as .00006%' while the Introduction states 'as little as 0.0006% additional parameters' (p. 2). These differ by an order of magnitude; please reconcile them.
  2. [§4.1 and Appendix B.2] Algorithm 1 multiplies selected sub-network outputs by routing weights, but Appendix B.2 says 'unselected experts simply pass their input through without modification.' Please clarify the exact output aggregation and how the straight-through estimator interacts with the M-normalized softmax weights.
  3. [Figure 4] The x-axis label 'train/epoch' is ambiguous; it should be 'epoch' or 'training step.' Also, the 'Top-K' values (5, 20, 50) for the top-K KL variant should be defined in the caption.
  4. [§5.2, Figure 8] The robustness study trains on 10 subsets of ImageNet based on categories from a GitHub repository [39], but the paper does not specify how the subsets were split from the training set or whether the held-out evaluation images overlap with the training classes. A short description of the split and evaluation protocol would aid reproducibility.
  5. [Figure 11 caption] There is a typo in the caption: 'capcaity' should be 'capacity.'
  6. [References] Reference [39] points to a GitHub repository; a versioned citation or commit identifier would make the data split reproducible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported capacity thresholds are measured outcomes of a distillation training objective, not inputs to it; the main weakness is the use of teacher-fidelity proxies for LLM/ViT performance, which is an evaluation gap rather than a circular derivation.

full rationale

The derivation chain in ElastiFormer is not circular. The routers are trained with a self-distillation loss (Eq. 1) to minimize the difference between the frozen teacher and the elastic student, and the headline claims (e.g., 38% attention heads, 56% MLP experts, 60% image tokens) are read off empirically measured scaling curves (Figs. 5-7, 9). Nothing in the objective forces these thresholds: a model with no redundancy would require near-100% capacity to match the teacher, so achieving teacher-level LM loss or cosine similarity at reduced capacity is a nontrivial empirical finding. The normalization in Algorithm 1 that makes uniform routing reproduce the pretrained network is an identity-preserving initialization, not a hidden fit used as a prediction. The only self-citation ([33], which includes co-author Tingkai Liu) appears in related work on patch pruning and is not load-bearing for any central claim. The genuine limitation is evaluative rather than circular: for Elasti-LLM and Elasti-ViT, 'performance' is measured by teacher-output fidelity (LM loss, cosine similarity to the frozen MAE decoder) rather than downstream task accuracy, and Section 5.3 explicitly concedes that on detail-oriented VLM tasks (LLava-Bench detail, OpenChair) Elasti-VLM 'offers no clear advantage.' This weakens the breadth of the 'without noticeable impact on performance' claim, but it does not mean the result is equivalent to its input by construction. The VLM results do use external benchmarks (LLava-Bench, OpenChair), and the capacity thresholds are genuine measured outcomes. Therefore the paper has no significant circularity.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The compute-saving claims rest on the validity of teacher-matching proxies, the inference-time threshold approximation, the lossless MoE conversion, and the representative nature of the evaluated models. None of these are derived; they are assumptions.

free parameters (7)
  • Capacity factor for MLP input selection (c_input_mlp) = 0.8 (drop 20% tokens)
    Smallest capacity at which teacher-level LM loss is reached in Elasti-LLM (Figure 5); determines reported 20% token savings.
  • Capacity factor for MHA parameter selection (c_param_mha) = 12/32 heads (0.375)
    Capacity at which teacher-level LM loss is reached in Phi-3.5-mini; determines reported 38% active heads.
  • Capacity factor for MLP parameter selection (c_param_mlp) = 18/32 experts (0.5625)
    Capacity at which teacher-level LM loss is reached; determines reported 56% active MLP parameters.
  • Inference threshold for token routing = 0.5
    Used instead of top-k at inference for causal LMs (Section B.1); mismatch with training capacity is mitigated by auxiliary loss but not measured.
  • LoRA rank for Q,V projections = 1 (also 2,4,8)
    Minimum rank that allows 20% token dropping from MHA in Gemma-2 (Figure 6); a design choice.
  • Auxiliary loss weights lambda_load and lambda_topk = 1.0
    Set by hand; not tuned; affects router training stability.
  • Number of experts M for MLP conversion = 32 for Phi-3.5 and ViT-MAE-L
    Granularity of MLP parameter selection; exact block partition is not specified.
assumptions (5)
  • domain assumption Self-distillation to a frozen teacher preserves downstream task performance.
    Evaluation uses teacher-matching proxies (LM loss, cosine similarity, win rate); no downstream task accuracy is reported.
  • domain assumption Threshold-based inference matches top-k training behavior for causal LMs.
    Section B.1 switches from top-k to threshold 0.5 at inference; auxiliary loss is used but alignment is not quantitatively validated.
  • standard math Dense MLP split into block experts is lossless at full capacity.
    Equation in Section 4.1 shows exact equivalence when all experts are selected with uniform weights.
  • standard math Straight-through estimator yields usable gradients for top-k selection.
    Common MoE practice; gradients are biased but workable.
  • domain assumption Evaluated models (Gemma-2-2b, Phi-3.5-mini, ViT-MAE-L, LLaVA-1.5) support cross-modal generality.
    The paper generalizes to all modalities from a small set of models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ElastiFormer: Learned Redundancy Reduction in Transformer via Self-Distillation." pith.science (2026). https://pith.science/paper/5T3KF2IY

@misc{pith2026241115281,
  author       = {Pith},
  title        = {Pith review of: ElastiFormer: Learned Redundancy Reduction in Transformer via Self-Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5T3KF2IY}},
  note         = {Machine review of arXiv:2411.15281}
}
read the original abstract

We introduce ElastiFormer, a post-training technique that adapts pretrained Transformer models into an elastic counterpart with variable inference time compute. ElastiFormer introduces small routing modules (as low as .00006% additional trainable parameters) to dynamically selects subsets of network parameters and input tokens to be processed by each layer of the pretrained network in an inputdependent manner. The routing modules are trained using self-distillation losses to minimize the differences between the output of the pretrained-model and their elastic counterparts. As ElastiFormer makes no assumption regarding the modality of the pretrained Transformer model, it can be readily applied to all modalities covering causal language modeling, image modeling as well as visual-language modeling tasks. We show that 20% to 50% compute saving could be achieved for different components of the transformer layer, which could be further reduced by adding very low rank LoRA weights (rank 1) trained via the same distillation objective. Finally, by comparing routing trained on different subsets of ImageNet, we show that ElastiFormer is robust against the training domain.

Figures

Figures reproduced from arXiv: 2411.15281 by the authors.

Figure 1
Figure 1. Overview of ElastiFormer for language, visual, and multi-modal transformers. (Left) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Difference in language modeling loss (blue) and top-1 token prediction agreement (red) [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Illustration of the two subset selection schemes employed in the current work. For [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Comparison between different distillation losses for language output modality. The three [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Scaling of various modules in Elasti-LLM against compute for elastic [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Scaling of Gemma-2-2B-it across a percentage of processed tokens with LoRA adapter for Q, V projections in MHA. The ElastiFormer module is trained with input subset selection for both MHA and MLP modules, as well as parameter subset selection for the MLP module (with 4…
Figure 7
Figure 7. Figure 7: Scaling of Elasti-ViT (ViT-MAE-L) performance on held-out ImageNet-1K test set as measured by cosine similarity between MAE decoder output when given encoder output of Elasti-ViT and base pretrained encoder (as shown in (A)). Experiments are shown for both ViT-MAE with…
Figure 8
Figure 8. Figure 8: Comparing learned routing of Elasti-ViT trained using different subsets of ImageNet. (Left) [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Evaluation of Elasti-LLaVA on LLava-Bench and OpenChair benchmarks against capacity. [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: Example output of Gemma-2-2b-it. The capacity of input selection for MLP/MHA and [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Example reconstruction (decoder output) of ImageNet images from pretrained ViT-MAE [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: Example responses of LLava-Bench comparing GPT-4 response and Elasti-VLM response [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

89 extracted references · 28 canonical work pages

  1. [1]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matt...

    2024

  2. [2]

    Fluctuation-based adaptive structured pruning for large language models

    Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Fluctuation-based adaptive structured pruning for large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 10865–10873, 2024

    2024

  3. [3]

    Dynamic context pruning for efficient and interpretable autoregressive transformers

    Sotiris Anagnostidis, Dario Pavllo, Luca Biggio, Lorenzo Noci, Aurelien Lucchi, and Thomas Hofmann. Dynamic context pruning for efficient and interpretable autoregressive transformers. Advances in Neural Information Processing Systems, 36, 2024

    2024

  4. [4]

    A general language assistant as a laboratory for alignment, 2021

    Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. A general language assistant as a labora...

    2021

  5. [5]

    Mitigat- ing open-vocabulary caption hallucinations, 2024

    Assaf Ben-Kish, Moran Yanuka, Morris Alper, Raja Giryes, and Hadar Averbuch-Elor. Mitigat- ing open-vocabulary caption hallucinations, 2024

    2024

  6. [6]

    GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021

    Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021. If you use this software, please cite it using these metadata

    2021

  7. [7]

    Token merging: Your vit but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022

    arXiv 2022

  8. [8]

    Emerging properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021

    2021

Show all 89 references
  1. [9]

    Vision transformer slimming: Multi-dimension searching in continuous optimization space

    Arnav Chavan, Zhiqiang Shen, Zhuang Liu, Zechun Liu, Kwang-Ting Cheng, and Eric P Xing. Vision transformer slimming: Multi-dimension searching in continuous optimization space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4931–4941, 2022

    2022

  2. [10]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G...

    2021

  3. [11]

    The lottery ticket hypothesis for pre-trained bert networks

    Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. Advances in neural information processing systems, 33:15834–15846, 2020

    2020

  4. [12]

    The principle of diversity: Training stronger vision transformers calls for reducing all levels of redundancy

    Tianlong Chen, Zhenyu Zhang, Yu Cheng, Ahmed Awadallah, and Zhangyang Wang. The principle of diversity: Training stronger vision transformers calls for reducing all levels of redundancy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, page...

    2022

  5. [13]

    A toy model of universality: Reverse engineering how networks learn group operations

    Bilal Chughtai, Lawrence Chan, and Neel Nanda. A toy model of universality: Reverse engineering how networks learn group operations. In International Conference on Machine Learning, pages 6243–6267. PMLR, 2023

    2023

  6. [14]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  7. [15]

    Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models

    Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066, 2024

    2024 arXiv

  8. [16]

    Analyzing redundancy in pretrained transformer models

    Fahim Dalvi, Hassan Sajjad, Nadir Durrani, and Yonatan Belinkov. Analyzing redundancy in pretrained transformer models. arXiv preprint arXiv:2004.04010, 2020

    2004 arXiv

  9. [17]

    Imagenet: A large- scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009

    2009

  10. [18]

    Qlora: Efficient finetuning of quantized llms

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024

    2024

  11. [19]

    Eventful transformers: leveraging temporal redundancy in vision transformers

    Matthew Dutson, Yin Li, and Mohit Gupta. Eventful transformers: leveraging temporal redundancy in vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16911–16923, 2023

    2023

  12. [20]

    A mathematical framework for transformer circuits

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12, 2021

    2021

  13. [21]

    Depgraph: Towards any structural pruning

    Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. Depgraph: Towards any structural pruning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16091–16101, 2023

    2023

  14. [22]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022

    2022

  15. [23]

    Transformer feed-forward layers are key-value memories

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.14913, 2020

    2012 arXiv

  16. [24]

    Successor heads: Recurring, interpretable attention heads in the wild

    Rhys Gould, Euan Ong, George Ogden, and Arthur Conmy. Successor heads: Recurring, interpretable attention heads in the wild. arXiv preprint arXiv:2312.09230, 2023

    2023 arXiv

  17. [25]

    MiniLLM: Knowledge distillation of large language models

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, 2024

    2024

  18. [26]

    Learning efficient vision transformers via fine-grained manifold distillation

    Zhiwei Hao, Jianyuan Guo, Ding Jia, Kai Han, Yehui Tang, Chao Zhang, Han Hu, and Yunhe Wang. Learning efficient vision transformers via fine-grained manifold distillation. Advances in Neural Information Processing Systems, 35:9164–9175, 2022

    2022

  19. [27]

    Masked autoencoders are scalable vision learners, 2021

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners, 2021

    2021

  20. [28]

    What matters in transformers? not all attention is needed

    Shwai He, Guoheng Sun, Zheyu Shen, and Ang Li. What matters in transformers? not all attention is needed. arXiv preprint arXiv:2406.15786, 2024. 14

    2024 arXiv

  21. [29]

    Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015

    Geoffrey Hinton. Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  22. [30]

    Sparse progressive distillation: Resolving overfitting under pretrain-and-finetune paradigm

    Shaoyi Huang, Dongkuan Xu, Ian EH Yen, Yijue Wang, Sung-En Chang, Bingbing Li, Shiyang Chen, Mimi Xie, Sanguthevar Rajasekaran, Hang Liu, et al. Sparse progressive distillation: Resolving overfitting under pretrain-and-finetune paradigm. arXiv preprint arXiv:2110.08190, 2021

    arXiv

  23. [31]

    Mixture of nested experts: Adaptive processing of visual tokens

    Gagan Jain, Nidhi Hegde, Aditya Kusupati, Arsha Nagrani, Shyamal Buch, Prateek Jain, Anurag Arnab, and Sujoy Paul. Mixture of nested experts: Adaptive processing of visual tokens. arXiv preprint arXiv:2407.19985, 2024

    2024 arXiv

  24. [32]

    Self-distillation into self-attention heads for improving transformer-based end-to-end neural speaker diarization

    Ye-Rin Jeoung, Jeong-Hwan Choi, Ju-Seok Seong, JeHyun Kyung, and Joon-Hyuk Chang. Self-distillation into self-attention heads for improving transformer-based end-to-end neural speaker diarization. In Proceedings of the Annual Conference of the International Speech Communicatio...

    2023

  25. [33]

    Expedited training of visual conditioned language generation via redundancy reduction

    Yiren Jian, Tingkai Liu, Yunzhe Tao, Chunhui Zhang, Soroush V osoughi, and Hongxia Yang. Expedited training of visual conditioned language generation via redundancy reduction. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1:...

    2024

  26. [34]

    Mixtral of experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  27. [35]

    Self-supervised 3d anatomy segmentation using self-distilled masked image transformer (smit)

    Jue Jiang, Neelam Tyagi, Kathryn Tringale, Christopher Crane, and Harini Veeraraghavan. Self-supervised 3d anatomy segmentation using self-distilled masked image transformer (smit). In International Conference on Medical Image Computing and Computer-Assisted Intervention, page...

    2022

  28. [36]

    Tinybert: Distilling bert for natural language understanding

    Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:1909.10351, 2019

    1909 arXiv

  29. [37]

    Shortened llama: A simple depth pruning for large language models

    Bo-Kyeong Kim, Geonmin Kim, Tae-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung-Kyu Song. Shortened llama: A simple depth pruning for large language models. arXiv preprint arXiv:2402.02834, 11, 2024

    2024 arXiv

  30. [38]

    Self-distillation for further pre-training of transformers

    Seanie Lee, Minki Kang, Juho Lee, Sung Ju Hwang, and Kenji Kawaguchi. Self-distillation for further pre-training of transformers. arXiv preprint arXiv:2210.02871, 2022

    2022 arXiv

  31. [39]

    Clustered imagenet labels for training production-friendly image classifier

    Youngsoo Lee and Seongjoo Moon. Clustered imagenet labels for training production-friendly image classifier. https://github.com/Prev/clustered-imagenet-labels , 2021

    2021

  32. [40]

    Knowledge distillation via the target-aware transformer

    Sihao Lin, Hongwei Xie, Bing Wang, Kaicheng Yu, Xiaojun Chang, Xiaodan Liang, and Gang Wang. Knowledge distillation via the target-aware transformer. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10915–10924, 2022

    2022

  33. [41]

    Fq-vit: Post-training quantization for fully quantized vision transformer

    Yang Lin, Tianyu Zhang, Peiqin Sun, Zheng Li, and Shuchang Zhou. Fq-vit: Post-training quantization for fully quantized vision transformer. arXiv preprint arXiv:2111.13824, 2021

    2021 arXiv

  34. [42]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  35. [43]

    Oscillation-free quantization for low-bit vision transformers

    Shih-Yang Liu, Zechun Liu, and Kwang-Ting Cheng. Oscillation-free quantization for low-bit vision transformers. In International Conference on Machine Learning, pages 21813–21824. PMLR, 2023

    2023

  36. [44]

    Post-training quantization for vision transformer

    Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. Post-training quantization for vision transformer. Advances in Neural Information Processing Systems , 34:28092–28103, 2021

    2021

  37. [45]

    Anytime dense prediction with confidence adaptivity

    Zhuang Liu, Zhiqiu Xu, Hung-Ju Wang, Trevor Darrell, and Evan Shelhamer. Anytime dense prediction with confidence adaptivity. arXiv preprint arXiv:2104.00749, 2021

    2021 arXiv

  38. [46]

    A transformer-based model with self-distillation for multimodal emotion recognition in conversations

    Hui Ma, Jian Wang, Hongfei Lin, Bo Zhang, Yijia Zhang, and Bo Xu. A transformer-based model with self-distillation for multimodal emotion recognition in conversations. IEEE Trans- actions on Multimedia, 2023. 15

    2023

  39. [47]

    Llm-pruner: On the structural pruning of large language models

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702–21720, 2023

    2023

  40. [48]

    Copy suppression: Comprehensively understanding an attention head

    Callum McDougall, Arthur Conmy, Cody Rushing, Thomas McGrath, and Neel Nanda. Copy suppression: Comprehensively understanding an attention head. arXiv preprint arXiv:2310.04625, 2023

    2023 arXiv

  41. [49]

    Locating and editing factual associations in gpt

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems , 35:17359–17372, 2022

    2022

  42. [50]

    Circuit component reuse across tasks in transformer language models

    Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Circuit component reuse across tasks in transformer language models. arXiv preprint arXiv:2310.08744, 2023

    2023 arXiv

  43. [51]

    Zoom in: An introduction to circuits

    Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5(3):e00024–001, 2020

    2020

  44. [52]

    In-context learning and induction heads

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022

    2022 arXiv

  45. [53]

    Ia-red 2: Interpretability-aware redundancy reduction for vision transformers

    Bowen Pan, Rameswar Panda, Yifan Jiang, Zhangyang Wang, Rogerio Feris, and Aude Oliva. Ia-red 2: Interpretability-aware redundancy reduction for vision transformers. Advances in Neural Information Processing Systems, 34:24898–24911, 2021

    2021

  46. [54]

    Self-evolving vision transformer for chest x-ray diagnosis through knowledge distillation

    Sangjoon Park, Gwanghyun Kim, Yujin Oh, Joon Beom Seo, Sang Min Lee, Jin Hwan Kim, Sungjun Moon, Jae-Kwang Lim, Chang Min Park, and Jong Chul Ye. Self-evolving vision transformer for chest x-ray diagnosis through knowledge distillation. Nature communications, 13(1):3848, 2022

    2022

  47. [55]

    A practical review of mech- anistic interpretability for transformer-based language models.arXiv preprint arXiv:2407.02646, 2024

    Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov, and Ziyu Yao. A practical review of mech- anistic interpretability for transformer-based language models.arXiv preprint arXiv:2407.02646, 2024

    2024

  48. [56]

    Dynamicvit: Efficient vision transformers with dynamic token sparsification

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34:13937–13949, 2021

    2021

  49. [57]

    Mixture-of-depths: Dynamically allocating compute in transformer-based language models

    David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-depths: Dynamically allocating compute in transformer-based language models. arXiv preprint arXiv:2404.02258, 2024

    2024 arXiv

  50. [58]

    Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter

    Victor Sanh, L Debut, J Chaumond, and T Wolf. Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter. arxiv 2019. arXiv preprint arXiv:1910.01108, 2019

    2019 arXiv

  51. [59]

    Confident adaptive language modeling

    Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. Advances in Neural Information Processing Systems, 35:17456–17472, 2022

    2022

  52. [60]

    Omniquant: Omnidirectionally calibrated quantiza- tion for large language models

    Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantiza- tion for large language models. arXiv preprint arXiv:2308.13137, 2023

    2023 arXiv

  53. [61]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017

    2017 arXiv

  54. [62]

    A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis

    Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis. arXiv preprint arXiv:2305.15054, 2023

    2023 arXiv

  55. [63]

    Tasked: transformer-based adversarial learning for human activity recognition using wearable sensors via self-knowledge distillation

    Sungho Suh, Vitor Fortes Rey, and Paul Lukowicz. Tasked: transformer-based adversarial learning for human activity recognition using wearable sensors via self-knowledge distillation. Knowledge-Based Systems, 260:110143, 2023

    2023

  56. [64]

    Self-distilled vision transformer for domain generalization

    Maryam Sultana, Muzammal Naseer, Muhammad Haris Khan, Salman Khan, and Fahad Shah- baz Khan. Self-distilled vision transformer for domain generalization. In Proceedings of the Asian Conference on Computer Vision, pages 3068–3085, 2022

    2022

  57. [65]

    Patient knowledge distillation for bert model compression

    Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprint arXiv:1908.09355, 2019. 16

    1908 arXiv

  58. [66]

    Mobilebert: a compact task-agnostic bert for resource-limited devices

    Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert: a compact task-agnostic bert for resource-limited devices. arXiv preprint arXiv:2004.02984, 2020

    2004 arXiv

  59. [67]

    Patch slimming for efficient vision transformers

    Yehui Tang, Kai Han, Yunhe Wang, Chang Xu, Jianyuan Guo, Chao Xu, and Dacheng Tao. Patch slimming for efficient vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12165–12174, 2022

    2022

  60. [68]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan...

    2024

  61. [69]

    Training data-efficient image transformers & distillation through attention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347–10357. PMLR, 2021

    2021

  62. [70]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V on Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors,Advances in Neural ...

    2017

  63. [71]

    Rkld: Reverse kl-divergence- based knowledge distillation for unlearning personal information in large language models, 2024

    Bichen Wang, Yuzhe Zi, Yixin Sun, Yanyan Zhao, and Bing Qin. Rkld: Reverse kl-divergence- based knowledge distillation for unlearning personal information in large language models, 2024

    2024

  64. [72]

    Interpretability in the wild: a circuit for indirect object identification in gpt-2 small

    Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593, 2022

    2022 arXiv

  65. [73]

    Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33:5776–5788, 2020. 17

    2020

  66. [74]

    Last: Label-free self-distillation contrastive learning with transformer architecture for remote sensing image scene classification

    Xuying Wang, Jiawei Zhu, Zhengliang Yan, Zhaoyang Zhang, Yunsheng Zhang, Yansheng Chen, and Haifeng Li. Last: Label-free self-distillation contrastive learning with transformer architecture for remote sensing image scene classification. IEEE Geoscience and Remote Sensing Lette...

    2022

  67. [75]

    Tinyvit: Fast pretraining distillation for small vision transformers

    Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Tinyvit: Fast pretraining distillation for small vision transformers. In European conference on computer vision, pages 68–85. Springer, 2022

    2022

  68. [76]

    Sheared llama: Accelerating language model pre-training via structured pruning

    Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared llama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.06694, 2023

    2023 arXiv

  69. [77]

    Structured pruning learns compact and accurate models

    Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Structured pruning learns compact and accurate models. arXiv preprint arXiv:2204.00408, 2022

    2022 arXiv

  70. [78]

    Smoothquant: Accurate and efficient post-training quantization for large language models

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087–38099. PMLR, 2023

    2023

  71. [79]

    Think: Thinner key cache by query-driven pruning

    Yuhui Xu, Zhanming Jie, Hanze Dong, Lei Wang, Xudong Lu, Aojun Zhou, Amrita Saha, Caiming Xiong, and Doyen Sahoo. Think: Thinner key cache by query-driven pruning. arXiv preprint arXiv:2407.21018, 2024

    2024 arXiv

  72. [80]

    X-pruner: explainable pruning for vision transformers

    Lu Yu and Wei Xiang. X-pruner: explainable pruning for vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24355–24363, 2023

    2023

  73. [81]

    Unified visual transformer compression

    Shixing Yu, Tianlong Chen, Jiayi Shen, Huan Yuan, Jianchao Tan, Sen Yang, Ji Liu, and Zhangyang Wang. Unified visual transformer compression. arXiv preprint arXiv:2203.08243, 2022

    2022 arXiv

  74. [82]

    Moefication: Transformer feed-forward layers are mixtures of experts

    Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Moefication: Transformer feed-forward layers are mixtures of experts. arXiv preprint arXiv:2110.01786, 2021

    2021 arXiv

  75. [83]

    Knowledge distillation based on transformed teacher matching

    Kaixiang Zheng and EN-HUI YANG. Knowledge distillation based on transformed teacher matching. In The Twelfth International Conference on Learning Representations, 2024

    2024

  76. [84]

    The clock and the pizza: Two stories in mechanistic explanation of neural networks

    Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks. Advances in Neural Information Processing Systems, 36, 2024

    2024

  77. [85]

    Llama-moe: Building mixture-of-experts from llama with continual pre-training

    Tong Zhu, Xiaoye Qu, Daize Dong, Jiacheng Ruan, Jingqi Tong, Conghui He, and Yu Cheng. Llama-moe: Building mixture-of-experts from llama with continual pre-training. arXiv preprint arXiv:2406.16554, 2024

    2024 arXiv

  78. [86]

    Moe- bert: from bert to mixture-of-experts via importance-guided adaptation

    Simiao Zuo, Qingru Zhang, Chen Liang, Pengcheng He, Tuo Zhao, and Weizhu Chen. Moe- bert: from bert to mixture-of-experts via importance-guided adaptation. arXiv preprint arXiv:2204.07675, 2022. 18 A Pruning Experiments for Analyzing Redundancy in Pretrained LLMs Here we descr...

    2022 arXiv

  79. [87]

    Possible solutions could include:

    Implementing public transportation: Encouraging the use of public transportation, The city might be facing traffic congestion issues, particularly in the four-lane sections of the highway since most cars have their brake lights on, indicating slow or stopped traffic. Possible ...

  80. [88]

    Implementing traffic management strategies, such as ramp meters to control the rate at which cars enter the highway or optimizing traffic signal timings to improve traffic flow

  81. [89]

    Encouraging the use of public Top-2 Token Score =0.25 Top-32 Token Score =1.12 GPT-4 Elasti-VLMLinear Top-4Token Score=0.25 Top-350Token Score=1.12 Thecreativetwistonaclassicworkofart, whereadogisdepictedwearinga Renaissance-styledressandahat,canbe interpreteddifferentlybyvari...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.