Efficient Large Language Models with Zero-Shot Adjustable Acceleration

Mohammad Sharifkhani; Sajjad Kachuee

REVIEW 4 major objections 5 minor 13 references

Efficient Large Language Models with Zero-Shot Adjustable Acceleration

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A uniform sampling of preservation rates during fine-tuning lets a single LLM be accelerated at inference over a continuous range, up to 11x, without retraining.

desk verdict The random-alpha training trick is a neat idea, but the paper's reported acceleration numbers contradict its own pruning recursion, so the evidence as presented does not hold up. read the letter →

arxiv 2509.01190 v2 pith:7VHYZGUT submitted 2025-09-01 cs.CL

Sajjad Kachuee , Mohammad Sharifkhani This is my paper

classification cs.CL

keywords zero-shotaccelerationtokenpruninghiddenactivationLLMinferencespeedupsoftmaxcontributionadjustablefine-tuningpolicyefficiency

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a single LLM can be made adjustable at inference time: after fine-tuning with a preservation rate sampled uniformly from 2% to 100% at every step, the same model can be run at any acceleration level by setting one hyperparameter, without retraining. Applied to GPT-2, LLaMA-3, and Gemma-2, the approach claims up to 11x speedups on IMDB and around 9x average latency reduction on GLUE, with perplexity and MMLU scores degrading only modestly at moderate speeds. The mechanism prunes hidden activations (tokens) with the lowest softmax contribution to the attention output, using a minimum threshold to preserve stability. The central contribution is the fine-tuning policy, which replaces conventional fine-tuning at no extra cost and produces a model robust to a wide range of pruning rates.

What carries the argument

The load-bearing object is the soft-score of causal attention, used to rank hidden activations by contribution to the layer output. Pruning keeps the tokens with the largest softmax-weighted contribution, governed by the preservation rate α (Equation 2) and a floor τl to prevent collapse to a single token. The training policy in Equation 3 — sampling α uniformly from 0.02 to 1.00 each step — is what makes the model robust across the entire acceleration range.

What would settle it

Run the exact same fine-tuning policy but prune tokens at random instead of by softmax contribution, holding α and τl fixed; if random pruning matches the reported accuracy on IMDB or GLUE, the contribution criterion is not doing the work and the method reduces to a token-dropout schedule. Conversely, a large gap between random and soft-score pruning would confirm the criterion.

Watch

Extended reading notes

Core claim

The central claim is that token-level pruning of hidden activations in decoder-only Transformers can be made both fast and adjustable if the model is fine-tuned with a randomized preservation rate. Specifically, sampling α from U(0.02, 1.00) at each fine-tuning step, instead of fixing a single rate, lets a user choose any acceleration setting at inference time with no additional training. The paper demonstrates this on sentiment classification, GLUE tasks, language modeling, and instruction tuning, reporting up to 11x speedup on IMDB, and shows the same policy works on GPT-2, LLaMA-3, and Gemma-2 including 4-bit quantized versions fine-tuned with LoRA.

Load-bearing premise

The claim rests on the unproven assumption that a token whose attention soft-score is small is genuinely dispensable for subsequent layers; the paper never checks whether randomly dropping tokens at the same rate performs equally well.

Editorial extensions

If this is right

A deployment can change its speed–accuracy trade-off per request or per load simply by setting α, with no model swap.
The method composes with QLoRA-style quantization and LoRA fine-tuning, so large models can be adjusted without full fine-tuning.
Since the base architecture is unchanged, the pruning layer can be stacked with other accelerators like FlashAttention or speculative decoding.
The reported acceleration ceiling grows with sequence length, suggesting the method exploits redundancy in long contexts.
The same fine-tuning recipe should transfer to any decoder-only Transformer with a similar causal attention structure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not compare contribution-based pruning against random token pruning; absent such a control, the reported gains may be substantially due to the dropout-like fine-tuning schedule rather than the importance criterion. A reader should treat the mechanism's necessity as unproven.
If the uniform-α policy is the active ingredient, it could be reinterpreted as a regularization scheme that makes models robust to missing tokens, and might transfer to other structured pruning (layers, heads) with the same recipe.
The method opens a simple path to latency-aware serving: a scheduler could pick α from the observed queue depth, treating accuracy as a soft resource.
A natural extension is to test whether the same policy enables adjustable acceleration in encoder-decoder or multimodal models, where the soft-score contribution is less well-defined.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The core idea here is genuinely simple and appealing: during fine-tuning, sample the preservation rate alpha from U(0.02,1) at every step, and you get a single model with a continuous inference-time speed/accuracy knob. That is a real twist on the authors' earlier latency-adjustable transformer work, and the practical value of a one-knob accelerator is clear. Applying it to decoder-only LLMs is a routine extension, but fine-tuning fully replaces standard fine-tuning, so there is no extra training cost. The paper also covers a broad spread of tasks and models, which deserves credit.

That said, the evidence does not support the claims. The most damaging issue is Table 3. The paper's own Eq. (2), N_l = max(alpha N_{l-1}, tau_l), caps the achievable MACs reduction. For LLaMA-3-8B with 32 layers and alpha=0.99, even if attention dominates completely, the max reduction is about 1.34x; including FFN lowers it. Yet Table 3 reports 1.70x MACs reduction at alpha=0.99. The same holds for Gemma-2. This is not a missing baseline; the reported efficiency gains cannot be produced by the described mechanism unless the table is mislabeled or the pruning law is different. A serious referee would need this resolved before anything else.

Beyond that, the experimental methodology is weak. There are no baselines: no standard fine-tuning under the same compute budget, no comparison to existing token pruning methods like LazyLLM or Dodo, and no random-pruning ablation. Without that, you cannot tell whether the soft-score contribution heuristic does anything beyond a token-dropout schedule. There are no error bars, no code, and no measurement protocol for the headline "up to 11x speedup." The paper itself concedes task-specific degradation and code unavailability. The 11x speedup on IMDB comes with an accuracy drop from 92.4 to 86.5, which is substantial.

For a reader working on dynamic inference, the random-alpha training idea is worth thinking about, and the paper is a reasonable cautionary example of how pruning papers should be evaluated. But the quantitative core is internally inconsistent, and the missing baselines mean the central claim is unsupported. It deserves a serious peer review, not a desk reject, because the idea is testable and the authors clearly know what they are trying to build. The review should demand corrected tables, real baseline comparisons, and code. If those come, the paper could be salvageable; as it stands, I would not cite it.

Referee Report

4 major / 5 minor

Summary. The paper proposes a training policy and inference-time pruning mechanism for decoder-only LLMs: during fine-tuning, the preservation rate α is sampled per step from U(0.02, 1.00) (Eq. 3), and at inference a fixed α is applied through a Resource Adjustment Layer that prunes hidden activations with low attention contribution, following N_l = max(αN_{l−1}, τ_l) (Eq. 2). The authors claim this yields zero-shot adjustable acceleration, with up to 11× speedup on GPT-2 and improved efficiency on LLaMA-3-8B and Gemma-2-2B as measured by MACs reduction.

Significance. The central idea—training a model under a distribution of preservation rates so that a single checkpoint can be accelerated at inference time without further fine-tuning—is conceptually interesting and would be practically valuable if substantiated. The paper also makes a commendable effort to state limitations and to visualize per-layer token preservation. However, the empirical evidence as reported is not reliable: the flagship modern-model results in Table 3 are inconsistent with the paper's own pruning law, the 11× speedup claim is not supported by measured latency data, and the method is not compared with standard fine-tuning, random pruning, or existing token-pruning baselines. As a result, the main claims are not established.

major comments (4)

[Table 3 / Eq. (2)] The reported MACs reductions for LLaMA-3-8B and Gemma-2-2B are not attainable under the mechanism defined by Eq. (2). With N_l = α N_{l−1} (τ_l = 5 is inactive for MMLU-length sequences), the total attention MACs scale as Σ N_l^2. For LLaMA-3 (L=32) and α=0.99, the attention-only MACs ratio is α^2(1−α^{2L})/(L(1−α^2)) ≈ 0.73, giving a maximum speedup of about 1.36×; including FFN costs (which are linear in N_l) only lowers this ceiling. Yet Table 3 reports a MACs reduction of 1.70× at α=0.99. The same inconsistency appears for Gemma-2 (L=26): the theoretical ceiling is about 1.30×, while the table reports 1.67×. Thus the quantitative evidence for zero-shot acceleration on state-of-the-art models is internally inconsistent with the proposed algorithm.
[Section 4.4 / Tables 1–3] The headline speedup claim is not backed by actual latency measurements. Table 1 lists speedups 1×–11× but reports only accuracy/F1, and the speedup values appear to be nominal labels, not measured TTFT or end-to-end inference times. Table 2 states "up to a 3× improvement in Time-to-First-Token," yet the table contains only perplexity values with no TTFT column. Table 3 reports MACs ratios, not wall-clock speedups. Since the abstract and introduction claim up to 11× acceleration, direct latency measurements on the evaluated hardware are needed to support that claim.
[Section 3.3 / Section 4.4] The core pruning criterion is assumed rather than validated. The paper states in Section 3.3, "We assume that hidden activations with lower contributions can be eliminated in subsequent layers," but no experiment compares contribution-based pruning with random pruning, and no evidence shows that the soft-score contribution is a reliable importance measure for causal decoder-only attention. Without this ablation, the reported accuracy maintenance could be an artifact of any aggressive token-dropping schedule rather than a property of the proposed contribution criterion.
[Appendix B / Eq. (2)] Appendix B states that "all hidden activations are displayed, including those marked as pruned but not removed," and that "this allows some activations to be reactivated in later layers, which is an exceptional case compared to the method description." This directly conflicts with the deterministic recursion in Eq. (2) and with the MACs accounting in Section 4.4. If pruned activations can reappear in later layers, the actual compute reduction is ambiguous and the theoretical inconsistency in Table 3 cannot be resolved without a precise description of the reactivation mechanism and its cost.

minor comments (5)

[General] No code, trained checkpoints, or random seeds are provided; Section 6 acknowledges this, but given the magnitude of the claims, reproducibility remains a concern.
[Appendix A, Table 4 vs. Section 4.2] The model size for GPT-2 is inconsistent: Section 4.2 and Table 1 describe GPT-2 as 137M parameters, while Appendix A Table 4 lists 355M parameters and 24 layers. The authors should clarify which GPT-2 variant was used.
[Section 4.1] MMLU is described as an "instruction-tuning" task, but MMLU is a multiple-choice knowledge benchmark typically evaluated in few-shot setting. This wording is inaccurate.
[Table 3] The term "MACs Reduction" is not defined: it is unclear whether the values are baseline-to-pruned ratios, percentage reductions, or estimates from an analytical model. The GPU footprint column is also not explained (e.g., how constant self-model memory is excluded).
[Section 4.4] The claim that the method "does not alter the core model architecture" is somewhat misleading: inserting a Resource Adjustment Layer after each attention layer is an architectural modification, even if parameters are not added.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central claims are empirical measurements under a specified training policy, not predictions derived from fitted parameters or self-citations.

full rationale

The paper's core proposal is a training-time sampling policy (Eq. 3) and a layer-wise token retention law (Eq. 2), followed by measured inference results. The acceleration and quality numbers in Tables 1-3 are reported measurements, not quantities derived from Eq. (2) or from the training distribution in a way that would make them true by construction. The α values are evaluation hyperparameters, not fitted parameters renamed as predictions. The hidden-activation-contribution heuristic in Section 3.3 is an explicit assumption, not a circular definition. The only self-citation (Kachuee and Sharifkhani 2025) appears in related-work enumerations and is not used as evidence for the central claim. The post-hoc selection of per-dataset α ranges and the use of the proposed fine-tuning policy for the 1× baseline are experimental-design choices, not constructional equivalences. The reported Table 3 MACs reductions that appear inconsistent with the theoretical maximum implied by Eq. (2) point to a correctness/consistency problem, not to circularity in the derivation chain. No load-bearing step reduces to its own input by definition.

Assumptions & free parameters 2 free parameters · 3 assumptions · 1 invented entities

The central claim rests on the heuristic pruning criterion, the chosen τ_l=5, the per-model α sampling ranges, and the assumption that the compute reduction follows the simple recursion. LoRA rank, quantization bits, and data splits for MMLU are unspecified, adding further unaccounted degrees of freedom.

free parameters (2)

minimum activation threshold τ_l = 5 (fixed for all models)
Chosen by hand to prevent collapse to a single token; no ablation reported.
α sampling range = U(0.02,1.00) for GPT-2, U(0.50,1.00) for LLaMA-3/Gemma-2
The uniform range is a design choice; the narrower range for larger models is a post-hoc deviation from the proposed method in Section 3.4.

assumptions (3)

domain assumption Low contribution to attention output implies low importance for downstream layers
Stated in Section 3.3 as an assumption; no theoretical or empirical justification, and no comparison to random pruning.
domain assumption The token-count recursion N_l = max(α N_{l-1}, τ_l) accurately models the compute reduction
Equation 2 defines the pruning schedule, but the actual speedup on GPU is not derived from it; the table values (e.g., MACs reduction vs α) appear inconsistent.
domain assumption Uniformly random α during fine-tuning does not degrade the model's full-accuracy baseline
The 1x baseline is the model trained with random α, not a standard fine-tuned baseline, so the cost of the training policy is not measured.

invented entities (1)

Resource Adjustment Layer (pruning module)
purpose: Prunes hidden activations based on attention contribution at each layer
This is the core architectural addition; its effectiveness is only demonstrated within this paper, no external validation or comparison.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Efficient Large Language Models with Zero-Shot Adjustable Acceleration." pith.science (2026). https://pith.science/paper/7VHYZGUT

@misc{pith2026250901190,
  author       = {Pith},
  title        = {Pith review of: Efficient Large Language Models with Zero-Shot Adjustable Acceleration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7VHYZGUT}},
  note         = {Machine review of arXiv:2509.01190}
}

read the original abstract

Using Large Language Models (LLMs) in real-world applications presents significant challenges, particularly in balancing computational efficiency with model performance. Optimizing acceleration after fine-tuning and during inference is critical for building efficient architectures. This paper introduces Zero-Shot Adjustable Acceleration, a novel training and inference method that dynamically adjusts hardware utilization during inference without requiring additional fine-tuning. The proposed approach is applied to recent LLMs and evaluated across multiple classification and text generation tasks. Experimental results demonstrate that the method supports a wide range of zero-shot acceleration and achieves up to 11x speedup compared to the baseline.

Figures

Figures reproduced from arXiv: 2509.01190 by the authors.

**Figure 1.** The proposed dynamic resource attention architecture. on the number of preserved activations, preventing extreme cases where only a single token survives in the attention layer—a scenario that would destabilize the model. This clipping mechanism not only improves stability during fine-tuning but also enhances inference performance when operating under high pruning rates. 3.4 Zero-Shot Adjustable Acceleration Optimi… view at source ↗

**Figure 2.** Visualization of GPT-2 (137M parameters) performance across three dimensions: (A) zero-shot accelera [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Binary visualization of hidden activation preservation across GPT-2 layers on the IMDB ( [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

13 extracted references · 7 canonical work pages

[4]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al

Lazyllm: Dynamic token pruning for effi- cient long context llm inference.arXiv preprint arXiv:2407.14057. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of mod- els.arXiv preprint arXiv:2407.21783. Han Guo, Philip...

arXiv 2024
[7]

SmartBERT: A Promotion of Dynamic Early Exiting Mechanism for Accelerating BERT Inference

Smartbert: A promotion of dynamic early exiting mechanism for accelerating bert inference. arXiv preprint arXiv:2303.09266. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adap- tation of large language models.arXiv preprint arXiv:2106.09685. Quzhe Huang, Zhenwei An, Nan Zh...

work page Pith review arXiv 2021
[8]

Heejun Lee, Minki Kang, Youngwan Lee, and Sung Ju Hwang

A fast post-training pruning framework for transformers.Advances in Neural Information Pro- cessing Systems, 35:24101–24116. Heejun Lee, Minki Kang, Youngwan Lee, and Sung Ju Hwang. 2022. Sparse token transformer with atten- tion back tracking. InThe Eleventh International Conference on Learning Representations. Yaniv Leviathan, Matan Kalman, and Yossi Matias

work page 2022
[9]

RT-LM: Uncertainty-Aware Resource Management for Real-Time Inference of Language Models

Fast inference from transformers via spec- ulative decoding. InInternational Conference on Machine Learning, pages 19274–19286. PMLR. Yufei Li, Zexin Li, Wei Yang, and Cong Liu. 2023. Rt- lm: Uncertainty-aware resource management for real- time inference of language models.arXiv preprint arXiv:2309.06619. Wei Lou, Lei Xun, Amin Sabet, Jia Bi, Jonathon Har...

work page Pith review arXiv 2023
[11]

InProceedings of the 16th conference of the European chapter of the association for computational linguistics: Main V olume, pages 91–104

Berxit: Early exiting for bert with better fine- tuning and extension to regression. InProceedings of the 16th conference of the European chapter of the association for computational linguistics: Main V olume, pages 91–104. Ting-Bing Xu and Cheng-Lin Liu. 2020. Deep neural network self-distillation exploiting data representa- tion invariance.IEEE Transact...

work page 2020
[12]

In 2023 60th ACM/IEEE Design Automation Confer- ence (DAC), pages 1–6

Tf-mvp: Novel sparsity-aware transformer accelerator with mixed-length vector pruning. In 2023 60th ACM/IEEE Design Automation Confer- ence (DAC), pages 1–6. IEEE. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for...

work page 2023
[2011]

Blue squares represent preserved tokens, white squares represent pruned tokens, and the red line indicates the overall preservation trend per layer

dataset at different preservation rates (α). Blue squares represent preserved tokens, white squares represent pruned tokens, and the red line indicates the overall preservation trend per layer

work page
[2014]

One billion word benchmark for measuring progress in statistical language modeling. Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023.URL https://arxiv. org/abs/2307.08691. Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashattention: Fast and memory-efficient exact attention with...

arXiv 2023

Show all 13 references

[2020]

Bodun Hu, Jiamin Li, Le Xu, Myungjin Lee, Ak- shay Jajoo, Geon-Woo Kim, Hong Xu, and Aditya Akella

Measuring massive multitask language under- standing.arXiv preprint arXiv:2009.03300. Bodun Hu, Jiamin Li, Le Xu, Myungjin Lee, Ak- shay Jajoo, Geon-Woo Kim, Hong Xu, and Aditya Akella. 2024. Blockllm: Multi-tenant finer-grained serving for large language models.arXiv preprint...

2009 arXiv
[2021]

InICASSP 2021-2021 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7713–7717

Elbert: Fast albert with confidence-window based early exit. InICASSP 2021-2021 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7713–7717. IEEE. Ji Xin, Raphael Tang, Yaoliang Yu, and Jimmy Lin

2021
[2022]

Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao

Token merging: Your vit but faster.arXiv preprint arXiv:2210.09461. Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao

arXiv
[2023]

arXiv preprint arXiv:2311.12023

Lq-lora: Low-rank plus quantized matrix de- composition for efficient language model finetuning. arXiv preprint arXiv:2311.12023. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt

arXiv
[2024]

org/abs/2401.10774

Medusa: Simple llm inference acceleration framework with multiple decoding heads.URL https://arxiv. org/abs/2401.10774. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson

arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [4]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al

Lazyllm: Dynamic token pruning for effi- cient long context llm inference.arXiv preprint arXiv:2407.14057. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of mod- els.arXiv preprint arXiv:2407.21783. Han Guo, Philip...

arXiv 2024

[2] [7]

SmartBERT: A Promotion of Dynamic Early Exiting Mechanism for Accelerating BERT Inference

Smartbert: A promotion of dynamic early exiting mechanism for accelerating bert inference. arXiv preprint arXiv:2303.09266. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adap- tation of large language models.arXiv preprint arXiv:2106.09685. Quzhe Huang, Zhenwei An, Nan Zh...

work page Pith review arXiv 2021

[3] [8]

Heejun Lee, Minki Kang, Youngwan Lee, and Sung Ju Hwang

A fast post-training pruning framework for transformers.Advances in Neural Information Pro- cessing Systems, 35:24101–24116. Heejun Lee, Minki Kang, Youngwan Lee, and Sung Ju Hwang. 2022. Sparse token transformer with atten- tion back tracking. InThe Eleventh International Conference on Learning Representations. Yaniv Leviathan, Matan Kalman, and Yossi Matias

work page 2022

[4] [9]

RT-LM: Uncertainty-Aware Resource Management for Real-Time Inference of Language Models

Fast inference from transformers via spec- ulative decoding. InInternational Conference on Machine Learning, pages 19274–19286. PMLR. Yufei Li, Zexin Li, Wei Yang, and Cong Liu. 2023. Rt- lm: Uncertainty-aware resource management for real- time inference of language models.arXiv preprint arXiv:2309.06619. Wei Lou, Lei Xun, Amin Sabet, Jia Bi, Jonathon Har...

work page Pith review arXiv 2023

[5] [11]

InProceedings of the 16th conference of the European chapter of the association for computational linguistics: Main V olume, pages 91–104

Berxit: Early exiting for bert with better fine- tuning and extension to regression. InProceedings of the 16th conference of the European chapter of the association for computational linguistics: Main V olume, pages 91–104. Ting-Bing Xu and Cheng-Lin Liu. 2020. Deep neural network self-distillation exploiting data representa- tion invariance.IEEE Transact...

work page 2020

[6] [12]

In 2023 60th ACM/IEEE Design Automation Confer- ence (DAC), pages 1–6

Tf-mvp: Novel sparsity-aware transformer accelerator with mixed-length vector pruning. In 2023 60th ACM/IEEE Design Automation Confer- ence (DAC), pages 1–6. IEEE. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for...

work page 2023

[7] [2011]

Blue squares represent preserved tokens, white squares represent pruned tokens, and the red line indicates the overall preservation trend per layer

dataset at different preservation rates (α). Blue squares represent preserved tokens, white squares represent pruned tokens, and the red line indicates the overall preservation trend per layer

work page

[8] [2014]

One billion word benchmark for measuring progress in statistical language modeling. Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023.URL https://arxiv. org/abs/2307.08691. Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashattention: Fast and memory-efficient exact attention with...

arXiv 2023