Pith. sign in

REVIEW 5 major objections 5 minor 29 references

FTP: A Fine-grained Token-wise Pruner for Large Language Models via Token Routing

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a fine-grained token-wise pruner can skip 22–40% of token computations inside each transformer block without retraining the LLM, retaining about 99% of benchmark accuracy at low sparsity and outperforming block- and…

desk verdict FTP is a plausible token-pruning pipeline whose headline SOTA claim is undercut by tuning on WinoGrande while reporting it in the average, and by comparing token-skip ratios to depth-pruning ratios without matching measured speedup. read the letter →

arxiv 2412.11494 v1 pith:7QY5GLA7 submitted 2024-12-16 cs.CL

Zekai Li , Jintu Zheng , Ji Liu , Han Liu , Haowei Zhu , Zeping Li , Fuwei Yang , Haiduo Huang
show 4 more authors
Jinzhang Peng Dong Li Lu Tian Emad Barsoum
This is my paper · ORCID
classification cs.CL
keywords token-wisepruningtokenroutingLLMinferenceaccelerationstructuredconditionalcomputationsparsityschedulingattentionsinklargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that transformer blocks in large language models contain substantial token-level redundancy, and that a lightweight learned router can identify which tokens in each block can be skipped without retraining the model. The proposed FTP pipeline first searches a per-block sparsity schedule with a static router that always keeps the first and last tokens, then trains a shared dynamic router on four low-dimensional features, then refines the schedule. On LLaMA2-7B, LLaMA2-13B, LLaMA3-8B, and Qwen1.5-7B, FTP retains roughly 99% of average benchmark accuracy at 22–25% token sparsity, about 96–98% at 30%, and 85–93% at 40%. If true, this means inference cost can be cut substantially simply by skipping redundant token computations inside each block, with no weight removal and no model retraining.

What carries the argument

The load-bearing mechanism is the token router: a two-layer MLP shared across blocks that consumes a 4-dimensional vector per token—position, absolute attention score, relative attention-score rank, and the block's required sparsity—and outputs a compute-or-skip gate, trained with the straight-through estimator. A genetic-algorithm sparsity scheduler allocates how many tokens each block may skip, and the static router used for the initial search encodes the attention-sink heuristic that the first token and the last few tokens are the most important. The three losses (guide, sparsity constraint, distillation) bootstrap the router from the static rule, enforce the target sparsity, and align the pruned model's outputs with the dense model.

What would settle it

Run FTP on a long-document QA task where the answer and the decisive tokens sit in the middle of the sequence: if the static router's first-and-last priority and the router trained on it lose more than about 5 accuracy points relative to the dense model at 30% sparsity, while a router that attends mid-sequence tokens does not, then the core redundancy assumption is specific to the evaluated benchmark distribution.

Watch

Extended reading notes

Core claim

The central discovery is that token-wise pruning across blocks, rather than layer or width pruning, is the right granularity for LLM acceleration, and that routing decisions can be made from four cheap scalar signals—token position, absolute attention score, relative attention rank, and the block's sparsity requirement—instead of from high-dimensional hidden states. The paper shows that a static priority rule—always compute the first token and the last few tokens—already outperforms prior depth- and width-pruning methods, and that a dynamic router trained with a guide loss, a sparsity constraint, and a distillation loss improves on it further. The reported results exceed the dense-model average on Qwen1.5-7B at 22% sparsity, which the authors attribute to the router suppressing noisy or redundant computations.

Load-bearing premise

The method assumes the first token and the last few tokens of every sequence are the most important in every block, so a fixed priority rule—and the guide loss that bootstraps the dynamic router from it—preserves accuracy without dataset-specific validation.

Editorial extensions

If this is right

  • At 22–25% token sparsity, FTP reports average benchmark retention of 98.8–99.2% on LLaMA2-7B/13B and 100% on Qwen1.5-7B, so skipping roughly one token in four need not cost measurable accuracy.
  • At 40% sparsity, FTP still reports 85–93% retention, and on LLaMA2-7B it beats BlockPruner at 22% and ShortGPT at 27%, suggesting token-wise pruning dominates depth pruning at comparable budgets.
  • Because the router uses only four scalars per token and is shared, its overhead is small, and the measured speedup grows with sequence length (1.28x at 30% sparsity with 1000 tokens versus 1.39x with 2000 tokens).
  • The router can be constrained to limit last-token sparsity, making FTP compatible with KV-cache autoregressive decoding with negligible accuracy change.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The static priority rule suggests that attention-sink behavior generalizes beyond streaming and context-compression settings; a direct test would be whether the same first-and-last priority holds on tasks where the decisive tokens sit mid-sequence, because the guide loss inherits that bias if it does not.
  • Because the router is trained only on Alpaca and evaluated on ARC/MMLU/HellaSwag/WinoGrande, the 100% retention on Qwen1.5-7B may partly reflect benchmark insensitivity rather than true preservation of generative quality; perplexity on longer generations would be a stricter test.
  • The four-factor design implies that hidden states are unnecessary for routing, which points toward a broader class of conditional-computation methods where cheap scalar features could replace expensive hidden-state predictors.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes FTP, a fine-grained token-wise pruning method for LLM inference. The method has three stages: (1) a GA-based sparsity scheduler allocates per-block token-skip ratios using a static router that prioritizes the first and last tokens; (2) a dynamic router, shared across blocks, is trained on four low-dimensional features (token position, absolute attention score, relative attention rank, and block sparsity requirement) with guide, sparsity-constraint, and distillation losses; (3) the scheduler is re-searched with the trained router. Experiments on LLaMA2-7B/13B, LLaMA3-8B, and Qwen1.5-7B report high accuracy retention at nominal token sparsities of 22-40%, with speedups up to 1.61x on LLaMA2-7B.

Significance. If the efficiency-accuracy trade-off claimed in the paper were established under controlled conditions, FTP would be a useful contribution to token-level conditional computation for LLM inference: it avoids retraining the LLM, keeps the router lightweight, and the three-stage decoupling of sparsity allocation and router training is a reasonable design. The static-router analysis and ablations give some insight into which factors matter for token skipping. The paper also reports measured speedups, which is a strength. However, the headline SOTA claim is currently not established because the main comparisons mix incommensurable pruning metrics and the benchmark average is partially contaminated by the optimization set.

major comments (5)
  1. [§4.2, Table 1] The claimed 'comparable sparsity levels' is not a controlled comparison. FTP's Ratio is the fraction of tokens skipped within every retained block, whereas ShortGPT, BlockPruner, LaCo, and RM remove entire layers. Removing 21% of layers deletes all compute and memory traffic for those layers, while skipping 22% of tokens keeps every block and only shortens the per-block sequence length; the actual compute reduction differs and depends on sequence length and the attention/FFN cost split. Table 6 gives FTP's measured speedup (1.28x at 30% and 1.41x at 40%, length 1000) but no baseline speedups, so the roughly 10-point accuracy-retention margins in Table 1 cannot be read as an efficiency-accuracy advantage. A matched-latency or matched-FLOPs comparison, reported on the same hardware, is required before the SOTA claim can be assessed.
  2. [Table 1, §2] MoD (Raposo et al., 2024) is the natural token-routing baseline and is absent from all experiments. The paper cites MoD in the related work but never reports MoD numbers. Since FTP is a conditional-computation token router, a comparison only against depth/width pruners does not establish SOTA within the method's own category. Include MoD under identical sparsity schedules and sequence-length settings.
  3. [§4.1 Datasets, §4.4, Table 1] WinoGrande is used both as the dataset for GA-based sparsity optimization and for tuning the KV-cache threshold, and it is also included in the reported benchmark average. This is a partial optimization leak: the WinoGrande row and the average are not independent test results. Report the average with WinoGrande excluded, and ideally reserve a separate validation set for the scheduler and threshold.
  4. [Table 6, §4.4] The speedup measurements are under-specified. It is unclear whether the measured time includes the router forward passes, whether skipped tokens are physically gathered/scattered in the MHA and FFN kernels, and what batch size and GPU implementation were used. Without these details, and without identical measurements for the baselines, the speedup numbers cannot be compared with the baseline ratios in Table 1.
  5. [§4.4, Appendix A.8] The threshold strategy for the last token is acknowledged not to strictly meet the target KV-cache sparsity ('the threshold strategy can not strictly constrain the sparsity' in Appendix A.8). This means the reported ratio for the KV-cache-compatible variant is approximate. The main text should state which variant (threshold or strict constraint) corresponds to the headline accuracy numbers, and the strict-constraint accuracy should be reported in the main tables if that is the deployable version.
minor comments (5)
  1. [Figure 2] The similarity metric and the 0.8 threshold are undefined in the main text; specify how hidden-state cosine similarity is aggregated over tokens and blocks.
  2. [Table 3] The text says uniform allocation causes a 24.25% performance drop, but the Avg Percentage values in the table imply a drop of about 25.23 points; reconcile the arithmetic.
  3. [Equation (3), Algorithm 1] Equation (3) and Algorithm 1 refer to a generic evaluation dataset; the main text should explicitly state that WinoGrande is used for the GA search, as indicated in §4.1.
  4. [Abstract, §1] The phrase 'does not need to retrain the LLMs' may confuse readers because the router is trained on Alpaca data; clarify that the LLM weights remain frozen and only the router is trained.
  5. [Table 2] There is a typo in the caption: 'roubst' should be 'robust'. Also, 'A vg.' in Table 1 headings should be 'Avg.'.

Circularity Check

1 steps flagged · score 6.0 of 10

Partial circularity: the GA sparsity schedule is fitted on WinoGrande and WinoGrande then appears in the reported average retention; the other four benchmarks remain independent.

  1. fitted input called prediction [Section 4.1 (Datasets), Section 3.2 Eq. (3), Table 1]
    "Specifically, we utilize the WinoGrande to optimize the sparsity ratios due to its various token length. We report the accuracies together with average accuracy retention percentages on these benchmarks."

    Equation (3) defines the search objective as S* = argmax_S Accuracy(theta(R(S), X), Y), and Section 4.1 identifies the evaluation set for this search as WinoGrande. The same WinoGrande accuracy is then reported as one of the five benchmark columns entering Table 1's average retention. FTP's WinoGrande score is therefore the optimized objective of the GA scheduler, not an out-of-sample prediction, and the reported average is partly forced by the fit. The other four benchmarks (ARC-c, ARC-e, HellaSwag, MMLU) were not used in the search and remain independent evidence for the method's accuracy.

full rationale

The only load-bearing circularity I can exhibit from the paper's own text is the WinoGrande fit-and-report loop. Section 4.1 states that WinoGrande is used to optimize the sparsity ratios, and Equation (3) makes the optimization objective explicit: the GA selects the block-wise sparsity configuration that maximizes accuracy on exactly that evaluation set. Table 1 then includes WinoGrande in the average retention percentage for FTP, so part of the headline number is the value of the fitted objective rather than an independent benchmark result. This matches the fitted_input_called_prediction pattern and justifies a partial circularity score of 6. The KV-cache threshold is also selected on WinoGrande (Section 4.4), which compounds the selection bias, though the reported KV-cache results in Appendix A.8 are on ARC-c, MMLU, and perplexity rather than WinoGrande. I do not find self-citation load-bearing circularity: the static-router priority rule is an external heuristic imported from the attention-sink literature, not a derivation, and there is no uniqueness-theorem self-citation. The concern that FTP's token-skip ratio is compared with baselines' layer-removal ratios under the label 'comparable token sparsity levels' is a correctness and comparability issue, not a circularity issue. The non-WinoGrande benchmarks and the ablations provide genuinely independent evidence, which is why the circularity is partial rather than total.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The core method rests on a small set of hand-chosen hyperparameters and domain assumptions. The sparsity ratios are directly fitted to WinoGrande, and the router's input features rely on unvalidated assumptions about attention scores and token positions. No new physical or conceptual entities are postulated; the router is a standard MLP.

free parameters (5)
  • Per-block sparsity ratios S* = e.g., LLaMA2-7B initial: block 16 0.2596, block 30 0.5962 (Appendix A.7)
    Searched via genetic algorithm to maximize accuracy on WinoGrande, which is also a reported benchmark, so these are fitted values that leak into evaluation.
  • KV cache last-token threshold = 0.5
    Determined through evaluations on the WinoGrande dataset (Section 4.4).
  • Loss weights lambda_d, lambda_s, lambda_g = 1 initially, lambda_g decays to 0 after halfway
    Chosen by hand; no sensitivity study is reported.
  • Router hidden size = 64
    Architecture choice, no ablation reported.
  • GA hyperparameters = population 50, generations 10, mutation 0.2
    Chosen without reported sensitivity analysis.
assumptions (5)
  • domain assumption Token redundancy measured by hidden-state similarity implies a token can be skipped without performance loss.
    Section 3.1 uses similarity between input and output hidden states of each block; assumes high similarity means the block does not need to compute that token.
  • domain assumption Attention scores reflect token importance and redundancy; high association means a token is replaceable.
    Section 3.2 and A.5 define the attention-score factor; the premise that attention scores are a good pruning signal is assumed.
  • domain assumption The first token and the last few tokens are the most important and must always be computed.
    Borrowed from attention-sink literature (Xiao et al. 2024); used to construct the static router in Section 3.2.
  • domain assumption A router trained on Alpaca generalizes to other datasets and models.
    The dynamic router is trained on Alpaca (Section 4.1) and evaluated on ARC, HellaSwag, MMLU, and WinoGrande.
  • standard math The straight-through estimator gives usable gradients for the binary gate.
    Standard technique, cited to Jang et al. 2016.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FTP: A Fine-grained Token-wise Pruner for Large Language Models via Token Routing." pith.science (2026). https://pith.science/paper/7QY5GLA7

@misc{pith2026241211494,
  author       = {Pith},
  title        = {Pith review of: FTP: A Fine-grained Token-wise Pruner for Large Language Models via Token Routing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7QY5GLA7}},
  note         = {Machine review of arXiv:2412.11494}
}
read the original abstract

Recently, large language models (LLMs) have demonstrated superior performance across various tasks by adhering to scaling laws, which significantly increase model size. However, the huge computation overhead during inference hinders the deployment in industrial applications. Many works leverage traditional compression approaches to boost model inference, but these always introduce additional training costs to restore the performance and the pruning results typically show noticeable performance drops compared to the original model when aiming for a specific level of acceleration. To address these issues, we propose a fine-grained token-wise pruning approach for the LLMs, which presents a learnable router to adaptively identify the less important tokens and skip them across model blocks to reduce computational cost during inference. To construct the router efficiently, we present a search-based sparsity scheduler for pruning sparsity allocation, a trainable router combined with our proposed four low-dimensional factors as input and three proposed losses. We conduct extensive experiments across different benchmarks on different LLMs to demonstrate the superiority of our method. Our approach achieves state-of-the-art (SOTA) pruning results, surpassing other existing pruning methods. For instance, our method outperforms BlockPruner and ShortGPT by approximately 10 points on both LLaMA2-7B and Qwen1.5-7B in accuracy retention at comparable token sparsity levels.

Figures

Figures reproduced from arXiv: 2412.11494 by the authors.

Figure 1
Figure 1. Overview of LLM structure and router workflow. (Left) Dense Transformer where all [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Token similarity across different transformer blocks. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Overview of our method. (a) Our Fine-Grained Token Pruning uses token position [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 4 canonical work pages

  1. [1]

    Shadowllm: Predictor-based contextual sparsity for large language models

    Yash Akhauri, Ahmed F AbouElhamayed, Jordan Dotzel, Zhiru Zhang, Alexander M Rush, Safeen Huda, and Mohamed S Abdelfattah. Shadowllm: Predictor-based contextual sparsity for large language models. arXiv preprint arXiv:2406.16635,

    arXiv

  2. [3]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    arXiv

  3. [5]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  4. [6]

    Depth-adaptive transformer

    Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. Depth-adaptive transformer. In ICLR 2020-Eighth International Conference on Learning Representations , pp. 1–14,

    work page 2020

  5. [8]

    Xl-sum: Large-scale multilingual abstractive sum- marization for 44 languages

    Tahmid Hasan, Abhik Bhattacharjee, Md Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M Sohel Rahman, and Rifat Shahriyar. Xl-sum: Large-scale multilingual abstractive sum- marization for 44 languages. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pp. 4693–4703,

    work page 2021

  6. [10]

    In-context learning distilla- tion: Transferring few-shot learning ability of pre-trained language models

    Yukun Huang, Yanda Chen, Zhou Yu, and Kathleen McKeown. In-context learning distilla- tion: Transferring few-shot learning ability of pre-trained language models. arXiv preprint arXiv:2212.10670,

    arXiv

  7. [13]

    Losparse: Structured compression of large language models based on low-rank and sparse approx- imation

    Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, and Tuo Zhao. Losparse: Structured compression of large language models based on low-rank and sparse approx- imation. In International Conference on Machine Learning , pp. 20336–20350. PMLR, 2023a. Yucheng Li, Bo Dong, Chenghua Lin, and Frank Guerin. Compressing context to enhance i...

    arXiv

  8. [14]

    Anytime dense pre- diction with confidence adaptivity

    Zhuang Liu, Zhiqiu Xu, Hung-Ju Wang, Trevor Darrell, and Evan Shelhamer. Anytime dense pre- diction with confidence adaptivity. arXiv preprint arXiv:2104.00749,

    arXiv

Show all 29 references
  1. [15]

    Shortgpt: Layers in large language models are more redundant than you expect

    Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.03853,

    arXiv

  2. [16]

    Large language models: A survey

    Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Am- atriain, and Jianfeng Gao. Large language models: A survey. arXiv preprint arXiv:2402.06196,

    arXiv

  3. [17]

    Mixture-of-depths: Dynamically allocating compute in transformer-based lan- guage models

    David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-depths: Dynamically allocating compute in transformer-based lan- guage models. arXiv preprint arXiv:2404.02258,

    arXiv

  4. [18]

    Weight subcloning: direct initialization of transformers using larger pretrained ones

    Mohammad Samragh, Mehrdad Farajtabar, Sachin Mehta, Raviteja Vemulapalli, Fartash Faghri, Devang Naik, Oncel Tuzel, and Mohammad Rastegari. Weight subcloning: direct initialization of transformers using larger pretrained ones. arXiv preprint arXiv:2312.09299,

    arXiv

  5. [19]

    A simple and effective pruning approach for large language models

    12 Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695,

    arXiv

  6. [20]

    Llama 2: Open founda- tion and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288,

    arXiv

  7. [21]

    Model compression and efficient inference for large language models: A survey

    Wenxiao Wang, Wei Chen, Yicong Luo, Yongliu Long, Zhengkai Lin, Liye Zhang, Binbin Lin, Deng Cai, and Xiaofei He. Model compression and efficient inference for large language models: A survey. arXiv preprint arXiv:2402.09748,

    arXiv

  8. [22]

    Sheared llama: Accelerating language model pre-training via structured pruning

    Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared llama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.06694,

    arXiv

  9. [23]

    Laco: Large language model pruning via layer collapse

    Yifei Yang, Zouying Cao, and Hai Zhao. Laco: Large language model pruning via layer collapse. arXiv preprint arXiv:2402.11187,

    arXiv

  10. [24]

    Did you read the instructions? rethinking the effectiveness of task definitions in instruction learning

    Fan Yin, Jesse Vig, Philippe Laban, Shafiq Joty, Caiming Xiong, and Chien-Sheng Jason Wu. Did you read the instructions? rethinking the effectiveness of task definitions in instruction learning. arXiv preprint arXiv:2306.01150,

    arXiv

  11. [25]

    Hellaswag: Can a ma- chine really finish your sentence? arXiv preprint arXiv:1905.07830,

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence? arXiv preprint arXiv:1905.07830,

    1905 arXiv

  12. [27]

    A survey of large language models

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223,

    arXiv

  13. [28]

    Blockpruner: Fine- grained pruning for large language models

    Longguang Zhong, Fanqi Wan, Ruijun Chen, Xiaojun Quan, and Liangzhi Li. Blockpruner: Fine- grained pruning for large language models. arXiv preprint arXiv:2406.10594,

    arXiv

  14. [29]

    Table 8: Block-wise sparsity ratios obtained by sparsity scheduler for overall 30% sparsity. Model Results (Block ID: Sparsity ratio) LLama-2-7B 16: 0.2596, 17: 0.3987, 18: 0.4808, 19: 0.5481, 20: 0.5451, 21: 0.6642, 22: 0.682, (Initial) 23: 0.7337, 24: 0.7589, 25: 0.7973, 26:...

    1904

  15. [2016]

    Llmlingua: Compressing prompts for accelerated inference of large language models

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Llmlingua: Compressing prompts for accelerated inference of large language models. arXiv preprint arXiv:2310.05736 , 2023a. Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and...

    arXiv

  16. [2019]

    Apt: Adaptive pruning and tuning pretrained language models for efficient training and inference

    Bowen Zhao, Hannaneh Hajishirzi, and Qingqing Cao. Apt: Adaptive pruning and tuning pretrained language models for efficient training and inference. arXiv preprint arXiv:2401.12200,

    arXiv

  17. [2020]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord

    URL https: //openreview.net/pdf?id=r1xMH1BtvB. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457,

    arXiv

  18. [2021]

    Measuring massive multitask language understanding

    11 Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300,

    2009 arXiv

  19. [2022]

    Categorical reparameterization with gumbel-softmax.arXiv preprint arXiv:1611.01144,

    Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax.arXiv preprint arXiv:1611.01144,

    arXiv

  20. [2023]

    Discrete model compression with resource constraint for deep neural networks

    Shangqian Gao, Feihu Huang, Jian Pei, and Heng Huang. Discrete model compression with resource constraint for deep neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1899–1908,

    1908

  21. [2024]

    Slicegpt: Compress large language models by deleting rows and columns

    Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401.15024,

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.