Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

CLaSp: In-Context Layer Skip for Self-Speculative Decoding

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CLaSp claims that dynamically skipping intermediate decoder layers, selected by a dynamic programming search over hidden states after each verification, accelerates LLaMA decoding by 1.3x–1.7x without training a draft model.

desk verdict CLaSp's per-verification DP layer-skip selection is genuinely new, but the table contradicts the abstract's speedup range. read the letter →

arxiv 2505.24196 v1 pith:RPLELBL5 submitted 2025-05-30 cs.CL

Longze Chen , Renke Shan , Huiming Wang , Lu Wang , Ziqiang Liu , Run Luo , Jiawei Wang , Hamid Alinejad-Rokny
show 1 more author
Min Yang
This is my paper
classification cs.CL
keywords speculativedecodingself-speculativelayerskippingdynamicprogramminginferenceaccelerationlargelanguagemodelstraining-freeLLaMA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces CLaSp, a self-speculative decoding method that speeds up large language model generation without training an extra draft model. Unlike prior methods that fix a layer-skip pattern ahead of time, CLaSp recomputes which layers to skip after every verification round, guided by the hidden states of the most recently accepted token. This dynamic layer skipping creates a compressed draft model without altering the output distribution, since the full model still verifies every token. The authors report wall-clock speedups of 1.3x to 1.7x across LLaMA3, LLaMA2, and LLaMA3.1 models on the Spec-Bench benchmark.

What carries the argument

The core object is a dynamic programming recurrence over hidden states, where state $g(i,j)$ represents the best hidden state obtainable by skipping $j$ of the first $i$ transformer layers, scored by cosine similarity to the full-model hidden state $h_i$. The recurrence compares keeping versus skipping layer $i-1$, and a backtracking pass recovers the best skip set $S^*$. Two empirical properties make the method work: an 'approximate Markov property' (the next hidden state depends mostly on the immediately preceding layer, so a greedy forward pass finds good skip sets) and 'sparse persistence' (skip sets for nearby tokens are highly similar, so the layer optimization can be run less frequently). A sequence-parallel implementation reduces the dynamic programming overhead to roughly the cost of one verification step.

What would settle it

Run CLaSp on a model where intermediate embeddings are known to change abruptly between layers (or a model with heterogeneous layer functions), and compare the DP-selected skip set to a brute-force search over all skip sets for the same context; if the cosine similarity of the resulting hidden states falls below the level reported in Figure 3b, or the average acceptance length drops below the ranges in Table 1, the approximate Markov assumption fails. Alternatively, on a task with rapidly shifting topics, measure whether sparse persistence breaks down and whether the dynamic updates actually track the optimal skip set.

Watch

Extended reading notes

Core claim

The central claim is that the optimal set of layers to skip when drafting changes with the context, and that the last verification step's hidden states carry enough information to recompute a near-optimal skip set cheaply. CLaSp frames the search as a dynamic programming problem: for each prefix of transformer layers and each allowed number of skips, it tracks the hidden state that best matches the full model's hidden state in cosine similarity. The solution selects a skip set, the compressed draft model generates K tokens, and after verification the whole process repeats using the newly obtained hidden states. The paper reports consistent speedups of 1.3x–1.7x over vanilla autoregressive decoding on Spec-Bench, with larger models such as LLaMA3.1-405B gaining more (up to 1.8x on some tasks).

Load-bearing premise

The load-bearing premise is the 'approximate Markov property': the paper admits that the dynamic programming does not strictly satisfy the Markov property, so the skip set it finds is only approximately optimal, and the whole method relies on this approximation being close enough that the speedup holds; this premise is validated only on 12 of 32 layers of LLaMA3-8B in Figure 3b, not on the models and tasks used for the headline speedups.

Editorial extensions

If this is right

  • If CLaSp holds up, it provides a training-free acceleration method for any LLM with layer redundancy, including models for which no compatible draft model exists.
  • The method implies that layer redundancy in LLMs is context-dependent, so static layer-skip configurations leave speed on the table.
  • The reported speedups are hardware-dependent, but the acceptance-length improvements suggest the gains will translate across memory-bound decoding settings.
  • Larger models benefit more, so CLaSp could help offset the inference cost of scaling model size.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An open extension suggested by the paper is combining CLaSp's dynamic skip selection with tree-based drafting or a draft-exiting threshold, since the paper keeps those dimensions orthogonal and untested.
  • The reliance on cosine similarity and the approximate Markov property implies the method should transfer to other Transformer-architecture LMs as long as their layer embeddings evolve smoothly; a direct test would be running CLaSp on models like Qwen or Mistral.
  • If sparse persistence holds for longer token distances in larger models, the layer optimization interval could be increased beyond 128 tokens to save even more overhead.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes CLaSp, a training-free self-speculative decoding method that dynamically selects which intermediate layers to skip at each decoding step. The selection is guided by a dynamic programming algorithm that uses the hidden states of the last verified token to maximize cosine similarity with the full model's hidden states. The authors claim wall-clock speedups of 1.3x to 1.7x on LLaMA3 series models without altering the generated-text distribution, based on experiments on Spec-Bench. The paper includes analyses of parallelization, update frequency, and hyperparameter sensitivity.

Significance. If the claims hold, CLaSp would be a useful plug-and-play acceleration technique: it requires no training, no additional draft modules, and adapts per context, addressing a limitation of prior self-speculative methods with fixed skip sets. The paper also contributes the 'sparse persistence' observation and a parallelized DP that reduces optimization overhead to a small fraction of total latency. However, the significance is currently undercut by internal inconsistencies in the reported speedups, an admitted approximation in the core DP that is validated only on a fraction of one model, and the absence of any direct evidence for the distribution-preservation claim. These issues must be resolved before the contribution can be considered established.

major comments (5)
  1. [Abstract; Section 4.1; Table 1] The headline speedup range "1.3x ~ 1.7x" is inconsistent with the data in Table 1. In the greedy setting for LLaMA-3-70B, CLaSp reports speedups of 1.72x and 1.75x on NQ and GSM8K, which exceed the stated upper bound of 1.7x. In the non-greedy setting for LLaMA-3-8B, the overall speedup is 1.08x, well below the stated lower bound of 1.3x. Since the speedup ratio is the paper's central quantitative claim, the authors must reconcile the stated range with the table entries or revise the claim appropriately.
  2. [Section 3.5; Figure 3b] The paper explicitly states that CLaSp does not strictly satisfy the Markov property and that this makes it "theoretically impossible to find an exact optimal solution using Algorithm 1." The only evidence that the approximate DP is near-optimal comes from Figure 3b, which fixes the first and last 10 layers of LLaMA3-8B and validates on the remaining 12 layers. This does not directly support the near-optimality of the DP for LLaMA3-70B, LLaMA3.1-405B, or the Spec-Bench tasks where the headline speedups are reported. The authors should provide validation of the approximation on the models and tasks used in the main experiments, or explicitly qualify the speedup claims as conditional on an unvalidated approximation.
  3. [Section 4; Section 5.3.3] The paper claims CLaSp works "without altering the original distribution of the generated text," but no distributional or output-quality measurements are reported. The rationale that speculative sampling is lossless is not sufficient by itself because CLaSp introduces a Draft-Exiting Threshold (DET) that controls the acceptance of draft tokens, and the paper does not prove that this threshold preserves the target distribution. The authors should either provide a formal argument that DET does not change the sampling distribution or report empirical measures such as perplexity, output divergence, or downstream task accuracy.
  4. [Section 4, Table 1; Section 5] All speedup numbers in Table 1 appear to come from single runs, with no error bars, repeated trials, or variance estimates reported. Since speedup ratios are hardware-dependent and can vary with runtime conditions, the absence of variance reporting weakens the empirical claims, especially for comparisons that differ by only a few percentage points (e.g., CLaSp vs. SWIFT in some settings). Reporting mean and standard deviation over multiple runs would substantially strengthen the paper.
  5. [Section 3.4; Section 5.3] The three hyperparameters M (number of skipped layers), LOI (Layer Optimization Interval), and DET (Draft-Exiting Threshold) are central to the method's performance, but their values used for the main results are not disclosed, and the sensitivity analysis in Section 5.3 is performed only on LLaMA3-70B with MT-Bench. Without reporting the chosen values and their dependence on model size or task, the headline speedups are difficult to reproduce and may rely on favorable tuning.
minor comments (5)
  1. [Algorithm 1] The pseudocode has several notational ambiguities: the initialization of g arrays, the indexing in the transition when i ≤ M, and the backtracking condition (which checks equality of state vectors rather than storing decisions) are unclear and should be rewritten for reproducibility.
  2. [Figure 4 caption] The caption refers to "Draft-Existing Threshold," which appears to be a typo for "Draft-Exiting Threshold" used in Section 5.3.3.
  3. [References] The reference for LISA is listed as "2024. Lisa: Layerwise importance sampling..." with no author names and a link-like placeholder; this should be corrected to the full citation.
  4. [Section 3.4] The transition equation D(i,j) = max{cosine(h_i, g(i−1,j−1)), cosine(h_i, f_{i−1}(g(i−1,j)))} is not fully aligned with Algorithm 1, which uses a comparison of σ values rather than an explicit max; the relationship between the mathematical formulation and the implementation should be clarified.
  5. [Section 5.1] The latency breakdown in Figure 3c is described qualitatively, but the figure does not show the exact proportion of layer optimization latency (stated as 4.8% in the text); the figure and text should be consistent and legible.

Circularity Check

1 steps flagged · score 4.0 of 10

Hyperparameters tuned on MT-Bench make that benchmark column an in-sample fit; the method itself is otherwise a self-contained feedback controller.

  1. fitted input called prediction [Section 5.3 (Key Hyper-Parameters) and Table 1]
    "We show the effect of key hyper-parameters on the acceleration benefits of CLaSp, where all experiments were performed using the LLaMA3-70B model on MT-Bench. ... reaching an optimal value of 1.64× when 44 layers are skipped."

    The key hyper-parameters (number of skipped layers, layer-optimization interval, draft-exit threshold) are selected by maximizing speedup on MT-Bench, as shown in Figure 4. Table 1 then reports the MT-Bench column as evidence of CLaSp's speedup (1.64× on LLaMA-3-70B), so that particular cell is an in-sample optimum rather than an independent prediction. The Overall row includes this same MT-Bench result, so the headline 1.67× figure is partly contaminated by the tuning target. The other five Spec-Bench tasks are not tuning targets and retain independent evidential value, which is why the circularity is only partial.

full rationale

CLaSp is fundamentally an online feedback controller: the dynamic program selects skipped layers from the hidden states of the last accepted token, and verification is standard speculative sampling, so the central mechanism does not reduce to its inputs. The paper relies on external facts (speculative sampling preserves the target distribution; layer embeddings change slowly) rather than load-bearing self-citations. The main circularity is in the experimental reporting: Section 5.3 tunes the key hyper-parameters on MT-Bench, and Table 1 presents MT-Bench as one of the benchmark tasks supporting the claimed 1.3x-1.7x speedup. That one column, and the overall average containing it, is selected on the evaluation target rather than independently measured. The remaining tasks and the scaling-law results provide independent support, so the central claim retains genuine content. Separate non-circular concerns, such as Table 1 containing speedups above the stated 1.7x upper bound and the Markov approximation being validated only on 12 layers of one model, are correctness risks rather than circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

CLaSp's central speedup depends on three tuned hyper-parameters (M, LOI, DET) and on three empirical regularities about LLM hidden states (slowly changing embeddings, sparse persistence, cosine proxy for acceptance) that are asserted and only partially validated. No invented entities are introduced.

free parameters (3)
  • Number of skipped layers M = 44 out of 80 layers for LLaMA3-70B; about 50-60% of layers generally
    Section 5.3.1 tunes M on MT-Bench to maximize speedup; M is a per-model choice, not derived.
  • Layer Optimization Interval (LOI) = 128 (optimal on MT-Bench)
    Section 5.3.2 and Figure 4b; interval between layer optimization runs, tuned to balance acceptance length against update cost.
  • Draft-Exiting Threshold (DET) = 0.7 (optimal on MT-Bench)
    Section 5.3.3 and Figure 4c; controls when to stop drafting based on draft confidence, tuned on MT-Bench.
assumptions (4)
  • domain assumption Approximate Markov property (no aftereffect) holds for the DP state evolution in Algorithm 1.
    Section 3.5 admits the DP does not strictly satisfy no aftereffect, yet the algorithm's near-optimality rests on this holding approximately for LLM hidden states.
  • domain assumption Hidden states change slowly across adjacent transformer layers and adjacent tokens (Sparse Persistence).
    Invoked in Sections 1, 3.5, and 3.7 (Figure 3a,b) to justify the DP approximation and the lower update frequency.
  • domain assumption Cosine similarity between draft and verify hidden states is a valid proxy for token acceptance probability.
    Section 3.3 formulates the objective as minimizing cosine distance; the paper does not prove this proxy is monotone with acceptance rate.
  • standard math Speculative sampling with a matching draft distribution is lossless.
    Section 4 cites Leviathan et al. 2023 and Chen et al. 2023; this is the standard theoretical foundation the speedup claim inherits.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CLaSp: In-Context Layer Skip for Self-Speculative Decoding." pith.science (2026). https://pith.science/paper/RPLELBL5

@misc{pith2026250524196,
  author       = {Pith},
  title        = {Pith review of: CLaSp: In-Context Layer Skip for Self-Speculative Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RPLELBL5}},
  note         = {Machine review of arXiv:2505.24196}
}
read the original abstract

Speculative decoding (SD) is a promising method for accelerating the decoding process of Large Language Models (LLMs). The efficiency of SD primarily hinges on the consistency between the draft model and the verify model. However, existing drafting approaches typically require additional modules to be trained, which can be challenging to implement and ensure compatibility across various LLMs. In this paper, we propose CLaSp, an in-context layer-skipping strategy for self-speculative decoding. Unlike prior methods, CLaSp does not require additional drafting modules or extra training. Instead, it employs a plug-and-play mechanism by skipping intermediate layers of the verify model to construct a compressed draft model. Specifically, we develop a dynamic programming algorithm that optimizes the layer-skipping process by leveraging the complete hidden states from the last verification stage as an objective. This enables CLaSp to dynamically adjust its layer-skipping strategy after each verification stage, without relying on pre-optimized sets of skipped layers. Experimental results across diverse downstream tasks demonstrate that CLaSp achieves a speedup of 1.3x ~ 1.7x on LLaMA3 series models without altering the original distribution of the generated text.

Figures

Figures reproduced from arXiv: 2505.24196 by the authors.

Figure 1
Figure 1. Previous Self-SD method vs. CLaSp. Com￾pared to the previous Self-SD method, which requires costly Bayesian optimization on training dataset to se￾lect a fixed set of skipped layers, CLaSp employs a dy￾namic layer-skipping strategy that adjusts in real-time based on context. inference latency. This latency primarily stems from the autoregressive nature of LLMs, where model parameters must be loaded into GPU SRAM for… view at source ↗
Figure 2
Figure 2. The overall framework of CLaSp consists of three stages: (1) Draft, (2) Verify, (3) Layer Optimization. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. (a) Sparse Persistence Observation: Skipped layer sets selected for adjacent tokens exhibit high similarity, with this similarity gradually decreasing as the token gap increases. This observation enables layer optimization on the current token to guide subsequent drafting processes. (b) Approximate Markov Property: Cosine similarity comparisons of hidden states obtained using Brute Force, Random, and CLaSp’s dynamic… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The impact of key hyper-parameters on speedup: (a) Number of Skipped Layers; (b) Layer Optimization [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Model Size Scaling Laws of CLaSp. plug-and-play solution, offering a versatile method to enhance inference speed for a range of LLMs. 5 Analysis We present an extensive analysis of CLaSp, focus￾ing on three key aspects: the benefits of the parallel strategy (Section 5.…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. KnapSpec: Self-Speculative Decoding via Adaptive Layer Selection as a Knapsack Problem

    cs.LG 2026-02 conditional novelty 6.0 of 10

    By modeling layer skipping as a knapsack problem with context-dependent attention/MLP latencies, KnapSpec adaptively selects draft sub-networks that speed up self-speculative decoding by up to 1.47×.

Reference graph

Works this paper leans on

48 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [1]

    https://arxiv.org/abs/2403.17919 Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning

    2024. https://arxiv.org/abs/2403.17919 Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning . arXiv preprint arXiv:2403.17919

    arXiv 2024

  2. [2]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . arXiv preprint arXiv:2303.08774

    arXiv 2023

  3. [3]

    Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, and Ramachandran Ramjee. 2023. https://arxiv.org/abs/2308.16369 Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills . arXiv preprint arXiv:2308.16369

    arXiv 2023

  4. [4]

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. https://arxiv.org/abs/2305.10403 Palm 2 technical report . arXiv preprint arXiv:2305.10403

    arXiv 2023

  5. [5]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    2020

  6. [6]

    Warren Burton

    F. Warren Burton. 1985. https://doi.org/10.1109/TC.1985.6312218 Speculative computation, parallelism, and functional programming . IEEE Transactions on Computers, C-34(12):1190--1193

    work page arXiv 1985

  7. [7]

    Lee, Deming Chen, and Tri Dao

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. 2024. https://openreview.net/forum?id=PEpbUobfJv Medusa: Simple LLM inference acceleration framework with multiple decoding heads . In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net

    work page 2024

  8. [8]

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. https://arxiv.org/abs/2302.01318 Accelerating large language model decoding with speculative sampling . arXiv preprint arXiv:2302.01318

    arXiv 2023

Show all 48 references
  1. [9]

    Zhuoming Chen, Avner May, Ruslan Svirschevski, Yuhsun Huang, Max Ryabinin, Zhihao Jia, and Beidi Chen. 2024. https://arxiv.org/abs/2402.12374 Sequoia: Scalable, robust, and hardware-aware speculative decoding . arXiv preprint arXiv:2402.12374

    2024 arXiv

  2. [10]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . arXiv preprint arXiv:2110.14168

    2021 arXiv

  3. [11]

    Luciano Del Corro, Allie Del Giorno, Sahaj Agarwal, Bin Yu, Ahmed Awadallah, and Subhabrata Mukherjee. 2023. https://arxiv.org/abs/2307.02628 Skipdecode: Autoregressive skip decoding with batching and caching for efficient llm inference . arXiv preprint arXiv:2307.02628

    2023 arXiv

  4. [12]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/c3ba4962c05c49636d4c6206a97e9c8a-Paper-Conference.pdf Gpt3.int8(): 8-bit matrix multiplication for transformers at scale . In Advances in Neural Inf...

    2022

  5. [13]

    Cunxiao Du, Jing Jiang, Yuanchen Xu, Jiawei Wu, Sicheng Yu, Yongqi Li, Shenggui Li, Kai Xu, Liqiang Nie, Zhaopeng Tu, and Yang You. 2024. https://openreview.net/forum?id=mk8oRhox2l Glide with a cape: A low-hassle method to accelerate speculative decoding . In Forty-first Inter...

    2024

  6. [14]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . arXiv preprint arXiv:2407.21783

    2024 arXiv

  7. [15]

    Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed Aly, Beidi Chen, and Carole-Jean Wu. 2024. https://doi.org/10.18653/v1/2024.acl-long.681 L ayer S kip: Enabling early e...

    2024 doi

  8. [16]

    Angela Fan, Edouard Grave, and Armand Joulin. 2020. https://openreview.net/forum?id=SylO2yStDr Reducing transformer depth on demand with structured dropout . In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . Open...

    2020

  9. [17]

    Oreshkin, and Mark Coates

    Theodore Glavas, Joud Chataoui, Florence Regol, Wassim Jabbour, Antonios Valkanas, Boris N. Oreshkin, and Mark Coates. 2024. https://arxiv.org/abs/2410.20022 Dynamic layer selection in decoder-only transformers . arXiv preprint arXiv:2410.20022

    2024 arXiv

  10. [18]

    Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Marc Sun, and Benjamin Bossan. 2022. Accelerate: Training and inference at scale made simple, efficient and adaptable. https://github.com/huggingface/accelerate

    2022

  11. [19]

    Zhenyu He, Zexuan Zhong, Tianle Cai, Jason Lee, and Di He. 2024. https://doi.org/10.18653/v1/2024.naacl-long.88 REST : Retrieval-based speculative decoding . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: H...

    2024 doi

  12. [20]

    Hennessy and David A

    John L. Hennessy and David A. Patterson. 2012. Computer Architecture: A Quantitative Approach, 5 edition. Morgan Kaufmann, Amsterdam

    2012

  13. [21]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. https://arxiv.org/abs/2001.08361 Scaling laws for neural language models . arXiv preprint arXiv:2001.08361

    2020 arXiv

  14. [22]

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.550 Dense passage retrieval for open-domain question answering . In Proceedings of the 2020 Conference on Empiric...

    2020 doi

  15. [23]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019 doi

  16. [24]

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. https://proceedings.mlr.press/v202/leviathan23a.html Fast inference from transformers via speculative decoding . In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine...

    2023

  17. [25]

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024 a . https://arxiv.org/abs/2406.16858 EAGLE-2 : Faster inference of language models with dynamic draft trees . arXiv preprint arXiv:2406.16858

    2024 arXiv

  18. [26]

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024 b . https://openreview.net/forum?id=1NdN7eXyb4 EAGLE: speculative sampling requires rethinking feature uncertainty . In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27,...

    2024

  19. [27]

    Fangcheng Liu, Yehui Tang, Zhenhua Liu, Yunsheng Ni, Kai Han, and Yunhe Wang. 2024. https://arxiv.org/abs/2404.18911 Kangaroo: Lossless self-speculative decoding via double early exiting . arXiv preprint arXiv:2404.18911

    2024 arXiv

  20. [28]

    Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, and Beidi Chen. 2023. https://proceedings.mlr.press/v202/liu23am.html Deja vu: Contextual sparsity for efficient LLM s at inference time . In P...

    2023

  21. [29]

    Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. 2024. https://doi.org/10.1145/3620666.3651335 Specinfer: Acc...

    2024

  22. [30]

    Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. https://doi.org/10.18653/v1/K16-1028 Abstractive text summarization using sequence-to-sequence RNN s and beyond . In Proceedings of the 20th SIGNLL Conference on Computational Natural Langu...

    2016 doi

  23. [31]

    Patterson

    David A. Patterson. 2004. https://doi.org/10.1145/1022594.1022596 Latency lags bandwith . Commun. ACM, 47(10):71–75

    2004

  24. [32]

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. https://arxiv.org/abs/2403.05530 Gemini 1.5: Unlocking multimodal understanding across m...

    2024 arXiv

  25. [33]

    Apoorv Saxena. 2023. https://github.com/apoorvumang/prompt-lookup-decoding/ Prompt lookup decoding

    2023

  26. [34]

    Noam Shazeer. 2019. https://arxiv.org/abs/1911.02150 Fast transformer decoding: One write-head is all you need . arXiv preprint arXiv:1911.02150

    2019 arXiv

  27. [35]

    Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, and Beidi Chen. 2024. https://arxiv.org/abs/2404.11912 Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding . arXiv preprint arXiv:2404.11912

    2024 arXiv

  28. [36]

    Ruslan Svirschevski, Avner May, Zhuoming Chen, Beidi Chen, Zhihao Jia, and Max Ryabinin. 2024. https://arxiv.org/abs/2406.02532 Specexec: Massively parallel speculative decoding for interactive llm inference on consumer devices . arXiv preprint arXiv:2406.02532

    2024 arXiv

  29. [37]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023 a . https://arxiv.org/abs/2302.13971 Llama: Open and efficient foundation language models . arXiv...

    2023 arXiv

  30. [38]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 b . https://arxiv.org/abs/2307.09288 Llama 2: Open foundation and fine-tuned chat models . arXiv preprint arX...

    2023 arXiv

  31. [39]

    Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.257 Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation . In Findings of the Association for Computational Ling...

    2023 doi

  32. [40]

    Heming Xia, Yongqi Li, Jun Zhang, Cunxiao Du, and Wenjie Li. 2024 a . https://arxiv.org/abs/2410.06916 Swift: On-the-fly self-speculative decoding for llm inference acceleration . arXiv preprint arXiv:2410.06916

    2024 arXiv

  33. [41]

    Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. 2024 b . https://doi.org/10.18653/v1/2024.findings-acl.456 Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding . In Fin...

    2024 doi

  34. [42]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. https://arxiv.org/abs/2407.10671 Qwen2 technical report . arXiv preprint arXiv:2407.10671

    2024 arXiv

  35. [43]

    Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. 2024. https://doi.org/10.18653/v1/2024.acl-long.607 Draft & verify: Lossless large language model acceleration via self-speculative decoding . In Proceedings of the 62nd Annual Meeting of the As...

    2024 doi

  36. [44]

    Kaiyan Zhang, Ning Ding, Biqing Qi, Xuekai Zhu, Xinwei Long, and Bowen Zhou. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.597 CR a S h: Clustering, removing, and sharing enhance fine-tuning without full large language model . In Proceedings of the 2023 Conference on Empir...

    2023 doi

  37. [45]

    Minjia Zhang and Yuxiong He. 2020. https://proceedings.neurips.cc/paper_files/paper/2020/file/a1140a3d0df1c81e24ae954d935e8926-Paper.pdf Accelerating training of transformer-based language models with progressive layer dropping . In Advances in Neural Information Processing Sy...

    2020

  38. [46]

    Gonzalez, and Ion Stoica

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. https://openreview.net/forum?id=uccHPGDlao Judging LLM -as-a-judge with MT -bench and chatbot ...

    2023

  39. [47]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  40. [48]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.