Pith. sign in

REVIEW 4 major objections 6 minor 4 cited by

This paper claims that EAGLE-based speculative decoding can be made fast enough for production-scale Llama serving through training changes (online distillation, longer training, dense 3-layer draft models) and inference engineering (split

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Meta reports EAGLE-based speculative decoding optimizations for Llama models, achieving state-of-the-art latency (about 4 ms/token for Llama4 Maverick) and 1.4-2.0x speedups for large batches.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Solid engineering report with real optimizations, but the headline SOTA claim leans on an unnamed baseline and a tree-vs-chain comparison; deserves review with revisions. the 4 major comments →

arxiv 2508.08192 v1 pith:4PX7HNEC submitted 2025-08-11 cs.CL

Efficient Speculative Decoding for Llama at Scale: Challenges and Solutions

Bangsheng Tang , Carl Chengyan Fu , Fei Kou , Grigory Sizov , Haoci Zhang , Jason Park , Jiawen Liu , Jie You
show 30 more authors
Qirui Yang Sachin Mehta Shengyong Cai Xiaodong Wang Xingyu Liu Yunlu Li Yanjun Zhou Wei Wei Zhiwei Zhao Zixi Qi Adolfo Victoria Aya Ibrahim Bram Wasti Changkyu Kim Daniel Haziza Fei Sun Giancarlo Delfin Emily Guo Jialin Ouyang Jaewon Lee Jianyu Huang Jeremy Reizenstein Lu Fang Quinn Zhu Ria Verma Vlad Mihailescu Xingwen Guo Yan Cui Ye Hu Yejin Lee
This is my paper
classification cs.CL
keywords speculative decodingEAGLEtree attentionLLM inferenceLlamalatency optimizationproduction servingKV cache
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reports that EAGLE-style speculative decoding—where a small draft model proposes several tokens and the large model verifies them at once—can be made fast enough for production-scale serving of Llama models, a regime where earlier open implementations saw speedups fall below 1x as batch size grew. The authors identify two bottlenecks: draft-model quality (how many proposed tokens are accepted) and GPU under-utilization from inefficient tree attention, sampling loops, and CPU-GPU synchronization. They address the first with online distillation, longer training, and a three-layer dense draft model, and the second with a split prefix/suffix tree attention, compiled multi-round sampling, disaggregated prefill/decode with latency hiding, GPU-graph capture, and KV-cache management. The claimed result is a new best decode latency for Llama, e.g., Llama4 Maverick at about 4 ms per token on 8 H100 GPUs (10% faster than the previously best known method), and 1.4x-2.0x speedups for EAGLE at large production batch sizes.

Core claim

The central claim is that EAGLE-based speculative decoding, previously regarded as useful mainly at small batch sizes, can be made the default path for large Llama models in production. On the training side, the draft model is trained with a weighted loss $L = 0.1\, L_{\mathrm{CE}} + 1.0\, L_{\mathrm{L1}}$ that matches both the base model's logits (cross-entropy) and its pre-softmax hidden states (smooth L1); training for 48k iterations and using a 3-layer dense draft model instead of a 1-layer or MoE draft gives accepted-token counts (TPC) comparable to or better than the EAGLE3 baseline at speculation length 3. On the inference side, the decisive mechanism is a two-pass tree attention that

What carries the argument

The load-bearing mechanism is split tree attention: draft tokens are flattened into one sequence, and attention is computed in two pieces—attention from query to the prefix context (large, but no custom mask needed) and attention among the draft tokens themselves (small, but needs the tree mask). The two partial results are merged with merge_attentions, avoiding the giant explicit mask that naive tree attention would require. The other central object is the draft model's training objective $L = \lambda_{\mathrm{CE}}\, L_{\mathrm{CE}} + \lambda_{\mathrm{L1}}\, L_{\mathrm{L1}}$ (with $\lambda_{\mathrm{CE}}=0.1$, $\lambda_{\mathrm{L1}}=1.0$), which distills the frozen base model's hidden states

Load-bearing premise

The load-bearing premise is that the comparison points are fair: the batched speedups are measured against a non-speculative baseline in the authors' own engine, and the '10% faster' claim is against an open-source chain-based implementation that the authors consider the best prior method; if a stronger or differently tuned baseline exists, the stated margins can shrink.

What would settle it

Run the same four models on the same 8xH100 hardware with an independently tuned tree-based speculative decoding engine that also uses split tree attention, GPU-graph capture, and a tree dispatcher, and compare TTIT at batch sizes 1, 16, and 32 at 8k context. If the alternative engine matches the 4 ms/token Llama4 Maverick number within 5% or the large-batch speedup relative to non-speculative decoding drops below 1.4x, the paper's central advantage claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the claims hold, EAGLE-style speculative decoding can be deployed on live Llama workloads without losing its advantage at large batch sizes; the paper reports 1.4x-2.0x speedup over non-speculative decoding in that regime.
  • The 4 ms-per-token figure for Llama4 Maverick implies that a 400B-parameter MoE model can sustain interactive single-user latency on 8 H100s, making speculative decoding a practical alternative to buying more GPUs.
  • Because the optimized tree attention is presented as reusable, other tree-based draft schemes (e.g., multi-head or recursive drafters) can adopt the same prefix/suffix split and gain similar latency improvements.
  • The draft-model training recipe (longer training, dense layers, online distillation) raises TPC enough that speculation length 3 is competitive with longer speculation lengths, reducing per-step validation cost.
  • Disaggregated serving architectures can hide prefill/decode communication behind validation kernels, improving time-to-first-token and time-to-inter-token latency without changing model weights.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper leaves implicit is an online/adaptive tree dispatcher: the current tree selection is based on precomputed static trees keyed to batch size, but a policy that also conditions on sequence length, remaining output length, or the draft model's recent acceptance rate could squeeze out additional gains at mixed traffic.
  • The training recipe's reliance on only hidden states and logits suggests it should transfer to non-Llama transformers, but the paper only demonstrates Llama; applying it to other base models would clarify whether the TPC gains are architecture-specific.
  • The INT4 draft quantization result (nearly unchanged TPC, lower drafting latency) points toward a broader design space: the draft model can be compressed and simplified as aggressively as acceptance rate allows, because the target model guarantees distribution-preserving sampling.
  • The claimed 10% over the best prior method is with respect to a chain-based baseline; a head-to-head against an equally optimized tree-based engine at batch size one would be the most direct external check.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper reports a set of training and inference engineering optimizations for EAGLE-based speculative decoding applied to Llama 3 and Llama 4 models in Meta's production environment. Training changes include online distillation with a weighted cross-entropy + smooth-L1 loss, longer training (48k iterations), and a 3-layer dense draft model. Inference changes include a two-pass tree attention implementation in xFormers, multi-round speculative sampling optimizations (torch.compile with a dynamic batch dimension, tensor-parallel sampling with RNG synchronization, greedy draft decoding), latency-hiding restructuring of disaggregated prefill/decode, a tree dispatcher, draft KV-cache alignment, CUDA graphs, paged/persistent KV-cache handling for tree attention, draft FFN quantization, and guided decoding support. The main empirical claims are: (1) a 10--30% TTIT improvement over vLLM at batch size 1 for four Llama models; (2) a new state-of-the-art latency for Llama4 Maverick of about 4 ms/token on 8 H100s, 10% faster than 'the previously best known method'; and (3) a 1.4--2.0x speed-up for EAGLE-based speculative decoding at production-scale batch sizes.

Significance. If the claims are substantiated, this is a useful systems contribution. The paper documents several non-obvious engineering details that others can adopt: the split prefix/suffix tree attention, dynamic-batch torch.compile to avoid recompilation spikes, tensor-parallel sampling with synchronized RNG slices, paged-KV compatibility with tree attention, and the draft/base KV synchronization policy. The xFormers tree attention code is pointed to by URL, which is a concrete artifact. However, the empirical validation currently does not support the strong 'state-of-the-art' and '10% faster than previously best known' assertions, because the baseline is not named, is compared mainly in a chain-versus-tree configuration, and the paper intentionally omits the external baseline in the large-batch regime. The absence of error bars and the baseline ambiguity are load-bearing given the modest claimed margins. The paper is an engineering report rather than a new algorithmic derivation; its value depends on whether the benchmarking can be made rigorous enough to support the headline claims.

major comments (4)
  1. [Abstract; Figure 1a; Figure 1b] The headline '10% faster than the previously best known method' is not supported as stated. The 'previously best known method' is never named. The only external comparison shown, Figure 1a, compares 'Ours (Tree)' against 'vLLM (Chain)', so it confounds the proposed system with the use of tree attention (credited to Miao et al.). Figure 1b explicitly drops the vLLM comparison at larger batch sizes, citing 'significant gaps'. A controlled comparison against a strong tree-based open-source stack (e.g., SGLang or vLLM with EAGLE3/tree attention) on identical hardware, context length, and tree configurations is needed before claiming a new state of the art. Without this, the reported 10% margin could reflect baseline choice rather than the proposed optimizations.
  2. [Section 3.5; Figure 9] The '1.4x--2.0x speed-up at production scale' claim is not precisely defined. Figure 9 says the speed-up is measured 'relative to the baseline performance', but the baseline is not specified. It is not clear whether the baseline is non-speculative decoding in the same internal engine, a speculative-decoding baseline in vLLM, or something else. The batch sizes, sequence lengths, GPU counts, tensor-parallel settings, and benchmark prompts are not reported in enough detail to reproduce the curves. Without absolute TTIT values and a precise baseline definition, the production-scale speed-up claim is unverifiable.
  3. [Table 1; Section 2.4] The statement that 'with the proposed changes, EAGLE achieves similar or better TPC than EAGLE3' is not supported for all listed configurations. For Llama3.1-8B, EAGLE3 reaches TPC 3.32 at speculation length 5 and 3.57 at length 7, while the proposed EAGLE reaches 2.78 at length 3; for Llama3.3-70B, EAGLE3 reaches 3.03 and 3.20 at lengths 5 and 7, exceeding the proposed EAGLE's 2.94 at length 3. The conclusion holds only at the specific length-3 configuration shown, and the paper does not explain why length 3 is the fair point of comparison. Since TPC alone does not determine end-to-end latency, the paper should either benchmark EAGLE3 end-to-end at matched speculation lengths and tree structures, or explicitly rephrase the claim to acknowledge the configuration dependence.
  4. [Figures 1, 8, 9; Table 1] All reported TTIT, TPC, and speed-up numbers are single point estimates with no error bars, no variance measures, and no statement about the number of repeated runs or request-level variability. The claimed improvements are small enough (e.g., 1.10x in Figure 1a for Llama4 Maverick) that run-to-run noise on shared H100 hardware could change the qualitative conclusion. For a paper whose central contribution is empirical, at least standard deviation across multiple independent runs, the number of benchmark requests, and a description of the traffic/load conditions for Figure 1c should be reported.
minor comments (6)
  1. [Section 2.1, Eq. (1)] The notation in Eq. (1) and surrounding text is inconsistent: logits are introduced as l^b and l^d, but later the subscripts are dropped and 'l1_loss(h^b_{2:n}, h^d_{2:n})' and 'ce_loss(l^b_{2:n}, l^d_{2:n})' are used without defining the slicing. Please align notation.
  2. [Table 1] The third row is labeled 'Llama3.1-8B w/ EAGLE' but the values 2.12/2.24/2.27 are presumably for Llama3.3-70B (given the next row is EAGLE3 for Llama3.3-70B). Correct the model label.
  3. [Figure 8 caption] Typo: 'Prunned' should be 'Pruned' in both subfigures and the legend.
  4. [Section 3.3] 'rewounded' should be 'rewound' in the description of KV-cache and hidden-state bookkeeping.
  5. [Throughout] The paper reports results on 8 NVIDIA H100 GPUs but does not state software versions (CUDA, PyTorch, vLLM commit, xFormers commit) or the exact prompts/request mix used for Figure 1c. This limits reproducibility even for the publicly available xFormers component.
  6. [Contributors/footnote] The footnote refers to a 'Contributions section', but the manuscript contains a 'Contributors' section instead. Update the cross-reference.

Circularity Check

0 steps flagged

No circularity: empirical engineering report; SOTA claim rests on external baseline choice, not on fitting or self-citation.

full rationale

This is an empirical systems/engineering report rather than a derivation. The training objective (Eq. 1) is a weighted sum of a smooth-L1 hidden-state loss and a cross-entropy logit loss, with hand-set coefficients (lambda_CE=0.1, lambda_L1=1.0); these coefficients do not encode TPC, TTIT, or end-to-end speed-up targets. Draft-model quality is independently measured via TPC on MT-Bench and an internal benchmark, and the inference section describes concrete engineering optimizations (tree attention via merge_attentions, PyTorch-2 compilation, TP-parallelized multi-round speculative sampling, CUDA graphs, paged KV adaptations, latency-hiding restructurings) whose effects are then measured. No equation in the paper makes a reported 'prediction' identically equal to a fitted input, and no parameter is fitted to the headline latency numbers. The paper's comparisons to EAGLE/EAGLE3/vLLM are external baselines, not self-citations; the only institutional self-reference (Meta's Llama 4 model card) is not load-bearing for any derivation. The main risks are external-validity concerns: the '10% faster than the previously best known method' abstract claim is supported by a batch-1 tree-vs-chain comparison against vLLM (Figure 1a), while Figure 1b explicitly omits vLLM at larger batch sizes 'because of significant gaps in TTIT'; also, Table 1 compares the authors' EAGLE at speculation length 3 with EAGLE3 at lengths 5/7, so 'similar or better TPC' is not apples-to-apples. These are benchmarking and baseline-selection concerns, not circularity: the reported numbers are measurements, not consequences of their own definitions.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

The paper introduces no new theoretical entities. It relies on standard components (EAGLE, tree attention, CuDA graphs, paged KV) and makes several structural assumptions about the validity of the base models, the loss function, and the benchmarks. The free parameters are engineering choices (loss weights, speculation length) that influence the reported numbers but are not fitted to the central claim. The most significant implicit assumption is that the comparison baseline (vLLM) is fairly configured and that the benchmarks generalize to production.

free parameters (2)
  • loss coefficients (λ_CE=0.1, λ_L1=1.0) = 0.1 and 1.0
    Weighted sum of cross-entropy and L1 losses in Eq. 1. Chosen by hand, not derived or fitted to the target latency result.
  • speculation length = 3 (for TPC evaluations and production default)
    Used in Section 2.4 and 3.4. Selected based on TPC/latency experiments; a design choice that affects the reported speedups.
axioms (5)
  • domain assumption EAGLE speculative decoding and tree attention are valid and correctly implemented as described in prior work
    Section 1 and 3.1 rely on the correctness of EAGLE (Li et al. 2024) and tree attention (Miao et al. 2024). The paper does not re-derive these methods.
  • domain assumption The base models (Llama 3.1, Llama 3.3, Llama 4 Scout, Llama 4 Maverick) expose hidden states and logits as required for the draft model training
    Section 2.1 assumes access to the base model's pre-final-layer hidden states and logits. This is a property of the chosen models and framework.
  • domain assumption The loss function L = λ_CE L_CE + λ_L1 L_L1 is a suitable training objective for draft models
    Section 2.1 defines the loss without a theoretical proof that it maximizes TPC. Its effectiveness is only shown empirically on benchmarks.
  • domain assumption MT-Bench and the private internal benchmark are representative of production workloads for measuring TPC and latency
    Section 2.2 and 3.5 use these benchmarks to justify design choices and report the main numbers. They are not necessarily representative of all production traffic.
  • domain assumption The hardware (NVIDIA H100 GPUs), software stack (xFormers, PyTorch 2, internal engine), and measurement methodology behave as described
    The paper provides no independent verification or reproduction of its environment; the speedup numbers depend on this unstated infrastructure.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Speculative Decoding for Llama at Scale: Challenges and Solutions." pith.science (2026). https://pith.science/paper/4PX7HNEC

@misc{pith2026250808192,
  author       = {Pith},
  title        = {Pith review of: Efficient Speculative Decoding for Llama at Scale: Challenges and Solutions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4PX7HNEC}},
  note         = {Machine review of arXiv:2508.08192}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Speculative decoding is a standard method for accelerating the inference speed of large language models. However, scaling it for production environments poses several engineering challenges, including efficiently implementing different operations (e.g., tree attention and multi-round speculative decoding) on GPU. In this paper, we detail the training and inference optimization techniques that we have implemented to enable EAGLE-based speculative decoding at a production scale for Llama models. With these changes, we achieve a new state-of-the-art inference latency for Llama models. For example, Llama4 Maverick decodes at a speed of about 4 ms per token (with a batch size of one) on 8 NVIDIA H100 GPUs, which is 10% faster than the previously best known method. Furthermore, for EAGLE-based speculative decoding, our optimizations enable us to achieve a speed-up for large batch sizes between 1.4x and 2.0x at production scale.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Test-Time Speculation

    cs.CL 2026-05 unverdicted novelty 7.0

    Test-Time Speculation adapts draft models online via target-model verifications to sustain high acceptance lengths during long LLM generations.

  2. Test-Time Speculation

    cs.CL 2026-05 unverdicted novelty 6.0

    TTS adapts speculator models online via target model verifications to improve acceptance lengths by up to 72% over prior methods, with gains increasing for longer generations.

  3. Speculative Decoding with a Speculative Vocabulary

    cs.CL 2026-02 conditional novelty 6.0

    SpecVocab dynamically selects a per-step candidate vocabulary for the speculative-decoding draft model, beating static 32K-token vocabularies on acceptance length and throughput (up to +8.1% vs EAGLE-3).

  4. HiSpec: Hierarchical Speculative Decoding for LLMs

    cs.CL 2025-10 conditional novelty 6.0

    HiSpec uses early-exit layers to verify draft tokens midway through the model, reporting 1.28×-2.01× faster decoding over baseline speculative decoding, but the accuracy claim is not empirically tested.

Reference graph

Works this paper leans on

19 extracted references · 3 canonical work pages · cited by 3 Pith papers

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Pith/arXiv arXiv

  2. [6]

    The curious case of neural text degeneration

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30,

    work page 2020

  3. [8]

    Hydragen: High-throughput llm inference with shared prefixes.arXiv preprint arXiv:2402.05099,

    Jordan Juravsky, Bradley Brown, Ryan Ehrlich, Daniel Y Fu, Christopher Ré, and Azalia Mirhoseini. Hydragen: High-throughput llm inference with shared prefixes.arXiv preprint arXiv:2402.05099,

    Pith/arXiv arXiv

  4. [9]

    Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980,

    Diederik P Kingma. Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980,

    Pith/arXiv arXiv

  5. [10]

    Eagle: Speculative sampling requires rethinking feature uncertainty.arXiv preprint arXiv:2401.15077,

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty.arXiv preprint arXiv:2401.15077,

    Pith/arXiv arXiv

  6. [11]

    Eagle-3: Scaling up inference acceleration of large language models via training-time test.arXiv preprint arXiv:2503.01840,

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-3: Scaling up inference acceleration of large language models via training-time test.arXiv preprint arXiv:2503.01840,

    Pith/arXiv arXiv

  7. [12]

    https://arxiv.org/abs/2406.14066. Meta. LLaMA 4: Multimodal Intelligence.https://ai.meta.com/blog/llama-4-multimodal-intelligence/,

    Pith/arXiv arXiv

  8. [14]

    14 Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu

    Figures are referred from this version of the paper:https://arxiv.org/abs/2305.09781. 14 Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: A kvcache-centric disaggregated architecture for llm serving.URL https://arxiv. org/abs/2407.00079,

    Pith/arXiv arXiv

  9. [15]

    Self-attention does not needo(n2)memory.arXiv preprint arXiv:2112.05682,

    Markus N Rabe and Charles Staats. Self-attention does not needo(n2)memory.arXiv preprint arXiv:2112.05682,

    Pith/arXiv arXiv

  10. [16]

    Ranajoy Sadhukhan, Jian Chen, Zhuoming Chen, Vashisth Tiwari, Ruihang Lai, Jinyuan Shi, Ian En-Hsu Yen, Avner May, Tianqi Chen, and Beidi Chen

    GitHub repository. Ranajoy Sadhukhan, Jian Chen, Zhuoming Chen, Vashisth Tiwari, Ruihang Lai, Jinyuan Shi, Ian En-Hsu Yen, Avner May, Tianqi Chen, and Beidi Chen. Magicdec: Breaking the latency-throughput tradeoff for long context generation with speculative decoding, 2025.https://arxiv.org/abs/2408.11049. Project SGLang. Sglang: A language for sgl projec...

    Pith/arXiv arXiv 2025

  11. [17]

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean

    Accessed: 2025-06-03. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538,

    Pith/arXiv arXiv 2025

  12. [19]

    Efficient guided generation for llms.arXiv preprint arXiv:2307.09702,

    Brandon T Willard and Rémi Louf. Efficient guided generation for llms.arXiv preprint arXiv:2307.09702,

    Pith/arXiv arXiv

  13. [2017]

    The synergy of speculative decoding and batching in serving large language models, 2023.https://arxiv.org/abs/2310.18813

    Qidong Su, Christina Giannoula, and Gennady Pekhimenko. The synergy of speculative decoding and batching in serving large language models, 2023.https://arxiv.org/abs/2310.18813. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of ...

    Pith/arXiv arXiv 2023

  14. [2020]

    Wonseok Jeon, Mukul Gagrani, Raghavv Goel, Junyoung Park, Mingu Lee, and Christopher Lott

    OpenReview.net, 2020.https://openreview.net/forum?id=rygGQyrFvH. Wonseok Jeon, Mukul Gagrani, Raghavv Goel, Junyoung Park, Mingu Lee, and Christopher Lott. Recursive speculative decoding: Accelerating llm inference via sampling without replacement.arXiv preprint arXiv:2402.14160,

    Pith/arXiv arXiv 2020

  15. [2021]

    Medusa: Simple llm inference acceleration framework with multiple decoding heads.arXiv preprint arXiv:2401.10774,

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple llm inference acceleration framework with multiple decoding heads.arXiv preprint arXiv:2401.10774,

    Pith/arXiv arXiv

  16. [2022]

    Flash-decoding for long-context inference.https: //crfm.stanford.edu/2023/10/12/flashdecoding.html, October 12

    Tri Dao, Daniel Haziza, Francisco Massa, and Grigory Sizov. Flash-decoding for long-context inference.https: //crfm.stanford.edu/2023/10/12/flashdecoding.html, October 12

    work page 2023

  17. [2023]

    Aaron Grattafiori et al

    Stanford University. Aaron Grattafiori et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Pith/arXiv arXiv

  18. [2024]

    Accelerating large language model decoding with speculative sampling.arXiv preprint arXiv:2302.01318,

    CharlieChen, SebastianBorgeaud, GeoffreyIrving, Jean-BaptisteLespiau, LaurentSifre, andJohnJumper. Accelerating large language model decoding with speculative sampling.arXiv preprint arXiv:2302.01318,

    Pith/arXiv arXiv

  19. [2025]

    Xupeng Miao et al

    Ac- cessed: 2025-06-05. Xupeng Miao et al. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. InProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, pages 932–949,

    work page 2025

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.