Pith. sign in

REVIEW 4 major objections 5 minor 4 cited by

SpecVocab claims dynamic per-step top-k vocabulary subsets beat fixed 32K subsets, raising throughput by up to 8.1% while keeping outputs identical.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 23:23 UTC pith:6ME6MMIY

load-bearing objection A genuinely new per-step vocabulary-selection mechanism for speculative decoding, with modest but real gains over EAGLE-3 on Spec-Bench; the missing recall analysis and benchmark-confinement are real caveats, but the paper deserves a serious referee. the 4 major comments →

arxiv 2602.13836 v2 pith:6ME6MMIY submitted 2026-02-14 cs.CL

Speculative Decoding with a Speculative Vocabulary

Miles Williams , Young D. Kwon , Rui Li , Alexandros Kouris , Stylianos I. Venieris This is my paper
classification cs.CL
keywords speculative decodingvocabulary speculationlow-rank logits approximationtop-k vocabulary subsetacceptance lengthinference accelerationdraft modelrejection sampling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that speculative decoding should not only guess the next tokens but also guess which part of the vocabulary is relevant at each step. Instead of committing to a fixed, frequency-ranked subset of the target model's vocabulary, SpecVocab uses a cheap low-rank approximation of the full vocabulary logits to pick a context-relevant top-k (typically 2,048 tokens) and then computes exact logits only over those. The authors claim this consistently beats the state-of-the-art static-vocabulary method, EAGLE-3, on acceptance length across all evaluated task types, and that the longer accepted guesses translate into real throughput gains of up to 8.1%. Because verification still uses the target model's true distribution, the method preserves identical outputs while cutting the draft model's vocabulary computation drastically.

Core claim

On the paper's own terms, the discovery is that vocabulary speculation—predicting a small, context-dependent subset of the vocabulary at each decoding step—strikes a better accuracy–speed trade-off than any fixed reduced vocabulary. SpecVocab trains a low-rank vocabulary ranking module, takes its top-k indices, and computes the exact draft distribution over that subset. Across four models from two families and six task categories, this yields higher acceptance length than EAGLE-3, FR-Spec, and VocabTrim, and the resulting throughput improvement reaches 8.1% over EAGLE-3 on OLMo 2 7B. The paper also reports that post-training vocabulary pruning (VocabTrim) outperforms EAGLE-3's pre-training p

What carries the argument

The central object is the vocabulary speculation module: a learned low-rank projection (W_down) that maps the draft model's hidden state to a small intermediate dimension, followed by a learned vocabulary matrix (W_vocab) that yields approximate logits over the entire vocabulary. The top-k indices of those approximate logits form the candidate subset, on which exact logits are computed with an indexed LM head. This reduces the draft's vocabulary computation from O(|V|·d) to O(|V|·d' + k·d) while keeping every vocabulary token reachable at every step.

Load-bearing premise

The load-bearing premise is that the top-k indices of the low-rank approximate logits contain the token the target model would actually sample often enough that longer draft sequences are accepted; this recall is never explicitly measured in the paper.

What would settle it

Compute a recall curve: for a diverse set of texts (including non-English, code, and rare-token-heavy domains), sample tokens from the target model and measure how often the sampled token lies in the top-k set of SpecVocab's approximate ranking for k = 2048. If recall is substantially below the level needed to sustain the reported acceptance lengths (e.g., below ~95%), the claimed end-to-end speedup would disappear in those settings.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If SpecVocab's central claim holds, draft models can evaluate exact logits on roughly 1.4–2.0% of the vocabulary without losing acceptance length, lowering the dominant drafting-time bottleneck for large-vocabulary LMs.
  • Higher acceptance length directly translates to system throughput gains across all four tested models, with the largest improvements on larger models, suggesting the benefit scales with model and vocabulary size.
  • Post-training vocabulary pruning is shown to beat pre-training pruning, implying that training a draft model on a fixed small vocabulary can unnecessarily degrade draft accuracy—a finding that should influence how reduced-vocab draft models are trained.
  • The custom fused kernel, which reads each selected embedding from memory only once, speeds up the indexed LM head by 3–5×, making the dynamic-subset approach practically deployable in inference engines.
  • Because verification uses the target model's own distribution, SpecVocab produces outputs identical to standard speculative decoding, so the speedup comes with no quality trade-off.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper does not directly measure the recall of its top-k approximations—the probability that the target model's next sampled token falls inside the chosen subset; a low recall on rare-token-heavy or non-English text would erode the claimed gains, so reporting recall curves would strengthen the case.
  • The method's advantage likely grows as vocabularies expand beyond 152K tokens, because fixed subsets become sparser relative to the full vocabulary while a dynamic subset can still target the relevant region; this is a testable prediction with larger models.
  • The regularization effect of the joint loss (λ) suggests that forcing the draft model's hidden states to be compressible for the vocabulary speculator actually improves drafting—an interesting interaction that could be studied in other distillation settings.
  • Adapting k per decoding step (e.g., reading the uncertainty of the approximate ranking) could yield further throughput gains beyond the fixed-k configuration explored here.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SpecVocab, a speculative-decoding draft model that replaces the fixed reduced vocabulary used by EAGLE-3, FR-Spec, and VocabTrim with a per-step dynamically selected vocabulary subset. The subset is obtained by computing approximate top-k logits through a low-rank projection of the draft model's hidden state, then running the exact LM head only on those k indices. The method is trained jointly with an auxiliary loss that distills the target model's full-vocabulary distribution into the low-rank ranking module. Experiments cover four models (Qwen3 4B/8B, OLMo 2 1B/7B) on six Spec-Bench categories over five seeds, implemented in SGLang. The paper reports higher acceptance length than EAGLE-3 and up to 8.1% higher average throughput, plus a custom fused kernel and a complexity analysis.

Significance. If the empirical claims hold, SpecVocab is a simple and useful contribution to speculative decoding: it shows that a context-dependent subset of roughly 2K tokens can outperform a static 32K subset in both draft accuracy and end-to-end throughput, while preserving identical target-model outputs via rejection sampling. The paper also provides reproducible-style empirical work: all baselines are re-implemented in the same framework, training details are reported, a custom kernel is benchmarked, and results are averaged over five seeds. The complexity argument in Table 4 is a clear and useful comparison. The main value is empirical rather than theoretical; the method's advantage rests on the quality of the low-rank vocabulary ranking, which is not directly measured.

major comments (4)
  1. [§5, Tables 1, 8–10] The acceptance-length tables report averages over five seeds but give no standard deviation, confidence interval, or per-seed results. Several headline differences are small on an absolute scale (e.g. Table 1: 4.78 vs 5.01; Table 8: 4.31 vs 4.53; Table 10: 4.73 vs 5.01). Since the central claim is that SpecVocab 'consistently outperforms' EAGLE-3, the absence of dispersion measures makes it impossible to assess whether these differences are stable across seeds. Report acceptance-length variance or per-seed values, or provide a statistical test.
  2. [§4 Baselines; §5 Tables 2, 11–13] The 'EAGLE-3' comparison is ambiguous and load-bearing. The official/third-party EAGLE-3 checkpoint appears only in the Qwen3 8B tables; for OLMo 2 1B/7B and Qwen3 4B the only EAGLE-3 baseline is the authors' own reproduction. The headline 8.1% throughput improvement for OLMo 2 7B is 204.7 vs 189.3 tokens/s, i.e. against the reproduction, not the official model. To support the broad claim 'SpecVocab consistently outperforms EAGLE-3', provide official EAGLE-3 numbers for all four models, or explicitly and consistently qualify every such claim as being relative to the authors' reproduction.
  3. [§6, Figure 4 and Table 3; §4 Evaluation] The hyperparameters k, d'/d, and λ appear to be selected by inspecting performance curves on the same Spec-Bench tasks that are later reported as the final results. Figure 4 and Table 3 are used to justify k=2048, d'/d=1/8 (or 1/16), and λ=0.1, but no held-out validation split is described. Because Spec-Bench is the only evaluation benchmark, tuning on it may inflate the reported gains. Describe the selection procedure, use a validation split, or report sensitivity of the final results to the chosen operating points.
  4. [§3.3, Steps 1–2; §6 Analysis] The core mechanism assumes that the top-k indices of the low-rank approximate logits s_t contain the tokens sampled by the target model. The paper never reports recall@k for this ranking; Figure 4 shows acceptance length and throughput as functions of k, but both metrics conflate ranking recall with the draft model's overall quality. Without a recall measurement, it is difficult to know whether the dynamic vocabulary is the bottleneck, and the discussion of failure modes outside Spec-Bench is necessarily speculative. Add a recall@k analysis or at least a direct comparison of the approximate ranking against the target model's full-vocabulary distribution.
minor comments (5)
  1. [§3.3] The symbol k is used both for the number of candidate tokens and for the index set k_t = top-k(s_t, k). Rename the index set (e.g. I_t) to avoid confusion.
  2. [Tables 1, 8–10] Several table cells are missing spaces between numbers (e.g. '3.804.524.52' in Table 1), likely a rendering issue; ensure the final PDF is correctly formatted.
  3. [§4 Baselines] The baseline list distinguishes 'EAGLE-3' from 'EAGLE-3 (Our Reproduction)', but the main text sometimes refers simply to 'EAGLE-3'. Add a sentence clarifying which of the two is meant in each part of Section 5 and Section 6.
  4. [§2 Related Work] The related-work discussion is clear but could mention that other dynamic-subset or clustering-based vocabulary methods exist, if any; otherwise the framing 'we argue for vocabulary speculation as an alternative' is slightly overclaimed.
  5. [Limitations] The limitations section appropriately concedes the English/German-only evaluation. It would be helpful to state explicitly that the measured gains may not transfer to high-entropy, rare-token, or morphologically non-Indo-European settings rather than only listing it as future work.

Circularity Check

0 steps flagged

No circularity: SpecVocab's gains are measured on an external benchmark; hyperparameter selection and limited language coverage are generalization concerns, not derivation-by-construction.

full rationale

SpecVocab's central claim (higher acceptance length and throughput than EAGLE-3) is an empirical result measured against the external Spec-Bench benchmark, not a quantity constructed by the method's equations. The derivation chain is: (1) train W_down/W_vocab by distillation loss L = L_TTT(p,q) + λ L_TTT(p,q_aux) over the full vocabulary; (2) at inference form K_t = top-k(W_vocab W_down h_t); (3) compute exact logits over K_t and verify with the target model. Each stage is implemented and benchmarked; acceptance length and throughput are reported with five seeds and compared to reproduced baselines. No step reduces to its input by definition: the low-rank ranking is a learned approximator, not a restatement of the reported metric, and no uniqueness theorem or self-citation is invoked to force the design. The paper's own Limitations section concedes that Spec-Bench is English/West-Germanic only, and the choice of k, d', and λ is made on the same benchmark used for the final tables (Figure 4, Table 3); these are external-validity and selection concerns, not circularity, because the reported numbers are still measured rather than fitted as predictions. No load-bearing self-citation or renaming is present. Score 0.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

Three hyperparameters (k, d', λ) are selected on the evaluation benchmark rather than a held-out set. No new physical or mathematical entities are posited. The top-k recall assumption is the key domain assumption on which the method's usefulness rests.

free parameters (3)
  • k (candidate vocabulary size) = 2048 for most models; 512 for OLMo 2 1B
    Selected from Figure 4 as the value maximizing throughput on Spec-Bench; fixed per model in the main evaluation.
  • d' (intermediate dimensionality ratio) = d'/d = 1/16 or 1/8 depending on model
    Selected from Figure 4; Qwen3 models use 1/16, OLMo 2 7B uses 1/8 (per Section 6); affects the accuracy/cost trade-off.
  • λ (auxiliary loss weight) = 0.1
    Chosen from a sweep on Qwen3 8B (Table 3); used for all models.
axioms (3)
  • domain assumption The top-k tokens of the low-rank approximate logits s_t = W_vocab W_down h_t contain the token sampled from the target distribution with sufficient recall.
    Foundation of SpecVocab (Section 3.3, Step 2). If recall is low, acceptance length collapses. The paper validates this empirically on Spec-Bench but provides no recall statistics.
  • domain assumption The draft hidden state h_t is informative for predicting the target's next-token vocabulary subset.
    SpecVocab uses only h_t to rank candidates (Section 3.3, Step 1).
  • standard math Restricted-support draft distributions preserve the exact output distribution through rejection sampling.
    Relies on the standard speculative-decoding guarantee (Leviathan et al., 2023); the paper does not re-derive it.

pith-pipeline@v1.3.0-alltime-deepseek · 15408 in / 15099 out tokens · 124857 ms · 2026-08-02T23:23:25.810589+00:00 · methodology

0 comments
read the original abstract

Speculative decoding has rapidly emerged as a leading approach for accelerating language model (LM) inference, as it offers substantial speedups while yielding identical outputs. This relies upon a small draft model, tasked with predicting the outputs of the target model. State-of-the-art speculative decoding methods use a draft model comprising a single decoder layer and output embedding matrix, with the latter dominating drafting time for the latest LMs. Recent work has sought to address this output distribution bottleneck by reducing the vocabulary of the draft model. While this can improve throughput, it compromises speculation effectiveness when the target token is out-of-vocabulary. In this paper, we argue for vocabulary speculation as an alternative to a reduced vocabulary. We propose SpecVocab, an efficient and effective method that selects a vocabulary subset per decoding step. Across a variety of tasks, we show that SpecVocab can achieve a higher acceptance length than state-of-the-art speculative decoding method, EAGLE-3. Notably, this yields up to an 8.1% increase in average throughput over EAGLE-3.

Figures

Figures reproduced from arXiv: 2602.13836 by Alexandros Kouris, Miles Williams, Rui Li, Stylianos I. Venieris, Young D. Kwon.

Figure 1
Figure 1. Figure 1: Vocabulary speculation accelerates specula￾tive decoding by computing the output distribution for only a contextually relevant subset of the vocabulary. drafting (Miao et al., 2024; Cheng et al., 2024; Wertheimer et al., 2024; Li et al., 2024b,a, 2025b; Zhang et al., 2025). However, Zhao et al. (2025) re￾cently identified that in the widely adopted EAGLE speculative decoding framework (Li et al., 2024a), t… view at source ↗
Figure 2
Figure 2. Figure 2: Outline of the draft model architectures for speculative decoding. EAGLE-2 forms predictions over the [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The sequence of memory access operations [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The acceptance length and throughput when varying both the number of candidate tokens ( [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Microbenchmark results for our custom fused [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Trees from Marginals: Autoregressive drafting with factorized priors

    cs.LG 2026-07 accept novelty 7.0

    Weaver restores conditional dependencies on top-K factorized marginals to build high-acceptance draft trees, plus a fused GDN tree-verify kernel, yielding 4.37× AR speedup and 24.7% over DFlash.

  2. Trees from Marginals: Autoregressive drafting with factorized priors

    cs.LG 2026-07 accept novelty 7.0

    Weaver builds conditional proposal trees from a factorized drafter’s top-K marginals and a rollback-free GDN tree-verify kernel, yielding 4.37× speedup over AR decoding.

  3. SlimSpec: Low-Rank Draft LM-Head for Accelerated Speculative Decoding

    cs.LG 2026-05 unverdicted novelty 7.0

    SlimSpec replaces the standard LM-head in draft models with a low-rank version to deliver 4-5x faster speculative decoding while preserving full vocabulary and competitive acceptance rates.

  4. Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decoding

    cs.CL 2026-05 unverdicted novelty 6.0

    Domino decouples causal dependency modeling from autoregressive draft execution via a parallel backbone plus lightweight causal head and a base-anchored training curriculum, reporting up to 5.49x speedup.

Reference graph

Works this paper leans on

43 extracted references · 1 canonical work pages · cited by 3 Pith papers

  1. [1]

    Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, and 106 others

    Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, and 106 others. 2025. https://arxiv.org/abs/2508.10925 gpt-oss-120b & gpt-oss-20b model card . Prepri...

    Pith/arXiv arXiv 2025

  2. [2]

    Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Ragan-Kelley, and William Brandon. 2024. https://openreview.net/forum?id=FbhjirzvJG Hydra: Sequentially-dependent draft heads for medusa decoding . In First Conference on Language Modeling

    2024

  3. [3]

    Ond r ej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Ale s Tamchyna. 2014. https://doi.org/10.3115/v1/W14-3302 Findings of the 2014 workshop on statistical machine translation . In Proceedings of the Ninth Workshop on ...

    doi:10.3115/v1/w14-3302 2014

  4. [4]

    Lee, Deming Chen, and Tri Dao

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. 2024. https://proceedings.mlr.press/v235/cai24b.html Medusa: Simple LLM inference acceleration framework with multiple decoding heads . In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, ...

    2024

  5. [5]

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. https://arxiv.org/abs/2302.01318 Accelerating large language model decoding with speculative sampling . Preprint, arXiv:2302.01318

    Pith/arXiv arXiv 2023

  6. [6]

    Yunfei Cheng, Aonan Zhang, Xuanyu Zhang, Chong Wang, and Yi Wang. 2024. https://arxiv.org/abs/2403.09919 Recurrent drafter for fast speculative decoding in large language models . Preprint, arXiv:2403.09919

    Pith/arXiv arXiv 2024

  7. [7]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . Preprint, arXiv:2110.14168

    Pith/arXiv arXiv 2021

  8. [8]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long a...

    doi:10.18653/v1/n19-1423 2019

  9. [9]

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.183 Enhancing chat language models by scaling high-quality instructional conversations . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3029--3051, Singapore...

    doi:10.18653/v1/2023.emnlp-main.183 2023

  10. [10]

    Raghavv Goel, Sudhanshu Agrawal, Mukul Gagrani, Junyoung Park, Yifan Zao, He Zhang, Tian Liu, Yiping Yang, Xin Yuan, Jiuyan Lu, Chris Lott, and Mingu Lee. 2025. https://arxiv.org/abs/2506.22694 VocabTrim : Vocabulary pruning for efficient speculative decoding in LLM s . Preprint, arXiv:2506.22694

    arXiv 2025

  11. [11]

    Milan Gritta, Huiyin Xue, and Gerasimos Lampouras. 2025. https://doi.org/10.18653/v1/2025.findings-acl.1017 DR e SD : Dense retrieval for speculative decoding . In Findings of the Association for Computational Linguistics: ACL 2025, pages 19822--19832, Vienna, Austria. Association for Computational Linguistics

    doi:10.18653/v1/2025.findings-acl.1017 2025

  12. [12]

    Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. https://proceedings.neurips.cc/paper_files/paper/2015/file/afdec7005cc9f14302cd0474fd0f3c96-Paper.pdf Teaching machines to read and comprehend . In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc

    2015

  13. [13]

    Hongzhi Huang, Defa Zhu, Banggu Wu, Yutao Zeng, Ya Wang, Qiyang Min, and Zhou Xun. 2025. https://proceedings.mlr.press/v267/huang25bb.html Over-tokenized transformer: Vocabulary is generally worth scaling . In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 26261--26282. PMLR

    2025

  14. [14]

    Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, and 196 others. 2025. https://arxiv.org/abs/2503.19786 G...

    Pith/arXiv arXiv 2025

  15. [15]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. https://doi.org/10.1162/tacl_a_00276 Natural questions: A benchma...

    doi:10.1162/tacl_a_00276 2019

  16. [16]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. https://doi.org/10.1145/3600006.3613165 Efficient memory management for large language model serving with PagedAttention . In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP '23, page 611–626, New York, ...

    arXiv 2023

  17. [17]

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. https://proceedings.mlr.press/v202/leviathan23a.html Fast inference from transformers via speculative decoding . In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 19274--19286. PMLR

    2023

  18. [18]

    Shenggui Li, Yikai Zhu, Chao Wang, Fan Yin, Shuai Shi, Yubo Wang, Yi Zhang, Yingyi Huang, Haoshuai Zheng, and Yineng Zhang. 2025 a . https://github.com/sgl-project/specforge SpecForge : Train speculative decoding models effortlessly

    2025

  19. [19]

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024 a . https://doi.org/10.18653/v1/2024.emnlp-main.422 EAGLE -2: Faster inference of language models with dynamic draft trees . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7421--7432, Miami, Florida, USA. Association for Computational Linguistics

    doi:10.18653/v1/2024.emnlp-main.422 2024

  20. [20]

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024 b . https://proceedings.mlr.press/v235/li24bt.html EAGLE : Speculative sampling requires rethinking feature uncertainty . In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 28935--28948. PMLR

    2024

  21. [21]

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2025 b . https://arxiv.org/abs/2503.01840 EAGLE-3 : Scaling up inference acceleration of large language models via training-time test . Preprint, arXiv:2503.01840

    Pith/arXiv arXiv 2025

  22. [22]

    Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. 2024. https://doi.org/10.1145/3620666.3651335 SpecInfer : Accelerating large language model serving with tree-based speculative inference an...

    arXiv 2024

  23. [23]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

    2019

  24. [24]

    Liu, and Christopher D

    Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. https://doi.org/10.18653/v1/P17-1099 Get to the point: Summarization with pointer-generator networks . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073--1083, Vancouver, Canada. Association for Computational Linguistics

    doi:10.18653/v1/p17-1099 2017

  25. [25]

    Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. 2023. https://www.cerebras.net/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama SlimPajama: A 627B token cleaned and deduplicated version of RedPajama

    2023

  26. [26]

    Benjamin Frederick Spector and Christopher Re. 2023. https://openreview.net/forum?id=RKHF3VYjLK Accelerating LLM inference with staged speculative decoding . In Workshop on Efficient Systems for Foundation Models @ ICML2023

    2023

  27. [27]

    Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. 2018. https://proceedings.neurips.cc/paper_files/paper/2018/file/c4127b9194fe8562c64dc0f5bf2c93bc-Paper.pdf Blockwise parallel decoding for deep autoregressive models . In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc

    2018

  28. [28]

    Xin Sun, Tao Ge, Furu Wei, and Houfeng Wang. 2021. https://doi.org/10.18653/v1/2021.acl-long.462 Instantaneous grammatical error correction with shallow aggressive decoding . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long ...

    doi:10.18653/v1/2021.acl-long.462 2021

  29. [29]

    Sho Takase, Ryokan Ri, Shun Kiyono, and Takuya Kato. 2025. https://doi.org/10.18653/v1/2025.findings-acl.57 Large vocabulary size improves large language models . In Findings of the Association for Computational Linguistics: ACL 2025, pages 1015--1026, Vienna, Austria. Association for Computational Linguistics

    work page doi:10.18653/v1/2025.findings-acl.57 2025

  30. [30]

    Bangsheng Tang, Carl Chengyan Fu, Fei Kou, Grigory Sizov, Haoci Zhang, Jason Park, Jiawen Liu, Jie You, Qirui Yang, Sachin Mehta, Shengyong Cai, Xiaodong Wang, Xingyu Liu, Yunlu Li, Yanjun Zhou, Wei Wei, Zhiwei Zhao, Zixi Qi, Adolfo Victoria, and 19 others. 2025. https://arxiv.org/abs/2508.08192 Efficient speculative decoding for llama at scale: Challenge...

    Pith/arXiv arXiv 2025

  31. [31]

    Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muennighoff, Zhongwei Wan, Ping Luo, Min Lin, and Ngai Wong. 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/cf5a019ae9c11b4be88213ce3f85d85c-Paper-Conference.pdf Scaling laws with vocabulary: Larger models deserve larger vocabularies . In Advances in Neural Information Processing Systems, volume ...

    2024

  32. [32]

    Evan Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Allyson Ettinger, and 23 others. 2025. https://openreview.net/forum?id=2ezugTT9kU 2 OLM ...

    2025

  33. [33]

    Davis Wertheimer, Joshua Rosenkranz, Thomas Parnell, Sahil Suneja, Pavithra Ranganathan, Raghu Ganti, and Mudhakar Srivatsa. 2024. https://arxiv.org/abs/2404.19124 Accelerating production LLM s with combined token/embedding speculators . Preprint, arXiv:2404.19124

    Pith/arXiv arXiv 2024

  34. [34]

    Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.257 Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3909--3925, Singapore. Association for Computational ...

    doi:10.18653/v1/2023.findings-emnlp.257 2023

  35. [35]

    Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. 2024. https://doi.org/10.18653/v1/2024.findings-acl.456 Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding . In Findings of the Association for Computational Linguistics: ACL 2024, pages 7655--7671,...

    doi:10.18653/v1/2024.findings-acl.456 2024

  36. [36]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. https://arxiv.org/abs/2505.09388 Qwen3 technical report . Preprint, arXiv:2505.09388

    Pith/arXiv arXiv 2025

  37. [37]

    Lefan Zhang, Xiaodan Wang, Yanhua Huang, and Ruiwen Xu. 2025. https://openreview.net/forum?id=T9u56s7mbk Learning harmonized representations for speculative sampling . In The Thirteenth International Conference on Learning Representations

    2025

  38. [38]

    Weilin Zhao, Tengyu Pan, Xu Han, Yudi Zhang, Ao Sun, Yuxiang Huang, Kaihuo Zhang, Weilun Zhao, Yuxuan Li, Jie Zhou, Hao Zhou, Jianyong Wang, Zhiyuan Liu, and Maosong Sun. 2025. https://doi.org/10.18653/v1/2025.acl-long.198 FR -spec: Accelerating large-vocabulary language models via frequency-ranked speculative sampling . In Proceedings of the 63rd Annual ...

    doi:10.18653/v1/2025.acl-long.198 2025

  39. [39]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/91f18a1287b398d378ef22505bf41832-Paper-Datasets_and_Benchmarks.pdf Judging LLM-as-a-Judge with MT-Bench and Chatbot ...

    2023

  40. [40]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/724be4472168f31ba1c9ac630f15dec8-Paper-Conference.pdf SGLang : Efficient execution of structured language model prog...

    2024

  41. [41]

    George Kingsley Zipf. 1949. Human Behavior and the Principle of Least Effort: An Introduction to Human Ecology. Addison-Wesley, Cambridge, MA

    1949

  42. [42]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  43. [43]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...