Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

FastDraft: How to Train Your Draft

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FastDraft shows that a 50M-parameter draft model trained in under 24 hours can give speculative decoding up to 3x theoretical speedup and about 2x wall-clock speedup for Phi-3-mini.

desk verdict A genuinely useful training recipe for speculative decoding drafts, with real wall-clock validation, but the headline 3x speedup is an idealized metric that the paper's own measurements contradict. read the letter →

arxiv 2411.11055 v3 pith:KV237ZSZ submitted 2024-11-17 cs.CL

Ofir Zafrir , Igor Margulis , Dorin Shteyman , Shira Guskin , Guy Boudoukh This is my paper · ORCID
classification cs.CL
keywords speculativedecodingdraftmodelknowledgedistillationcontinuedpre-trainingPhi-3-miniLlama-3.1-8Bmemory-boundspeedupedgeinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a small draft model for speculative decoding can be trained from scratch and aligned to any target LLM in under 24 hours on a single server with 8 accelerators, removing the usual reliance on existing draft models that share the target's vocabulary. The proposed FastDraft recipe combines three stages: pre-training on 5 billion tokens of natural language, continued pre-training on a mixture of 5 billion code tokens and 2.5 billion text tokens, and fine-tuning on synthetic instruction-response data generated by the target model itself. For Phi-3-mini this yields a 50M-parameter draft that reaches up to 67% acceptance rate, up to 3x memory-bound speedup on code completion, and about 2x wall-clock speedup on code and 1.5x on natural language when measured on the paper's laptop-class target hardware. The same pipeline produces a 150M draft for Llama-3.1-8B with similar gains, supporting the claim that FastDraft can make speculative decoding practical on edge devices.

What carries the argument

The load-bearing object is the three-stage FastDraft pipeline: (1) language-model pre-training on 5B tokens of FineWeb natural language, (2) continued pre-training with a mix of 5B code tokens from The Stack v2 and 2.5B text tokens, and (3) sequence-level knowledge distillation, meaning fine-tuning the draft on synthetic instruction-response sequences generated by the target model with cross-entropy loss but without teacher logits. The supporting theoretical object is the memory-bound speedup estimate $\mathrm{MBSU} = \tau^\gamma / (\hat{c}\gamma + 1)$, where $\tau^\gamma$ is block efficiency and $\hat{c}$ is the draft/target parameter-count ratio; this formula converts acceptance rates into the headline 3x speedup.

What would settle it

Measure the actual per-token latency of the 50M draft and of Phi-3-mini on the same hardware across sequence lengths, and compute TPOT speedup from Eq. (2) using the measured latency ratio instead of the parameter-count ratio. If that number falls below the reported MBSU, the 3x headline is an artifact of the approximation; a direct reproduction should also verify the up-to-67% acceptance rate and the ~2x wall-clock speedup on the paper's benchmarks.

Watch

Extended reading notes

Core claim

The central claim is that a task-specific draft model for speculative decoding does not need to be borrowed, pruned, or distilled from an existing model family. FastDraft builds one from scratch: pre-train a compact Transformer on 5B tokens of FineWeb text, continue pre-training on 5B code tokens plus 2.5B text tokens, then fine-tune on synthetic instruction-response data sampled from the target model. For Phi-3-mini this yields a 50M draft with up to 67% acceptance rate and up to 3x MBSU; on the paper's laptop-class target hardware the measured wall-clock speedup is up to 2x on code and 1.5x on natural language. Applying the same recipe to Llama-3.1-8B produces a 150M draft with similar improvements, which the authors present as evidence that the pipeline generalizes across target models.

Load-bearing premise

The headline speedup rests on treating the ratio of parameter counts between draft and target as equal to their real per-token latency ratio in the memory-bound regime; if activation compute, implementation overhead, or memory access patterns break that equality, the theoretical 3x speedup overstates what users will see.

Editorial extensions

If this is right

  • Any LLM with a known vocabulary can get a custom draft in under 24 hours on one 8-accelerator server, removing the dependency on pre-existing draft models.
  • Laptop-class hardware can run the draft alongside a 3.8B target with about 2x code-completion speedup and 1.5x natural-language speedup, making local LLM inference more practical.
  • Sequence-level KD on target-generated data is the decisive alignment step; token-level KD with KL or TVD losses adds no consistent benefit in these experiments.
  • Continued pre-training on a code+text mixture beats both code-only continued pre-training and from-scratch mixed pre-training for both natural-language and code benchmarks.
  • Draft architecture choices matter: under a fixed parameter budget, shallow-wide drafts give a better acceptance-latency tradeoff than deep narrow ones on the evaluated hardware.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the 3x MBSU assumes the parameter-count ratio equals the real latency ratio, the practical ceiling is likely closer to the measured wall-clock 2x; deployers should benchmark actual latency ratios before trusting the theoretical number.
  • Editorial inference: the method's recipe is hardware-agnostic in spirit, but the optimal width/depth tradeoff will likely shift on GPUs or NPUs, so the same pipeline may need re-tuning on other platforms.
  • Editorial inference: since FastDraft drafts are standalone language models sharing only the vocabulary, they could be plugged into multi-sequence speculative decoding schemes or combined with self-speculative methods, though the paper does not test this.
  • Editorial inference: the English-only validation and the single-sequence speculation choice leave natural extensions: multilingual drafts could be tested with the same 24-hour budget, and multi-candidate speculation could raise acceptance further at higher compute cost.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces FastDraft, a method for training small, vocabulary-compatible draft models for speculative decoding. The pipeline consists of (1) pre-training a compact model on a 5B-token FineWeb sample, (2) continued pre-training on a mix of 5B code tokens from The Stack v2 and 2.5B text tokens, and (3) fine-tuning on synthetic instruction-response data generated by the target model. The authors demonstrate the approach by training a 50M-parameter draft for Phi-3-mini and a 150M draft for Llama-3.1-8B-Instruct, reporting acceptance rates, block efficiency, a hardware-agnostic 'memory-bound speedup' (MBSU) metric, and wall-clock speedups on an Intel Core Ultra 7 processor. The paper includes extensive ablations on pretraining data size, continued pretraining strategies, knowledge-distillation losses, and hardware-aware architectures, and claims that the draft can be trained in under 24 hours on 8 Intel Gaudi 2 accelerators.

Significance. If the results are correct, FastDraft offers a practical, low-cost training recipe for draft models, addressing the vocabulary-compatibility bottleneck in speculative decoding. The wall-clock validation on consumer hardware is a valuable contribution, as are the systematic ablations. The authors should be credited for measuring real end-to-end speedups rather than relying solely on a theoretical metric. However, the headline claim of 'up to 3x memory-bound speedup' is not supported by the measured 1.5-2x wall-clock speedups; the MBSU definition systematically overstates the achievable speedup, so the central result needs to be reinterpreted. The lack of error bars in the ablation tables also tempers the strength of the data-size and architecture recommendations.

major comments (3)
  1. [Section 4.3.1, Eq. (5); abstract and conclusion] The MBSU definition uses c-hat, 'the ratio of parameter counts between the draft and target models,' as a proxy for the latency ratio c = l_D/l_T. The paper's own wall-clock measurements on the Intel Core Ultra (Sections 5.6 and 6) contradict this proxy. Combining the reported up-to-2x wall-clock speedup on HumanEval with the greedy block efficiency tau = 1 + 0.663*3 = 2.99 gives c_real = (tau/S - 1)/gamma = 0.165, which is over 12 times larger than c-hat = 50M/3.8B = 0.013. Inserting c_real into Eq. (2) yields a speedup of about 2.0, not 3.0. The abstract and conclusion advertise 'up to 3x memory-bound speedup,' but this is an idealized estimate that does not correspond to user-visible speedup; the measured speedup is at most 2x. This is load-bearing because the 3x figure is a headline result. The authors should either re-estimate c from measured latencies or clearly state that MBSU is a theoretical upper bound that overstates the real speedup, with wall-clock numbers as the primary measurement.
  2. [Section 5.2, Tables 1 and 7] The ablation conclusions about pre-training dataset size rely on single point estimates with no error bars, multiple seeds, or statistical tests. For example, in Table 1, the 50M draft's CNN-DM acceptance rates at 2BT, 5BT, and 10BT are 0.323, 0.311, and 0.312, a variation of +/-0.006 that is within plausible run-to-run noise for such models. The conclusion that acceptance rate 'plateaus or decreases' with data growth, and the choice of 5BT as 'a promising middle-ground option,' are therefore not strongly supported. The authors should provide confidence intervals or multiple-run statistics, or temper the claims accordingly.
  3. [Section 6, 'Results & reproducibility'] The section is titled 'Results & reproducibility,' but no code, data, training configurations, or trained checkpoints are released, and the FastDraft evaluation framework is not made public. The acceptance rates, block efficiencies, and wall-clock numbers therefore cannot be independently reproduced or verified. Please provide the code, data, evaluation scripts, or at minimum detailed per-sample results and complete hyperparameters, so that the central empirical claims can be checked.
minor comments (6)
  1. [Appendix A] The word 'accelarators' appears; it should be 'accelerators.'
  2. [Appendix D] The opening sentence says 'Table 4 summarizes results ... using greedy sampling,' but Table 4 in the main text is for multinomial sampling; the greedy results are in Table 17.
  3. [Equation (2)] The inline typesetting of Eq. (2) is garbled and ambiguous; please typeset the formula with clear parentheses and a negative exponent, e.g., speedup = tau * (l_D/l_T * gamma + l_T^gamma/l_T^1)^-1.
  4. [Section 4.3.1] The definition of c-hat should specify whether embedding parameters are counted, since Table 14 explicitly excludes embeddings from the parameter budget and the parameter-count ratio is used to justify the 3x speedup claim.
  5. [Section 6] The sentence 'These models achieve a MBSU of ~2x for natural language tasks and ~3x for code completion tasks' does not state the block size (gamma=3 or 5) or the sampling method; please specify these conditions.
  6. [Section 8, Limitations] The limitations section is candid about single-sequence speculation and architectural homogeneity, but it does not mention that the reported 'up to 3x memory-bound speedup' is not reflected in the measured wall-clock speedups; this omission should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the training pipeline is evaluated with measured acceptance rates and independent wall-clock speedups; the MBSU proxy is a stated metric, not a prediction derived from its own output.

full rationale

The paper's central claims are empirical: a 50M-parameter draft is pre-trained and fine-tuned, and its acceptance rate and block efficiency are measured on held-out benchmarks (CNN-DM, TS, Dolly, HumanEval), not fitted or defined into existence. The acceptance-rate numbers in Tables 1-4 and 17 come from decoding the trained draft against the target, so they are external to the training procedure. The MBSU metric in Eq. (5) defines expected speedup as tau^gamma / (c_hat * gamma + 1), with c_hat set to the parameter-count ratio between draft and target. This is a stated modeling assumption for a memory-bound regime, not a fitted parameter, and it is not used to claim that the parameter-count ratio was measured from latency. Moreover, the paper independently reports measured wall-clock speedups on Intel Core Ultra in Sections 5.6 and 6 (average 1.5x on natural language and 2x on code), which ground the speedup claim outside the MBSU definition. Whether the parameter-count proxy overstates the user-visible speedup is a correctness or metric-validity concern, not circularity: no claimed result is equivalent to its input by construction, and the 'up to 3x' MBSU headline is explicitly labeled a memory-bound estimate rather than the wall-clock result. The paper's self-citations are limited to prior work, and the cited speculative-decoding framework (Leviathan et al.) is external and standard. No fitted input is renamed as a prediction, and no load-bearing argument reduces to a self-citation chain. Therefore the derivation chain is self-contained and non-circular.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claim relies on standard speculative decoding math and on three domain assumptions: parameter-count ratio as a latency proxy, latency independence of block size, and transferability of small-model pre-training. No free parameters are fitted and no new entities are invented.

assumptions (5)
  • standard math Speculative decoding acceptance/rejection procedure and its expected-speedup equations (Leviathan et al., 2023) are correct and applicable.
    Section 3 adopts Eqs. (1)-(2) from Leviathan et al. as background; these are unproved in this paper but standard.
  • domain assumption Target model latency for block size gamma is approximately equal to latency for block size 1 for the gamma values used (3 and 5).
    Stated in Section 3: the condition 'generally holds for most popular LLMs with sufficiently small gamma' is assumed for Phi-3-mini and Llama-3.1-8B.
  • domain assumption In the memory-bound regime, the ratio of draft-to-target latencies is approximated by the ratio of model parameter counts.
    Underlies MBSU in Eq. (5), Section 4.3.1; c-hat is defined as the ratio of parameter counts.
  • domain assumption Pre-training on 5B tokens of FineWeb and continued pre-training on Stack v2 code teaches a 50M model behaviors that transfer to the evaluation benchmarks.
    Empirical bet of Sections 5.1-5.3; no guarantee, tested only on two target models.
  • domain assumption Synthetic alignment data generated by the target model via Magpie-style prompting approximates real user prompts.
    Section 5.1; adoption of Xu et al. (2024); if the target model cannot generate realistic instructions, sequence-level KD may not align.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FastDraft: How to Train Your Draft." pith.science (2026). https://pith.science/paper/KV237ZSZ

@misc{pith2026241111055,
  author       = {Pith},
  title        = {Pith review of: FastDraft: How to Train Your Draft},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KV237ZSZ}},
  note         = {Machine review of arXiv:2411.11055}
}
abstract

Speculative Decoding has gained popularity as an effective technique for accelerating the auto-regressive inference process of Large Language Models. However, Speculative Decoding entirely relies on the availability of efficient draft models, which are often lacking for many existing language models due to a stringent constraint of vocabulary compatibility. In this work we introduce FastDraft, a novel and efficient approach for pre-training and aligning a draft model to any large language model by incorporating efficient pre-training, followed by fine-tuning over synthetic datasets generated by the target model. We demonstrate FastDraft by training two highly parameter efficient drafts for the popular Phi-3-mini and Llama-3.1-8B models. Using FastDraft, we were able to produce a draft model with approximately 10 billion tokens on a single server with 8 Intel$^\circledR$ Gaudi$^\circledR$ 2 accelerators in under 24 hours. Our results show that the draft model achieves impressive results in key metrics of acceptance rate, block efficiency and up to 3x memory bound speed up when evaluated on code completion and up to 2x in summarization, text completion and instruction tasks. We validate our theoretical findings through benchmarking on the latest Intel$^\circledR$ Core$^{\tiny \text{TM}}$ Ultra, achieving a wall-clock time speedup of up to 2x, indicating a significant reduction in runtime. Due to its high quality, FastDraft unlocks large language models inference on AI-PC and other edge-devices.

Figures

Figures reproduced from arXiv: 2411.11055 by the authors.

Figure 1
Figure 1. MBSU of 50M and 120M drafts pre-trained on 5BT FineWeb sample. AR increases with the amount of data which can be expected due to the nature of the benchmark of text completion versus instruction following and summarization. Overall, we obtain strong results across all dataset sizes, with the models trained on the 5BT dataset emerging as a promising middle￾ground option. When comparing the 50M draft to the 120M draft… view at source ↗
Figure 2
Figure 2. Block efficiency results for continued pre-training with code on tasks: CNN-DM, TS, Dolly and [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Latency of models obtained by varying either [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Estimated speedup on CNN-DM for models with constant parameter budget. L, H denote the number of layers and hidden dimension size a draft for Phi-3-mini. To further illustrate repro￾ducibility, we employ the same pipeline to produce a draft for Llama-3.1-8B-Instruct (D…
Figure 5
Figure 5. Figure 5: Estimated speedup on TS for models with constant parameter budget. L, H denote the number of layers and hidden dimension size H=256, L=32 H=384, L=12 H=448, L=8 H=512, L=6 H=576, L=4 H=640, L=3 H=768, L=2 Architecture 0.00 0.05 0.10 0.15 0.20 0.25 AR 0.50 0.75 1.00 1.2…
Figure 7
Figure 7. Figure 7: First and second token latencies of models obtained by varying either a hidden dimension or a number of [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Three lossless speculative decoding algorithms (SLEM, TLI, SLRS) remove the shared-vocabulary constraint, enabling any off-the-shelf model to serve as a drafter for LLM acceleration.

  2. Consultant Decoding: Yet Another Synergistic Mechanism

    cs.CL 2025-06 conditional novelty 4.0 of 10

    Consultant Decoding speeds up LLM generation by accepting draft tokens whose negative log-likelihood under the target model falls below a fixed threshold, reaching 2-3x speedups with comparable quality.

Reference graph

Works this paper leans on

43 extracted references · 7 canonical work pages · cited by 2 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

    arXiv 2024

  4. [4]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  5. [5]

    Viraat Aryabumi, Yixuan Su, Raymond Ma, Adrien Morisot, Ivan Zhang, Acyr Locatelli, Marzieh Fadaee, Ahmet \"U st \"u n, and Sara Hooker. 2024. To code, or not to code? exploring impact of code in pre-training. arXiv preprint arXiv:2408.10914

    arXiv 2024

  6. [6]

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. 2024. Medusa: Simple llm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774

    arXiv 2024

  7. [7]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    arXiv 2021

  8. [8]

    Ziyi Chen, Xiaocong Yang, Jiacheng Lin, Chenkai Sun, Jie Huang, and Kevin Chen-Chuan Chang. 2023. Cascade speculative drafting for even faster llm inference. arXiv preprint arXiv:2312.11462

    arXiv 2023

Show all 43 references
  1. [9]

    Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm Free dolly: Introducing the world's ...

    2023

  2. [10]

    Common crawl

    Common Crawl. Common crawl. https://commoncrawl.org/

  3. [11]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  4. [12]

    Ronen Eldan, Elad Liebman, Colin Cherry, and Yinhan Liu. 2023. https://arxiv.org/abs/2305.07759 Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.07759

    2023 arXiv

  5. [13]

    Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel decoding of conditional masked language models. arXiv preprint arXiv:1904.09324

    2019 arXiv

  6. [14]

    Raghavv Goel, Mukul Gagrani, Wonseok Jeon, Junyoung Park, Mingu Lee, and Christopher Lott. 2024. Direct alignment of draft model for speculative decoding with chat-fine-tuned llms. arXiv preprint arXiv:2403.00858

    2024 arXiv

  7. [15]

    John L Hennessy and David A Patterson. 2017. Computer architecture: a quantitative approach. Morgan kaufmann

    2017

  8. [16]

    Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. https://arxiv.org/abs/1506.03340 Teaching machines to read and comprehend . In Advances in Neural Information Processing Systems (NeurIPS), pages 1693--1701

    2015 arXiv

  9. [17]

    Geoffrey Hinton. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531

    2015 arXiv

  10. [18]

    Mark Horowitz. 2014. 1.1 computing's energy problem (and what we can do about it). In 2014 IEEE international solid-state circuits conference digest of technical papers (ISSCC), pages 10--14. IEEE

    2014

  11. [19]

    Nguyen Huu, Suri Sameer, Ken Tsui, Shahules786, Together.xyz team, and Christoph Schuhmann. 2023. Oig-small-chip2. https://huggingface.co/datasets/0-hero/OIG-small-chip2

    2023

  12. [20]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361

    2020 arXiv

  13. [21]

    Yoon Kim and Alexander M Rush. 2016. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317--1327

    2016

  14. [22]

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274--19286. PMLR

    2023

  15. [23]

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.15077

    2024 arXiv

  16. [24]

    Yuhui Li , Fangyun Wei , Chao Zhang , and Hongyang Zhang . 2025. https://doi.org/10.48550/arXiv.2503.01840 EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test . arXiv e-prints, arXiv:2503.01840

    2025 doi arXiv

  17. [25]

    Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, et al. 2023. A pretrainer's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. arXiv preprint...

    2023 arXiv

  18. [26]

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. 2024. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173

    2024 arXiv

  19. [27]

    Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568

    2023 arXiv

  20. [28]

    Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, et al. 2023. Specinfer: Accelerating generative large language model serving with tree-based speculative inference and verification. arX...

    2023 arXiv

  21. [29]

    Cohen, and Mirella Lapata

    Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. https://doi.org/10.18653/v1/D18-1050 Xsum: A new dataset for abstractive summarization of news articles . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 701--711

    2018 doi

  22. [30]

    Guilherme Penedo, Hynek Kydl \' c ek, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. 2024. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406.17557

    2024 arXiv

  23. [31]

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2023. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5

    2023

  24. [32]

    Salesforce wikitext dataset

    Salesforce. Salesforce wikitext dataset. https://huggingface.co/datasets/Salesforce/wikitext

  25. [33]

    Andrea Santilli, Silvio Severino, Emilian Postolache, Valentino Maiorca, Michele Mancusi, Riccardo Marin, and Emanuele Rodol \`a . 2023. Accelerating transformer inference for translation via parallel decoding. arXiv preprint arXiv:2305.10427

    2023 arXiv

  26. [34]

    Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. 2018. Blockwise parallel decoding for deep autoregressive models. Advances in Neural Information Processing Systems, 31

    2018

  27. [35]

    Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, and Beidi Chen. 2024. Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding. arXiv preprint arXiv:2404.11912

    2024 arXiv

  28. [36]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca

    2023

  29. [37]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  30. [38]

    Heming Xia , Yongqi Li , Jun Zhang , Cunxiao Du , and Wenjie Li . 2024. https://doi.org/10.48550/arXiv.2410.06916 SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration . arXiv e-prints, arXiv:2410.06916

    2024 doi arXiv

  31. [39]

    Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. 2024. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. arXiv preprint arXiv:2406.08464

    2024 arXiv

  32. [40]

    Minghao Yan, Saurabh Agarwal, and Shivaram Venkataraman. 2024. Decoding speculative decoding. arXiv preprint arXiv:2402.01528

    2024 arXiv

  33. [41]

    Aonan Zhang, Chong Wang, Yi Wang, Xuanyu Zhang, and Yunfei Cheng. 2024. Recurrent drafter for fast speculative decoding in large language models. arXiv preprint arXiv:2403.09919

    2024 arXiv

  34. [42]

    Jun Zhang , Jue Wang , Huan Li , Lidan Shou , Ke Chen , Gang Chen , and Sharad Mehrotra . 2023. https://doi.org/10.48550/arXiv.2309.08168 Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding . arXiv e-prints, arXiv:2309.08168

    2023 doi arXiv

  35. [43]

    Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, Jean-Fran c ois Kagy, and Rishabh Agarwal. 2023. Distillspec: Improving speculative decoding via knowledge distillation. arXiv preprint arXiv:2310.08461

    2023 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.