Pith. sign in

REVIEW 39 references

Relaxed Speculative Decoding Fails With Lightweight Drafters

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-10 02:56 UTC pith:ZCBPPTWA

load-bearing objection Solid empirical study with one load-bearing generalization gap

arxiv 2607.08690 v1 pith:ZCBPPTWA submitted 2026-07-09 cs.LG cs.AI

A Practical Investigation of Training-free Relaxed Speculative Decoding

Guoxuan Xia , Luka Ribar , Paul Balanca This is my paper
classification cs.LG cs.AI
keywords decodingspeculativerelaxedapproachescapabilitylosslessmodelsampling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Speculative decoding speeds up large language model inference by having a smaller, faster model draft tokens that the large model then verifies in parallel. Standard speculative decoding is lossless: the rejection and resampling steps exactly preserve the large model's output distribution. A recent line of research proposes relaxing that lossless guarantee, accepting slightly more draft tokens in exchange for either more speed, a controllable trade-off between speed and quality, or even improved capability. This paper unifies six training-free relaxed speculative decoding methods into a single framework, benchmarks them on modern reasoning tasks (math olympiad, graduate-level science, competitive programming) with contemporary drafter-verifier pairs, and finds a fundamental mismatch between what relaxed methods need and what modern drafters provide. The core finding is that most relaxed methods implicitly assume the drafter is a competent standalone language model whose distribution can be trusted when it diverges from the verifier. Lightweight dedicated multi-token-prediction (MTP) drafters, which the field is moving toward, are specialized for fast drafting under strict rejection and are not good language models in their own right. When relaxation tries to rely on their distribution, the result is degraded capability, longer rambling responses, and sometimes outright slowdowns even though raw token throughput increases. Only methods that tightly bound how far the relaxed distribution can drift from the verifier (CACTUS and mentored-dec) produce useful trade-offs with such drafters. A strong standalone drafter, by contrast, can yield approximately lossless speed-ups from relaxation, but at the cost of much higher drafter overhead that limits real-world gains. The paper also shows that optimizing draft length under strict speculative decoding has comparable impact on speed-up to relaxation itself, without requiring any capability re-evaluation, and that relaxation hyperparameters do not transfer across tasks.

Core claim

The central discovery is a conflict between the design assumptions of relaxed speculative decoding and the direction the inference community is heading. Relaxed methods that trade losslessness for speed implicitly require the drafter to be a reliable language model, because they accept draft tokens based on the drafter's own distribution or confidence. Dedicated MTP drafters are built to be fast and to maintain short-range acceptance under strict rejection, not to generate coherent long-form text. When these lightweight drafters are used with relaxation, the model produces longer, lower-quality responses, sometimes entering repetition loops that inflate generation length and erase any speed.

What carries the argument

The paper introduces a unified framework (Algorithm 2) that replaces three distributions in standard speculative decoding (rejection target, residual sampling distribution, and bonus token distribution) with relaxed targets parameterized by a relaxation parameter alpha. Six methods (CACTUS, mentored-dec, r-fuzzy, spec-casc-opt, ensemble, spec-cont-dec) are expressed as different choices for these relaxed distributions. Speed-up is modeled with a proxy that accounts for both token acceptance rates and changes in average response length, evaluated across a grid of draft lengths and relative drafter costs.

Load-bearing premise

The speed-up proxy model assumes that the relative cost of drafting versus verification (c_rel) and the average accepted draft length are sufficient to estimate real-world speed-up, but real wall-clock performance depends on hardware-software specifics that could systematically favor or penalize certain methods differently than the proxy predicts.

What would settle it

If a future MTP drafter architecture were designed to be both lightweight and a competent standalone language model, or if serving frameworks optimized the drafter path to reduce c_rel dramatically for standalone drafters, the conflict between relaxation and lightweight drafters would diminish and the trade-offs could shift in favor of relaxed methods.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Practitioners adopting bundled MTP drafters should prioritize optimizing draft length under strict speculative decoding before considering any relaxation, as the former yields comparable speed-up gains without capability re-evaluation.
  • Methods that tightly control deviation from the verifier distribution are the only viable relaxation path for lightweight drafters; methods that trust the drafter's distribution require a strong standalone language model as drafter.
  • Relaxation hyperparameters need per-task calibration, making relaxed speculative decoding a deployment-specific optimization rather than a drop-in replacement for strict speculative decoding.
  • Speculative contrastive decoding (spec-cont-dec) offers a distinct trade-off direction, sacrificing speed for potential capability gains when the drafter is weak, which could be useful in capability-constrained rather than speed-constrained deployments.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

0 major / 0 minor

Summary. This paper presents a practical investigation of training-free relaxed speculative decoding methods, which relax the strict distribution-preservation guarantee of standard speculative decoding in exchange for potential speed-ups or capability-speed trade-offs. The authors unify six existing approaches (CACTUS, mentored-dec, r-fuzzy, spec-casc-opt, ens, spec-cont-dec) within a shared algorithmic framework (Algorithm 2), benchmark them across three drafter-verifier pairs (dedicated MTP, weak LM, strong LM) and three reasoning benchmarks (AIME24, GPQA, LCB), and distill eight key findings for practitioners. The central empirical finding is that many relaxed approaches rely on the drafter being a good standalone language model, making them unsuited for lightweight dedicated multi-token-prediction (MTP) drafters—the direction the community is moving toward. The paper also provides a primer on strict speculative decoding, a speed-up model (Eqs. 2 and 4), and a proof that the reducible variant of fuzzy speculative decoding (r-fuzzy) weakly improves draft acceptance over the standard fuzzy variant while preserving the same output distribution.

Significance. The paper addresses a timely and practically important question. As speculative decoding sees increasing adoption in production LLMs (DeepSeek, Qwen, etc.) and inference frameworks (vLLM, SGLang), and as the community shifts toward bundled MTP drafters, understanding whether relaxed speculative decoding methods actually work in these settings is of clear value to practitioners. The unification of disparate methods under a single framework (Algorithm 2, Table 2) is a useful contribution that makes the landscape more navigable. The finding that MTP drafters are largely unsuited for relaxed methods (except CACTUS/mentored-dec) is a non-trivial and potentially impactful result that challenges the applicability of a body of recent literature. The authors provide reproducible code and a response-length-aware speed-up metric (Eq. 4) that corrects a common omission in prior work. The r-fuzzy proof (Appendix C) is a clean, self-contained result. The experimental design—spanning MTP/weak/strong drafters, multiple benchmarks, draft lengths, and relaxation parameters with standard error bars—is thorough relative to the existing literature, which the authors correctly note suffers from narrow,

Circularity Check

0 steps flagged

No circularity: empirical investigation with externally-sourced formulas and no self-citation chain

full rationale

This paper is an empirical benchmarking study, not a derivation paper. The speed-up proxy model (Eq. 2) is explicitly attributed to Leviathan et al. (2023) and used as a tool, not presented as a novel derivation. The acceptance probability formula (Eq. 3) is cited from Yin et al. (2024). The r-fuzzy proof (Appendix C) relies on the standard speculative-correction fact from Yin et al. (2024), an independent external result. The taxonomy (Table 2) organizes methods from Hao & Mou (2026), Tran-Thien (2023), Holsman et al. (2025), Narasimhan et al. (2025), Wang et al. (2026), and Yuan et al. (2024) — none of which are self-citations by the current authors (Xia, Ribar, Balanca). The one self-citation (Xia & Bouganis 2023) appears only as background context for early-exit cascades in Section 3 and is not load-bearing for any claim. The key findings (e.g., Key Finding 4 about MTP drafters) are directly measured empirical observations (capability scores on AIME24/GPQA/LCB), not predictions derived from a model that could reduce to its inputs. No step in the paper's argument chain reduces by construction to a fitted constant or a self-citation.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The paper introduces no new entities, particles, or postulated objects. It reformulates existing methods under a shared framework (Algorithm 2) and benchmarks them. The free parameters are experimental hyperparameters (c_rel, N_draft, alpha) swept over grids, not constants fitted to make a derivation work. The axioms are standard domain assumptions from the speculative decoding literature.

free parameters (3)
  • c_rel = {0.05, 0.2, 0.5}
    Relative drafter cost is swept over three values as a proxy for real deployment scenarios rather than measured end-to-end for each configuration.
  • N_draft = {3, 5, 10, 20}
    Draft length is swept over four values; not a fitted parameter per se but an inference-time hyperparameter grid.
  • alpha (per method) = varies by method, e.g. CACTUS {0.1, 0.25, 1.0, 10.0}
    Relaxation parameter grids are method-specific and chosen by the authors; the paper shows these do not generalize across tasks.
axioms (4)
  • domain assumption C_verify ≈ C_AR (verification cost approximates single AR step cost)
    Stated in §2.1 as a common assumption enabling the simplified speed-up model (Eq. 2). Holds when AR decoding is memory-bound but may break at high batch sizes.
  • standard math Standard speculative correction fact: accepting v~q with prob min{1, rho(v)/q(v)} and resampling from norm((rho-q)+) emits tokens with marginal distribution rho
    Used in the r-fuzzy proof (Appendix C), attributed to Yin et al. (2024). This is a well-established result in the spec-dec literature.
  • domain assumption Top-p/top-k softmax truncation is standard practice for LLM generation
    Used to define the plausibility set M_top-p/k for spec-cont-dec (Eq. 14) and as support truncation across all experiments. Cited to Noarov et al. (2025).
  • domain assumption Jensen-Shannon Divergence is the appropriate divergence criterion for r-fuzzy
    Recommended in Holsman et al. (2025) to avoid divide-by-zero issues with KL-divergence on non-overlapping support from top-p/k masking.

pith-pipeline@v1.1.0-glm · 38163 in / 2818 out tokens · 192160 ms · 2026-07-10T02:56:09.203767+00:00 · methodology

0 comments
read the original abstract

Speculative decoding accelerates sampling from an autoregressive LLM by using a faster auxiliary model to draft tokens which are then verified in parallel by the LLM. Standard speculative decoding is lossless: its rejection and resampling steps exactly preserve the LLM's sampling distribution. Recent work argues that relaxing this strict guarantee can yield further speed-ups, controlled capability-speed trade-offs, or even capability gains. We practically investigate training-free relaxed speculative decoding techniques, unify existing approaches within a shared framework, benchmark them on contemporary settings, and distil takeaways and empirical findings for practitioners. Important takeaways include: relaxation can require considerable capability evaluation unlike lossless speculative decoding, and many relaxed approaches rely on a drafter that is a good language model, making them unsuited for lightweight dedicated multi-token-prediction drafters.

Figures

Figures reproduced from arXiv: 2607.08690 by Guoxuan Xia, Luka Ribar, Paul Balanca.

Figure 1
Figure 1. Figure 1: Left: In strict speculative decoding tokens are rapidly drafted and then verified in parallel. Draft tokens are stochastically rejected; the first rejected token is resampled such that overall sampling is from verifier target distribution p. Right: Relaxed speculative decoding can replace p with relaxed target distribution π at different points. 2. We build a taxonomy over the literature of relaxed specula… view at source ↗
Figure 2
Figure 2. Figure 2: Left: The speed-up model in Eq. (2). Drafter relative cost crel has a strong impact on speed-up, the best choice of Ndraft, and the speed-up gain from increased average accepted length ¯laccept. Right: strict spec-dec gives lossless speed-up, whilst relaxed spec-dec trades capability for even more speed. If deployment is capability constrained, the former is lossless so it doesn’t require the burden of add… view at source ↗
Figure 3
Figure 3. Figure 3: Comparison between CACTUS and mentored-dec. Left: a visualisation of how the relaxed probability of the draft token x is boosted, right: empirical trade-off comparisons. Their algorithmic and narrative similarities (Sec. 4) result in similar empirical behaviour. We omit mentored-dec in favour of CACTUS in the main paper body for brevity. . drafter module, a weak language model drafter and a strong language… view at source ↗
Figure 4
Figure 4. Figure 4: Capability-speed trade-offs on AIME24 for relaxed spec-dec methods across drafter+verifier pairs, relative drafter costs crel and draft lengths Ndraft. Standard error lines indicate measurement noise from stochastic sampling. (A) Ndraft has a comparable effect on speed-up as relaxation. (B) large speed-ups/increases in average accepted length ¯laccept at high Ndraft may become irrelevant depending on crel,… view at source ↗
Figure 5
Figure 5. Figure 5: Left: (spec-casc-opt) drafters that are weak language models improve token throughput much more than length-adjusted speed-up. Right: relaxation into an MTP drafter increases generation length with rambling responses. models, and so can’t be relied on for relaxation, even if they are decent drafters under strict rejection [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Relaxation hyperparameters do not reliably generalise between evaluation tasks. Comparing [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Capability-speed trade-offs on AIME24 for relaxed spec-dec methods across drafter+verifier pairs, relative drafter costs crel and draft lengths Ndraft. Each block shows one drafter+verifier pair, with method columns and crel rows. The x-axis is length-adjusted speed-up (log scale) and the y-axis is task capability. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Capability-speed trade-offs on GPQA for relaxed spec-dec methods across drafter+verifier pairs, relative drafter costs crel and draft lengths Ndraft. Each block shows one drafter+verifier pair, with method columns and crel rows. The x-axis is length-adjusted speed-up (log scale) and the y-axis is task capability. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Capability-speed trade-offs on LCB for relaxed spec-dec methods across drafter+verifier pairs, relative drafter costs crel and draft lengths Ndraft. Each block shows one drafter+verifier pair, with method columns and crel rows. The x-axis is length-adjusted speed-up (log scale) and the y-axis is task capability. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Capability-speed trade-offs on GPQA for relaxed spec-dec methods for Llama3, relative drafter costs crel and draft lengths Ndraft. The x-axis is length-adjusted speed-up (log scale) and the y-axis is task capability. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Modelled memory cost for native-MTP Qwen3.5 models as batch size increases. The decomposition [PITH_FULL_IMAGE:figures/full_fig_p029_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

39 extracted references · 39 canonical work pages · 7 internal anchors

  1. [1]

    1 Gregor Bachmann, Sotiris Anagnostidis, Albert Pumarola, Markos Georgopoulos, Artsiom Sanakoyeu, Yuming Du, Edgar Schönfeld, Ali Thabet, and Jonas K Kohler

    Accessed: 2026-05-29. 1 Gregor Bachmann, Sotiris Anagnostidis, Albert Pumarola, Markos Georgopoulos, Artsiom Sanakoyeu, Yuming Du, Edgar Schönfeld, Ali Thabet, and Jonas K Kohler. Judge decoding: Faster speculative sampling requires going beyond model alignment. InThe Thirteenth International Conference on Learning Representations,

    work page 2026

  2. [2]

    URL https://openreview.net/forum?id=mtSSFiqW6y. 1, 5 Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris H...

    work page 1901

  3. [3]

    1 Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D

    URL https://proceedings.neurips.cc/paper_files/paper/ 2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf. 1 Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nu...

    work page 2020

  4. [4]

    5 Haw-Shiuan Chang, Nanyun Peng, Mohit Bansal, Anil Ramakrishna, and Tagyoung Chung

    URLhttps://proceedings.mlr.press/v235/cai24b.html. 5 Haw-Shiuan Chang, Nanyun Peng, Mohit Bansal, Anil Ramakrishna, and Tagyoung Chung. Explaining and improving contrastive decoding by extrapolating the probabilities of a huge and hypothetical LM. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.),Proceedings of the 2024 Conference on Empirical M...

    work page 2024

  5. [5]

    doi: 10.18653/v1/2024.emnlp-main.484

    Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.484. URLhttps://aclanthology.org/2024.emnlp-main.484/. 8 Jian Chen, Yesheng Liang, and Zhijian Liu. Dflash: Block diffusion for flash speculative decoding,

    work page doi:10.18653/v1/2024.emnlp-main.484 2024

  6. [6]

    DFlash: Block Diffusion for Flash Speculative Decoding

    URL https://arxiv.org/abs/2602.06036. 3, 5 Zhuoming Chen, Avner May, Ruslan Svirschevski, Yuhsun Huang, Max Ryabinin, Zhihao Jia, and Beidi Chen. Sequoia: scalable and robust speculative decoding. InProceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY , USA,

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    DeepSeek-V3 Technical Report

    URLhttps://arxiv.org/abs/2412.19437. 1, 6 Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 889–898, Melbourne, Australia, July

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    doi: 10.18653/v1/P18-1082

    Association for Computational Linguistics. doi: 10.18653/v1/P18-1082. URL https://aclanthology.org/P18-1082/. 8 13 Jiale Fu, Yuchu Jiang, Junkai Chen, Jiaming Fan, Xin Geng, and Xu Yang. Fast large language model collabo- rative decoding via speculation. InForty-second International Conference on Machine Learning,

    work page doi:10.18653/v1/p18-1082

  9. [10]

    GLM-5: from Vibe Coding to Agentic Engineering

    URL https://arxiv.org/abs/2602.15763. 6 Yongchang Hao and Lili Mou. Cactus: Accelerating auto-regressive decoding with constrained acceptance speculative sampling. InThe Fourteenth International Conference on Learning Representations,

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [11]

    4, 6, 7, 9, 23 Zhenyu He, Zexuan Zhong, Tianle Cai, Jason Lee, and Di He

    URL https://openreview.net/forum?id=lpUIkCAy9p. 4, 6, 7, 9, 23 Zhenyu He, Zexuan Zhong, Tianle Cai, Jason Lee, and Di He. REST: Retrieval-based speculative decoding. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.),Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technol...

    work page 2024

  11. [12]

    doi: 10.18653/v1/2024.naacl-long.88

    Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long.88. URLhttps://aclanthology.org/2024.naacl-long.88/. 5 Maximilian Holsman, Yukun Huang, and Bhuwan Dhingra. Fuzzy speculative decoding for a tunable accuracy-runtime tradeoff. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.),Findings of the Ass...

    work page doi:10.18653/v1/2024.naacl-long.88 2024

  12. [13]

    ISBN 979-8-89176-256-5

    Association for Computational Linguistics. ISBN 979-8-89176-256-5. doi: 10.18653/v1/2025.findings-acl.1346. URLhttps://aclanthology.org/2025.findings-acl.1346/. 1, 4, 7, 20, 23 Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. InInter- national Conference on Learning Representations,

    work page doi:10.18653/v1/2025.findings-acl.1346 2025

  13. [14]

    Accessed: 2026-05-29

    URL https://blog.google/innovation-and-ai/technology/ developers-tools/multi-token-prediction-gemma-4/. Accessed: 2026-05-29. 6 14 Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative de- coding. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.),Pro...

    work page 2026

  14. [15]

    doi: 10.18653/v1/2023.acl-long.687

    Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.687. URL https://aclanthology.org/2023.acl-long.687/. 8, 12 Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: speculative sampling requires rethinking feature uncertainty. InProceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org,

    work page doi:10.18653/v1/2023.acl-long.687 2023

  15. [16]

    3, 4 LiveCodeBench

    URLhttps://arxiv.org/abs/2601.11580. 3, 4 LiveCodeBench. LiveCodeBench code generation lite, release_v6,

    work page arXiv

  16. [17]

    The Llama 3 Herd of Models

    URL https://huggingface.co/datasets/ livecodebench/code_generation_lite. Versioned Hugging Face dataset configuration used for LiveCodeBench Lite v6. 22 Llama Team. The llama 3 herd of models.CoRR, abs/2407.21783,

    work page internal anchor Pith review Pith/arXiv arXiv

  17. [18]

    The Llama 3 Herd of Models

    doi: 10.48550/arXiv.2407.21783. URL https://doi.org/10.48550/arXiv.2407.21783. 9, 23 Xiaoyu Ma and David Patterson. Challenges and research directions for large language model inference hardware. Computer, 59(5):55–64,

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2407.21783

  18. [19]

    doi: 10.1109/MC.2026.3652916. 2 Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification....

    work page doi:10.1109/mc.2026.3652916 2026

  19. [20]

    ISBN 9798400703867

    Association for Computing Machinery. ISBN 9798400703867. doi: 10.1145/3620666.3651335. URLhttps://doi.org/10.1145/3620666.3651335. 5 Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Seungyeon Kim, Neha Gupta, Aditya Krishna Menon, and Sanjiv Kumar. Faster cascades via speculative decoding. InThe Thirteenth International Conference on Learni...

    work page doi:10.1145/3620666.3651335

  20. [21]

    4, 6, 7, 8, 11, 20, 23 Shashi Narayan, Shay B

    URL https://openreview.net/forum?id=vo9t20wsmd. 4, 6, 7, 8, 11, 20, 23 Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii (eds.),Proceedings of the 2018 Conference on Empiri...

    work page 2018

  21. [22]

    doi: 10.18653/v1/D18-1206

    Association for Computational Linguistics. doi: 10.18653/v1/D18-1206. URLhttps://aclanthology.org/D18-1206/. 8 15 Georgy Noarov, Soham Mallick, Tao Wang, Sunay Joshi, Yan Sun, Yangxinyu Xie, Mengxin Yu, and Edgar Dobriban. Foundations of top-k decoding for language models. InAdvances in Neural Information Processing Systems, volume 38,

    work page doi:10.18653/v1/d18-1206

  22. [23]

    9 NVIDIA

    URL https://proceedings.neurips.cc/paper_files/paper/2025/hash/ a96d4fda3017f1773b261a52a3efc8dd-Abstract-Conference.html. 9 NVIDIA. Nemotron 3 ultra: Open, efficient mixture-of-experts hybrid mamba-transformer model for agentic reasoning, June

    work page 2025

  23. [24]

    8, 12 OpenAI

    URL https://openreview.net/forum?id=SzV37yefM4. 8, 12 OpenAI. Why SWE-bench verified no longer measures frontier coding capabilities. https://openai.com/index/ why-we-no-longer-evaluate-swe-bench-verified/, February 2026a. Accessed: 2026-06-06. 5 OpenAI. Introducing gpt-5.3-codex-spark. https://openai.com/index/introducing-gpt-5-3-codex-spark/ , February ...

    work page 2026

  24. [25]

    URLhttps://openreview.net/forum?id=Ti67584b98. 9, 22 Sha Sajadieh, Loredana Fattorini, Raymond Perrault, Yolanda Gil, Vanessa Parli, Lapo Santarlasci, Juan Pava, Nestor Maslej, Russ Altman, Erik Brynjolfsson, Carla Brodley, Jack Clark, Virginia Dignum, Vipin Kumar, James Landay, Terah Lyons, James Manyika, Juan Carlos Niebles, Yoav Shoham, Elham Tabassi, ...

    work page 2026

  25. [26]

    1 Alexander Samarin, Sergei Krutikov, Anton Shevtsov, Sergei Skvortsov, Filipp Fisin, and Alexander Golubev

    URL https://hai.stanford.edu/ai-index/2026-ai-index-report. 1 Alexander Samarin, Sergei Krutikov, Anton Shevtsov, Sergei Skvortsov, Filipp Fisin, and Alexander Golubev. LK losses: Direct acceptance rate optimization for speculative decoding,

    work page 2026

  26. [27]

    URL https://arxiv.org/abs/2602.23881. ICML

    work page internal anchor Pith review Pith/arXiv arXiv

  27. [28]

    Accessed: 2026-05-29

    URL https://vllm.ai/blog/2025-12-13-speculators-v030. Accessed: 2026-05-29. 1, 6 Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1073–1083, Vancouver, Canada, July

    work page 2025

  28. [29]

    Liu, and Christopher D

    Association for Computational Linguistics. doi: 10.18653/v1/P17-1099. URLhttps://aclanthology.org/P17-1099/. 8 Sonali Singh, Karthik Sangaiah, Shenrun Zhang, Ryan Swann, and Ganesh Dasika. Accelerating LLM inference: Up to 3x speedup on MI300X with speculative decoding. ROCm Blogs, March

    work page doi:10.18653/v1/p17-1099

  29. [30]

    com/artificial-intelligence/spec_decode_mi300x/README.html

    URL https://rocm.blogs.amd. com/artificial-intelligence/spec_decode_mi300x/README.html. Accessed: 2026-05-30. 3 Ziteng Sun, Uri Mendlovic, Yaniv Leviathan, Asaf Aharoni, Jae Hun Ro, Ahmad Beirami, and Ananda Theertha Suresh. Block verification accelerates speculative decoding. InThe Thirteenth International Conference on Learning Representations,

    work page 2026

  30. [31]

    1 Ziyi Wang, Siva Rajesh Kasa, Ankith M S, Santhosh Kumar Kasa, Jiaru Zou, Sumit Negi, Ruqi Zhang, Nan Jiang, and Qifan Song

    URL https://proceedings.neurips.cc/paper_files/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf. 1 Ziyi Wang, Siva Rajesh Kasa, Ankith M S, Santhosh Kumar Kasa, Jiaru Zou, Sumit Negi, Ruqi Zhang, Nan Jiang, and Qifan Song. DIVERSED: Relaxed speculative decoding via dynamic ensemble verification. InThe 29th International Conference on Artificial...

    work page 2017

  31. [32]

    Qwen3 Technical Report

    7 An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang...

    work page internal anchor Pith review Pith/arXiv arXiv

  32. [33]

    Accessed: 2026-05-29

    URL https://epoch.ai/data-insights/ai-chip-production . Accessed: 2026-05-29. 1 Hongyi Yuan, Keming Lu, Fei Huang, Zheng Yuan, and Chang Zhou. Speculative contrastive decoding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.),Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 2: Short Papers), pp. 56–6...

    work page 2026

  33. [34]

    doi: 10.18653/v1/2024.acl-short.5

    Association for Compu- tational Linguistics. doi: 10.18653/v1/2024.acl-short.5. URL https://aclanthology.org/2024.acl-short.5/. 1, 6, 7, 8, 23 Yifan Zhang and Math-AI Team. American invitational mathematics examination (aime) 2024,

    work page doi:10.18653/v1/2024.acl-short.5 2024

  34. [35]

    Notation Meaning q,pDrafter and verifier models;pis the LLM being accelerated q(·),p(·)Drafter/verifier next-token distributions qt,pt,qt+i,pt+i Next-token distributions at positionstort+i x<t Autoregressive context before positiont xt,xt+i Tokens at positionstandt+i x,v,VDraft token, vocabulary index, and vocabulary size ∆V−1 Probability simplex over the...

    work page 2025

  35. [36]

    The exam is part of the U.S

    evaluates olympiad-style contest mathematics with a three-digit integer answer. The exam is part of the U.S. mathematical olympiad qualification pathway, which gives context for the level of contest problem solving being evaluated. We use the 30-question AIME 2024 set distributed by HuggingFaceH4, with 48 final repeats per problem for each reported condit...

    work page 2024

  36. [37]

    The questions are written for domain experts in biology, physics and chemistry, so the task is meant to probe expert-level scientific knowledge rather than general STEM familiarity

    evaluates graduate-level STEM scientific reasoning in a multiple-choice format. The questions are written for domain experts in biology, physics and chemistry, so the task is meant to probe expert-level scientific knowledge rather than general STEM familiarity. We use the 198-question Diamond split after applying a fixed answer-choice shuffle following th...

    work page 2025

  37. [38]

    ### Format: Read the inputs from stdin solve the problem and write the answer to stdout (do not directly test on the sample inputs)

    Print the sum, counting repeated table entries separately. ### Format: Read the inputs from stdin solve the problem and write the answer to stdout (do not directly test on the sample inputs). Enclose your code within delimiters as follows. Ensure that when the python program runs, it reads the inputs, runs the algorithm and writes output to STDOUT. ```pyt...

    work page 2025

  38. [39]

    The standard thinking budget is 32,768 tokens and the maximum generation length is 36,864 tokens

    for thinking mode, temperature=0.6, top- p=0.95 and top-k=20, omitting additional logit-penalty parameters for simplicity. The standard thinking budget is 32,768 tokens and the maximum generation length is 36,864 tokens. The Qwen3.5-27B native-MTP setting uses a 65,536-token thinking budget and 69,632-token maximum generation length onAIME24andLCB. The ge...

    work page 2023

  39. [40]

    Overall, broader results across Figs

    These figures use the same capability-speed grid as the main body, without the highlighted annotations, so the broader behaviour across tasks can be inspected directly. Overall, broader results across Figs. 7 to 10 match the takeaways and findings in the main body of the paper. 14Model cards: https://huggingface.co/Qwen/Qwen3-32B, https://huggingface.co/Q...

    work page 2048