Pith. sign in

REVIEW 4 major objections 4 minor 56 references

Short prompt suffixes can systematically collapse the acceptance rate inside speculative decoding, turning a lossless accelerator into a latency and cost amplifier while leaving final answers mostly correct.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 06:36 UTC pith:HASA57BP

load-bearing objection ADSD is a genuinely new attack surface — prompt-side verifier collapse — with a clean surrogate objective and convincing ablations, but the 'preserving task quality' claim only holds for the matched GSM8K setting and production relevance is unproven. the 4 major comments →

arxiv 2607.21804 v1 pith:HASA57BP submitted 2026-07-23 cs.CR cs.CLcs.LG

Adversarial Prompts for Acceptance Collapse in Speculative Decoding

Run Wang , Chaoyi Zhou , Xi Liu , Yi Zhu , Amir Salarpour , Pedram MohajerAnsari , Zhi-Qi Cheng , Feng Luo
show 2 more authors
Siyu Huang Mert D. Pes\'e
This is my paper
classification cs.CR cs.CLcs.LG
keywords speculative decodingadversarial prompt suffixverifier acceptance collapseavailability attackdenial-of-walletdraft-target alignmentlatency inflationwhite-box attack
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that a short text suffix appended to a benign prompt can systematically collapse the acceptance rate of speculative decoding, a lossless acceleration technique in which a small draft model proposes tokens and a larger target model verifies them in parallel. The attack works because the tokenwise acceptance rule α = min(1, p/q) rejects a draft token whenever the draft probability q exceeds the target probability p; by pushing draft probability mass into that rejection region, an attacker forces the verifier into repeated wasted rounds. On a math word-problem benchmark, the optimized suffix raises mean sample time from 26.05 to 42.29 seconds (+62.3%) while GSM8K accuracy only slips from 0.821 to 0.802, so the slowdown is not easily visible in answer quality. If correct, this means the 'lossless' guarantee of speculative decoding concerns only output semantics: the operational cost of a request is attacker-controllable, and one adversarial request can amplify tail latency for co-scheduled benign users in batched serving.

Core claim

The paper's central claim is that the draft-verifier acceptance mechanism inside speculative decoding is a usable attack surface. The authors introduce ADSD, a prompt-suffix optimization that maximizes the verifier's expected rejection by optimizing a verifier-aligned surrogate, Soft-Collapse, derived directly from the acceptance rule, and it anchors the target model's distribution to its benign behavior with a forward-KL term so the final answer remains coherent. They report that a single 5-token universal suffix, calibrated on 10 GSM8K prompts, increases mean sample time by 62.3% on the matching benchmark, that the same suffix transfers to code and summarization (with larger latency effect

What carries the argument

The load-bearing identity is the tokenwise acceptance rule α(v) = min(1, p(v)/q(v)) = exp(−ReLU(log q(v) − log p(v))), where q is the draft's next-token distribution and p is the target's. Rejection pressure appears exactly when the draft is overconfident, i.e., the positive log-margin Δ(v) = log q(v) − log p(v) > 0. The Soft-Collapse objective SC(q, p) = Σ_v q(v)·ReLU(Δ(v)) sums draft-weighted rejection risk over the vocabulary, and the target-preservation penalty KL(p0 || p^u) keeps the attacked target distribution close to the benign one. Gradient-guided sequential beam search over suffix token embeddings solves the discrete optimization, yielding a universal suffix of length m=5 with lin

Load-bearing premise

The attack assumes the target service uses the exact tokenwise acceptance rule α(v) = min(1, p(v)/q(v)) and that the attacker has white-box access to both the draft distribution q and the target distribution p at optimization time; a different verification rule or hidden internals would void the claimed collapse.

What would settle it

Run ADSD-style suffix optimization against a production speculative-decoding engine that uses tree-based verification or a modified acceptance rule (e.g., typical sampling or entropy-based acceptance) and measure whether block efficiency still falls by roughly 39% and latency rises by ~60%; alternatively, measure the same suffix in a continuous-batching server under load to see whether batching amortizes the wasted target passes and shrinks the per-request slowdown.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A deployed speculative-decoding service using tokenwise acceptance becomes a latency-amplification target: one 5-token universal suffix raises mean per-request time by roughly 60% on arithmetic reasoning tasks without visibly corrupting the answer.
  • The vulnerability holds across verification layouts (tokenwise, blockwise, hierarchical) and across draft-target architectures (independent small-to-large draft pairs, LLaMA-3 70B→8B, and shared-head EAGLE-3), so redesigning the verification procedure alone does not remove the attack.
  • Because the target model can repair the draft's corrupted prefix, the final user-visible answer can remain correct while internal verifier cost has already exploded—so post-hoc answer-quality checks are insufficient to detect the attack (paper's Appendix B.1 case study).
  • The authors' proposed mitigation, acceptance-rate monitoring with fallback to target-only decoding, can bound repeated cost amplification but cannot protect the first slowed request; a batch neighbor's latency can still be amplified through shared GPU execution.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • I would expect the same Soft-Collapse logic to transfer to any verification rule that rejects when the drafter is overconfident relative to a reference distribution; changing to a symmetric rule that also penalizes target-overconfidence would likely shrink the attack, but the paper does not test that variant.
  • The attack's white-box requirement could be relaxed: a suffix optimized against a distilled surrogate for the target distribution, or against the draft model alone plus a cheap proxy of p, might produce a transferable black-box attack; the paper only evaluates white-box optimization.
  • A practical defense the paper leaves implicit is to calibrate the drafter's confidence (e.g., temperature scaling on the draft head) so that overconfident proposals are less frequent benignly; this is testable and would reduce the ReLU margin the attack exploits, but it could also reduce benign acceptance.
  • The transfer results suggest latency transfer and quality preservation decouple: the GSM8K-optimized suffix slows code generation dramatically but visibly damages it, so in practice an adaptive attacker would likely generate per-domain suffices, and this is a testable extension of the threat model.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes ADSD, a prompt-suffix attack against speculative decoding. The attacker appends a short discrete suffix to a benign prompt to push the draft distribution toward tokens the target verifier is unlikely to accept, thereby reducing acceptance rate and increasing end-to-end latency while attempting to keep final answer quality close to benign. The method combines Soft-Collapse (Eq. 3), derived from the tokenwise acceptance rule α=min(1,p/q) in Eq. (1), with a forward-KL target-preservation penalty (Eq. 4), optimized via gradient-guided beam search over discrete tokens. Evaluations on GSM8K with Qwen2.5 models report a 62.3% mean-time increase with accuracy dropping from 0.821 to 0.802; additional experiments cover Blockwise/HSD schemes, cross-domain transfer (HumanEval, CNN/DailyMail), larger Qwen and LLaMA models, and EAGLE-3. The paper also includes an ablation showing both objective terms are needed.

Significance. If the results hold, ADSD identifies a new availability attack surface inside the speculative-decoding verification loop, distinct from output-length or guardrail attacks. The core derivation of Soft-Collapse from the tokenwise acceptance rule is mathematically sound, and the ablation in Table 6 supports the necessity of both objective terms. The paper ships detailed reproducibility settings for the main runs, which is a strength. However, the significance is currently narrowed by several unaddressed scope and evidence gaps: the transfer to Blockwise/HSD is asserted without specifying their acceptance rules, the abstract's unqualified 'preserving task quality' is contradicted by the EAGLE-3 accuracy drop, and most effect sizes lack confidence intervals. These issues do not invalidate the central idea but prevent the paper from supporting its broadest claims.

major comments (4)
  1. [§5.3 / Table 2 and Eq. (3)] The Soft-Collapse surrogate is derived from the tokenwise acceptance rule α=min(1,p/q) in Eq. (1). However, Table 2 claims transfer to Blockwise and HSD without specifying the acceptance rules of those schemes or justifying that SC remains a valid surrogate under block- or tree-level verification. If these schemes accept based on joint/tree probabilities or token-set membership, the per-token ReLU(log q − log p) margin in Eq. (3) need not track the actual rejection decision. Please state the verification algorithms used in the simulations and either derive the corresponding surrogate or restrict the theoretical claim to tokenwise schemes. As written, the cross-scheme claim is not supported.
  2. [Abstract vs. Table 5] The abstract states the attack 'preserves the task quality' on GSM8K. Yet Table 5 shows EAGLE-3 GSM8K accuracy dropping from 0.638 to 0.575 (a relative drop of about 9.9%) under ADSD. Since this is the same benchmark and a different architecture, the unqualified claim is contradicted by the paper's own data. Please qualify the quality-preservation claim (e.g., to the tokenwise Qwen settings) or provide an explanation for the EAGLE-3 accuracy loss and its implications for stealth. The phrase 'while preserving the task quality' in Section 1 has the same issue.
  3. [Tables 1, 3, 4, 5] The headline results (e.g., +62.3% time, −38.9% block efficiency) are reported as point estimates without confidence intervals, number of repeated runs, or seed variation. Table 2 includes error bars only on the ADSD rows, not on the benign baselines. Because latency and efficiency are hardware- and scheduling-sensitive, the reader cannot assess whether the reported differences are within run-to-run noise. Please report standard errors or 95% confidence intervals over at least three independent evaluations for every setting, or justify determinism of the measurement and state it explicitly.
  4. [§1, §3.1, §6] The introduction and systemic-impact discussion frame ADSD as a practical denial-of-wallet threat to serving stacks such as vLLM/TGI, but the limitation statement in §6 concedes the method is limited to white-box optimization with access to q and p at deployment time. Production engines generally do not expose these internal per-token distributions to user-space prompts, and may use tree or block verification. Please add an explicit statement in the abstract and introduction that the demonstrated attack requires a white-box deployment replica with the exact tokenwise acceptance rule, and indicate how this affects the practical threat assessment. This is a scope caveat rather than a correctness error in the optimization itself.
minor comments (4)
  1. [§5.4 / Abstract] The text in §5.4 acknowledges that cross-domain transfer harms task quality, but the abstract's phrase 'the vulnerability exists across different domains' could be misread as including stealth. Consider adding 'without preserving stealth' or similar qualification.
  2. [Table 2] The uncertainty notation '(±)' over the GSM8K questions for attack results is undefined; specify whether it is a standard error, standard deviation, or 95% confidence interval, and how it was computed.
  3. [Appendix A] The hyperparameter description says 'the main revised Qwen and EAGLE runs' but there is no earlier mention of 'revised' runs; clarify what distinguishes these from other runs.
  4. [References] Reference [41] appears to have a garbled author list ('Woosuk Kwon, Siheon Min, Sainbayar Sukhbaatar Seo, ...'); please verify the citation.

Circularity Check

0 steps flagged

No significant circularity: the Soft-Collapse objective is intentionally aligned with the acceptance rule, but the headline results are measured by independent wall-clock and quality metrics.

full rationale

The paper's core design step, Soft-Collapse (Eq. 3), is deliberately constructed from the tokenwise acceptance rule (Eqs. 1-2). This is an attack objective, not a scientific prediction: maximizing SC is by construction related to maximizing rejection pressure. However, the paper's empirical claims are not reductions of this objective. The headline result, a 62.3% increase in mean sample time on GSM8K, is measured as wall-clock time and block efficiency in Section 5, which are external to the surrogate loss. The target-preservation term (Eq. 4) is a KL proxy for quality, but task accuracy is evaluated separately, and the EAGLE-3 result (GSM8K accuracy dropping from 0.638 to 0.575) actually shows that the KL term does not definitionally guarantee quality preservation. The HSD [14] citation includes a co-author (Zhi-Qi Cheng), but it is used as an empirical system to test transfer, not as an unverified uniqueness theorem or as a load-bearing self-citation. There is no fitted parameter renamed as a prediction, no imported uniqueness claim, and no ansatz smuggled via citation. The main limitation, white-box access to draft and target distributions, is acknowledged in Section 6 and affects threat-model scope rather than circularity. Overall, the derivation chain is self-contained: the objective is designed from the verifier rule, but the claimed attack effectiveness is independently measured.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 0 invented entities

The attack objective introduces several hand-chosen hyperparameters (λc, λp, suffix length, calibration set, beam size) and relies on the standard speculative decoding acceptance rule and a white-box threat model. No new entities are postulated. The Soft-Collapse surrogate and the KL preservation term are design choices rather than derived facts, and the KL as a stealth proxy is empirically unreliable on non-math domains.

free parameters (7)
  • λc (Soft-Collapse weight) = 2.0
    Weight for the Soft-Collapse term in Eq. 5; chosen by hand, no sensitivity analysis reported.
  • λp (target-preserve weight) = 1.0 (Qwen), 2.0 (EAGLE-3)
    Weight for the forward-KL term in Eq. 5; tuned per architecture to balance slowdown and task-quality preservation.
  • Suffix length m = 5
    Length of the adversarial suffix; fixed without ablation.
  • Calibration prompts N = 10
    Number of GSM8K training prompts used as a calibration set; selected with offset 1 and stride 100.
  • Rollout horizon K = 64
    Number of token positions used in the objective over each calibration prompt.
  • Beam size / candidate set = B=4, top-8 candidates
    Search hyperparameters for the gradient-guided beam search; fixed.
  • Rollout weighting wi = linear decay K-i+1
    Weights earlier positions more heavily; a design choice.
axioms (5)
  • domain assumption The verifier accepts a drafted token v with probability α = min(1, p_i(v)/q_i(v)).
    Eq. (1); the attack's surrogate and evaluation assume this exact asymmetric rule, which holds in standard speculative decoding but not necessarily in all production implementations.
  • domain assumption The attacker has white-box access to draft and target distributions q_i and p_i, the tokenizer, and the verification mechanism.
    Threat model in §3.1; without this access the Soft-Collapse gradient cannot be computed.
  • ad hoc to paper Maximizing Soft-Collapse (Eq. 3) is a valid proxy for reducing verifier acceptance.
    Proposed without a formal equivalence; the ablation (Table 6) shows it is necessary but does not prove sufficiency or optimality.
  • ad hoc to paper A small forward-KL between attacked and benign target distributions (Eq. 4) preserves task quality.
    Empirically false on HumanEval and CNN/DailyMail (Table 3), where task quality drops sharply; used for GSM8K only.
  • standard math Standard probability and information-theoretic identities (e.g., ReLU/log/exp manipulations) are valid.
    Used in the derivation of Eqs. (2)-(4).

pith-pipeline@v1.3.0-alltime-deepseek · 13590 in / 14262 out tokens · 129359 ms · 2026-08-01T06:36:49.633412+00:00 · methodology

0 comments
read the original abstract

Lossless acceleration schemes, such as speculative decoding, promise significant inference speedups by relying on dynamic token-level alignment between a draft and a target model. However, this guarantee of semantic equivalence masks a severe operational vulnerability: draft-target alignment can be systematically attacked. In this paper, we introduce ADSD, which, to the best of our knowledge, is the first prompt-suffix attack that collapses verifier acceptance by pushing draft probability mass toward tokens the target is unlikely to accept. ADSD uses Soft-Collapse, a verifier-aligned surrogate derived from the asymmetric speculative acceptance rule, together with a target-preservation objective that discourages obvious task corruption. ADSD successfully generates highly effective adversarial suffixes. On the GSM8K dataset, our attack increases the mean sample time by 62.3% while preserving the task quality. We further show that this vulnerability exists across different domains, speculative decoding strategies, and model architectures.

Figures

Figures reproduced from arXiv: 2607.21804 by Amir Salarpour, Chaoyi Zhou, Feng Luo, Mert D. Pes\'e, Pedram MohajerAnsari, Run Wang, Siyu Huang, Xi Liu, Yi Zhu, Zhi-Qi Cheng.

Figure 1
Figure 1. Figure 1: Threat model overview. An attacker does not modify model weights or decoding logic. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the ADSD pipeline. ADSD optimizes a short discrete suffix over a calibration [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 18 linked inside Pith

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Pith/arXiv arXiv 2023

  3. [3]

    Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    Pith/arXiv arXiv 2023

  4. [4]

    A simple and effective pruning approach for large language models.arXiv preprint arXiv:2306.11695, 2023

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models.arXiv preprint arXiv:2306.11695, 2023

    Pith/arXiv arXiv 2023

  5. [5]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks.arXiv preprint arXiv:1803.03635, 2018

    Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks.arXiv preprint arXiv:1803.03635, 2018

    Pith/arXiv arXiv 2018

  6. [6]

    Smoothquant: Accurate and efficient post-training quantization for large language models

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. InInternational Conference on Machine Learning, pages 38087–38099. PMLR, 2023

    2023

  7. [7]

    Q-bert: Hessian based ultra low precision quantization of bert

    Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 8815–8821, 2020

    2020

  8. [8]

    Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation

    Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 3909–3925, 2023

    2023

  9. [9]

    Fast inference from transformers via speculative decoding

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. InProceedings of the 40th International Conference on Machine Learning, pages 19274–19286, 2023

    2023

  10. [10]

    Specinfer: Accelerating generative large language model serving with tree-based speculative inference and verification

    Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. Specinfer: Accelerating generative large language model serving with tree-based speculative inference and verification. InProceedings of the...

    2024

  11. [11]

    Lee, Deming Chen, and Tri Dao

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. InProceedings of the 41st International Conference on Machine Learning, pages 5209–5235, 2024

    2024

  12. [12]

    Adaspec: Adap- tive speculative decoding for fast, SLO-aware large language model serving

    Kaiyu Huang, Hao Wu, Zhubo Shi, Han Zou, Minchen Yu, and Qingjiang Shi. Adaspec: Adap- tive speculative decoding for fast, SLO-aware large language model serving. InProceedings of the 2025 ACM Symposium on Cloud Computing, 2025

    2025

  13. [13]

    Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding

    Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. InACL 2024 Findings1, 2024

    2024

  14. [14]

    Overcoming joint intractability with lossless hierarchical speculative decoding

    Yuxuan Zhou, Fei Huang, Heng Li, Fengyi Wu, Tianyu Wang, Jianwei Zhang, Junyang Lin, and Zhi-Qi Cheng. Overcoming joint intractability with lossless hierarchical speculative decoding. InICLR, 2026

    2026

  15. [15]

    Block verification accelerates speculative decoding.arXiv preprint arXiv:2403.10444, 2024

    Ziteng Sun, Uri Mendlovic, Yaniv Leviathan, Asaf Aharoni, Ahmad Beirami, Jae Hun Ro, and Ananda Theertha Suresh. Block verification accelerates speculative decoding.arXiv preprint arXiv:2403.10444, 2024. 9

    Pith/arXiv arXiv 2024

  16. [16]

    Optimized multi-token joint decoding with auxiliary model for llm inference

    Zongyue Qin, Ziniu Hu, Zifan He, Neha Prakriya, Jason Cong, and Yizhou Sun. Optimized multi-token joint decoding with auxiliary model for llm inference. InThe Thirteenth Interna- tional Conference on Learning Representations, 2025

    2025

  17. [17]

    Eagle: speculative sampling requires rethinking feature uncertainty

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: speculative sampling requires rethinking feature uncertainty. InProceedings of the 41st International Conference on Machine Learning, pages 28935–28948, 2024

    2024

  18. [18]

    Eagle-2: Faster inference of language models with dynamic draft trees

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7421–7432, 2024

    2024

  19. [19]

    EAGLE-3: Scaling up inference acceleration of large language models via training-time test

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-3: Scaling up inference acceleration of large language models via training-time test. InAnnual Conference on Neural Information Processing Systems, 2025

    2025

  20. [20]

    Litevlm: A low-latency vision-language model inference pipeline for resource-constrained environments.arXiv preprint arXiv:2506.07416, 2025

    Jin Huang, Yuchao Jin, Le An, and Josh Park. Litevlm: A low-latency vision-language model inference pipeline for resource-constrained environments.arXiv preprint arXiv:2506.07416, 2025

    arXiv 2025

  21. [21]

    Accelerating decode-heavy LLM inference with speculative decoding on AWS trainium and vLLM | artificial intelligence, 2026

    Yahav Biran and Truong Pham. Accelerating decode-heavy LLM inference with speculative decoding on AWS trainium and vLLM | artificial intelligence, 2026

    2026

  22. [22]

    Speculative speculative decoding

    Tanishq Kumar, Tri Dao, and Avner May. Speculative speculative decoding. InThe Fourteenth International Conference on Learning Representations, 2026

    2026

  23. [23]

    Accelerating large language model decoding with speculative sampling.arXiv preprint arXiv:2302.01318, 2023

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling.arXiv preprint arXiv:2302.01318, 2023

    Pith/arXiv arXiv 2023

  24. [24]

    Snakes and ladders: Accelerating ssm inference with speculative decoding

    Yangchao Wu, Yonatan Dukler, Matthew Trager, Alessandro Achille, Wei Xia, and Stefano Soatto. Snakes and ladders: Accelerating ssm inference with speculative decoding. InNeurIPS Efficient Natural Language and Speech Processing Workshop, pages 292–304. PMLR, 2024

    2024

  25. [25]

    Sdsat: Accelerating llm inference through speculative decoding with semantic adaptive tokens.arXiv preprint arXiv:2403.18647, 2024

    Cheng Liu and Yongxin Zhu. Sdsat: Accelerating llm inference through speculative decoding with semantic adaptive tokens.arXiv preprint arXiv:2403.18647, 2024

    Pith/arXiv arXiv 2024

  26. [26]

    Online speculative decoding.arXiv preprint arXiv:2310.07177, 2023

    Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang. Online speculative decoding.arXiv preprint arXiv:2310.07177, 2023

    Pith/arXiv arXiv 2023

  27. [27]

    Direct alignment of draft model for speculative decoding with chat-fine-tuned LLMs

    Raghavv Goel, Mukul Gagrani, Wonseok Jeon, Junyoung Park, Mingu Lee, and Christopher Lott. Direct alignment of draft model for speculative decoding with chat-fine-tuned LLMs. arXiv preprint arXiv:2403.00858, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Sudhanshu Agrawal, Wonseok Jeon, and Mingu Lee. Adaedl: Early draft stopping for specula- tive decoding of large language models via an entropy-based lower bound on token acceptance probability.arXiv preprint arXiv:2410.18351, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    Judge decoding: Faster speculative sampling requires going beyond model alignment.arXiv preprint arXiv:2501.19309, 2025

    Gregor Bachmann, Sotiris Anagnostidis, Albert Pumarola, Markos Georgopoulos, Artsiom Sanakoyeu, Yuming Du, Edgar Schönfeld, Ali Thabet, and Jonas Kohler. Judge decoding: Faster speculative sampling requires going beyond model alignment.arXiv preprint arXiv:2501.19309, 2025

    Pith/arXiv arXiv 2025

  30. [30]

    Alignment-augmented speculative decoding with alignment sampling and conditional verification

    Jikai Wang, Zhenxu Tian, Juntao Li, Qingrong Xia, Xinyu Duan, Zhefeng Wang, Baoxing Huai, and Min Zhang. Alignment-augmented speculative decoding with alignment sampling and conditional verification. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025

    2025

  31. [31]

    Decoding speculative decoding

    Minghao Yan, Saurabh Agarwal, and Shivaram Venkataraman. Decoding speculative decoding. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6460–6473, 2025. 10

    2025

  32. [32]

    When speculation spills secrets: Side channels via speculative decoding in LLMs.arXiv preprint arXiv:2411.01076, 2024

    Jiankun Wei, Abdulrahman Abdulrazzag, Tianchen Zhang, Adel Muursepp, and Gururaj Sailesh- war. When speculation spills secrets: Side channels via speculative decoding in LLMs.arXiv preprint arXiv:2411.01076, 2024

    arXiv 2024

  33. [33]

    Speculative safety-aware decoding

    Xuekang Wang, Shengyu Zhu, and Xueqi Cheng. Speculative safety-aware decoding. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 12827–12841, 2025

    2025

  34. [34]

    Sponge examples: Energy-latency attacks on neural networks

    Ilia Shumailov, Yiren Zhao, Daniel Bates, Nicolas Papernot, Robert Mullins, and Ross Ander- son. Sponge examples: Energy-latency attacks on neural networks. In2021 IEEE European Symposium on Security and Privacy (EuroS&P), pages 212–231, 2021

    2021

  35. [35]

    Morley Mao

    Qingzhao Zhang, Ziyang Xiong, and Z. Morley Mao. LLM safeguard is a double-edged sword: Exploiting false positives for denial-of-service attacks.arXiv preprint arXiv:2410.02916, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    An engorgio prompt makes large language model babble on

    Jianshuo Dong, Ziyuan Zhang, Qingjie Zhang, Tianwei Zhang, Hao Wang, Hewu Li, Qi Li, Chao Zhang, Ke Xu, and Han Qiu. An engorgio prompt makes large language model babble on. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  37. [37]

    Rethinking latency denial-of-service: Attacking the LLM serving framework, not the model.arXiv preprint arXiv:2602.07878, 2026

    Tianyi Wang, Huawei Fan, Yuanchao Shu, Peng Cheng, and Cong Wang. Rethinking latency denial-of-service: Attacking the LLM serving framework, not the model.arXiv preprint arXiv:2602.07878, 2026

    arXiv 2026

  38. [38]

    Amphista: Bi-directional multi-head decoding for acceler- ating llm inference.arXiv preprint arXiv:2406.13170, 2024

    Zeping Li, Xinlong Yang, Ziheng Gao, Ji Liu, Guanchen Li, Zhuang Liu, Dong Li, Jinzhang Peng, Lu Tian, and Emad Barsoum. Amphista: Bi-directional multi-head decoding for acceler- ating llm inference.arXiv preprint arXiv:2406.13170, 2024

    Pith/arXiv arXiv 2024

  39. [39]

    Speculative decoding: Performance or illusion?arXiv preprint arXiv:2601.11580, 2026

    Xiaoxuan Liu, Jiaxiang Yu, Jongseok Park, Ion Stoica, and Alvin Cheung. Speculative decoding: Performance or illusion?arXiv preprint arXiv:2601.11580, 2026

    arXiv 2026

  40. [40]

    An empirical study of speculative decoding for small language models

    Luca Mainardi, Selcuk Sandikci, and Joaquin Vanschoren. An empirical study of speculative decoding for small language models. In Vera Demberg, Kentaro Inui, and Lluís Marquez, editors,Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), 2026

    2026

  41. [41]

    Efficient memory management for large language model serving with PagedAttention

    Woosuk Kwon, Siheon Min, Sainbayar Sukhbaatar Seo, Minsoo Shin, Gilwoo Zhang, Jinsu Kim, Juhwan Kim, Hyeonjun Woo, and Yonghoon Kim. Efficient memory management for large language model serving with PagedAttention. InProceedings of the 29th Symposium on Operating Systems Principles, pages 611–626, 2023

    2023

  42. [42]

    Text generation inference (TGI)

    Hugging Face. Text generation inference (TGI). https://huggingface.co/docs/ text-generation-inference/en/index, 2026

    2026

  43. [43]

    Continuous batching from first principles.https://huggingface.co/blog/continuous_batching, 2025

    Rémi Ouazan Reboul, Arthur Zucker, and Luc Georges. Continuous batching from first principles.https://huggingface.co/blog/continuous_batching, 2025

    2025

  44. [44]

    Openai api pricing.https://openai.com/api/pricing/, 2026

    OpenAI. Openai api pricing.https://openai.com/api/pricing/, 2026

    2026

  45. [45]

    Rate limits

    OpenAI. Rate limits. https://developers.openai.com/api/docs/guides/ rate-limits, 2026

    2026

  46. [46]

    Performance and latency

    Microsoft Azure. Performance and latency. https://learn.microsoft.com/en-us/ azure/foundry/openai/how-to/latency, 2025

    2025

  47. [47]

    Use provisioned throughput

    Google Cloud. Use provisioned throughput. https://cloud.google.com/vertex-ai/ generative-ai/docs/provisioned-throughput/use-provisioned-throughput , 2025

    2025

  48. [48]

    Training verifiers to solve math word problems, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. 11

    2021

  49. [49]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    2021

  50. [50]

    Liu, and Christopher D

    Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), July 2017

    2017

  51. [51]

    Zico Kolter, and Matt Fredrikson

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models.arXiv preprint arXiv:2307.15043, 2023

    Pith/arXiv arXiv 2023

  52. [52]

    Detecting language model attacks with perplexity.arXiv preprint arXiv:2308.14132, 2023

    Gabriel Alon and Michael Kamfonas. Detecting language model attacks with perplexity.arXiv preprint arXiv:2308.14132, 2023

    Pith/arXiv arXiv 2023

  53. [53]

    Baseline de- fenses for adversarial attacks against aligned language models.arXiv preprint arXiv:2309.00614, 2023

    Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. Baseline de- fenses for adversarial attacks against aligned language models.arXiv preprint arXiv:2309.00614, 2023

    Pith/arXiv arXiv 2023

  54. [54]

    LLM self defense: By self examination, LLMs know they are being tricked.arXiv preprint arXiv:2308.07308, 2023

    Mansi Phute, Alec Helbling, Matthew Hull, ShengYun Peng, Sebastian Szyller, Cory Cornelius, and Duen Horng Chau. LLM self defense: By self examination, LLMs know they are being tricked.arXiv preprint arXiv:2308.07308, 2023

    Pith/arXiv arXiv 2023

  55. [55]

    The art of defending: A systematic evaluation and analysis of LLM defense strategies on safety and over-defensiveness

    Neeraj Varshney, Pavel Dolin, Agastya Seth, and Chitta Baral. The art of defending: A systematic evaluation and analysis of LLM defense strategies on safety and over-defensiveness. InFindings of the Association for Computational Linguistics: ACL 2024, pages 13111–13128, 2024

    2024

  56. [56]

    @drawable/box

    Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Pooven- dran. SafeDecoding: Defending against jailbreak attacks via safety-aware decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5587–5605, 2024. 12 A Additional Evaluation Settings Reproducib...

    2024