Pith. sign in

REVIEW 18 cited by

Online Speculative Decoding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.07177 v4 pith:PFL7JURC submitted 2023-10-11 cs.AI cs.CLcs.LG

Xiaoxuan Liu , Lanxiang Hu , Peter Bailis , Alvin Cheung , Zhijie Deng , Ion Stoica , Hao Zhang This is my paper
classification cs.AIcs.CLcs.LG
keywords modeldraftdecodingqueryspeculativedistributiononlinetarget
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Speculative decoding is a pivotal technique to accelerate the inference of large language models (LLMs) by employing a smaller draft model to predict the target model's outputs. However, its efficacy can be limited due to the low predictive accuracy of the draft model, particularly when faced with diverse text inputs and a significant capability gap between the draft and target models. We introduce online speculative decoding to address this challenge. The main idea is to continuously update the (multiple) draft model(s) on observed user query data. Adapting to query distribution mitigates the shifts between the training distribution of the draft model and the query distribution, enabling the draft model to more accurately predict the target model's outputs. We develop a prototype of online speculative decoding based on knowledge distillation and evaluate it using both synthetic and real query data. The results show a substantial increase in the token acceptance rate by 0.1 to 0.65, bringing 1.42x to 2.17x latency reduction. Our code is available at https://github.com/LiuXiaoxuanPKU/OSD.

Discussion (0). Sign in to comment.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. D^2SD: Accelerating Speculative Decoding with Dual Diffusion Draft Models

    cs.DC 2026-06 unverdicted novelty 7.0 of 10

    D^2SD uses two diffusion drafters in a prefix tree structure with confidence scores to select and recover alternative draft sequences, achieving higher acceptance rates in speculative decoding.

  2. Test-Time Speculation

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    Test-Time Speculation adapts draft models online via target-model verifications to sustain high acceptance lengths during long LLM generations.

  3. When RL Meets Adaptive Speculative Training: A Unified Training-Serving System

    cs.LG 2026-02 conditional novelty 7.0 of 10

    Aurora unifies speculative decoder training and serving via asynchronous RL on inference traces, delivering 1.5x day-0 speedup on frontier models and 1.25x adaptation gains on distribution shifts.

  4. Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs

    cs.CL 2024-12 unverdicted novelty 7.0 of 10

    o1-like models overthink easy tasks; self-training reduces compute use without accuracy loss on GSM8K, MATH500, GPQA, and AIME.

  5. A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A training-free self-speculative decoding system with a recallable sparse KV cache and entropy-guided adaptive speculation achieves up to 2.79× speedup while preserving the target distribution.

  6. Adversarial Prompts for Acceptance Collapse in Speculative Decoding

    cs.CR 2026-07 conditional novelty 6.0 of 10

    ADSD shows that a short adversarial suffix appended to a prompt can collapse the token-acceptance rate in speculative decoding, increasing latency by 62.3% on GSM8K while preserving answer accuracy.

  7. DFlare: Scaling Up Draft Capacity for Block Diffusion Speculative Decoding

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    DFlare replaces DFlash's shared fused representation with per-draft-layer attention to distinct target-layer combinations, enabling deeper drafts and 2.4M training samples for 5-11% higher speedups than DFlash on Qwen...

  8. Draft-OPD: On-Policy Distillation for Speculative Draft Models

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    Draft-OPD applies on-policy distillation via target-assisted generation and error replay to train speculative draft models, yielding over 5x lossless acceleration and gains over EAGLE-3 and DFlash.

  9. Performance-Driven Policy Optimization for Speculative Decoding with Adaptive Windowing

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    PPOW uses window-level RL with cost-aware speedup and proximity rewards plus adaptive divergence-aware windowing to reach 6.29-6.52 acceptance lengths and 3.39-4.36x speedups in speculative decoding.

  10. Test-Time Speculation

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    TTS adapts speculator models online via target model verifications to improve acceptance lengths by up to 72% over prior methods, with gains increasing for longer generations.

  11. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty

    cs.LG 2024-01 unverdicted novelty 6.0 of 10

    EAGLE resolves feature-level uncertainty in speculative sampling via one-step token advancement, delivering 2.7x-3.5x speedup on LLaMA2-Chat 70B and doubled throughput across multiple model families and tasks.

  12. AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding

    cs.CL 2026-07 conditional novelty 5.0 of 10

    AngelSpec + DFly pair a chat MTP drafter with a code/math block-diffusion drafter and load-aware verification pruning, reaching up to 2.4x AR throughput on Hy3-A21B.

  13. BudgetDraft: Acceptance-Aware Multi-View Training for Sparse-KV Speculative Decoding

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    BudgetDraft applies multi-view sparse training with an acceptance-aware full-cache loss branch to produce one budget-robust drafter that recovers acceptance rates across sparsity levels in speculative decoding for 4K-...

  14. Distributed Generative Inference of LLM at Internet Scales with Multi-Dimensional Communication Optimization

    cs.DC 2026-04 unverdicted novelty 5.0 of 10

    BloomBee is a distributed LLM inference system that achieves up to 1.76x higher throughput and 43.2% lower latency than prior decentralized systems by optimizing communication across multiple dimensions in low-bandwid...

  15. Multi-Drafter Speculative Decoding with Alignment Feedback

    cs.CL 2026-04 unverdicted novelty 5.0 of 10

    MetaSD integrates multiple heterogeneous drafters into speculative decoding, dynamically selecting them via alignment feedback modeled as a multi-armed bandit to consistently outperform single-drafter baselines.

  16. ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarios

    cs.DC 2026-03 unverdicted novelty 5.0 of 10

    ECHO uses sparse gating and elastic budget pivoting in a super-tree structure to achieve up to 5.35x speedup for LLM inference under high concurrency.

  17. Thinking While Speaking: Inference-Time Knowledge Transfer for Responsive and Intelligent Conversational Voice Agents

    cs.CL 2025-11 reject novelty 5.0 of 10

    A 360M talker model trained on synthetic data answers immediately while integrating streamed knowledge chunks from a large backend model, trading accuracy (46-52% vs 69-80%) for low latency.

  18. A Survey on Efficient Inference for Large Language Models

    cs.CL 2024-04 accept novelty 3.0 of 10

    The paper surveys techniques to speed up and reduce the resource needs of LLM inference, organized by data-level, model-level, and system-level changes, with comparative experiments on representative methods.

Pith tools