Pith. sign in

REVIEW 9 cited by

MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2408.11049 v5 pith:U6XM5DAD submitted 2024-08-20 cs.CL

Ranajoy Sadhukhan , Jian Chen , Zhuoming Chen , Vashisth Tiwari , Ruihang Lai , Jinyuan Shi , Ian En-Hsu Yen , Avner May
show 2 more authors
Tianqi Chen Beidi Chen
This is my paper
classification cs.CL
keywords batchspeedupthroughputdecodinghighlatencylonglong-context
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large Language Models (LLMs) have become more prevalent in long-context applications such as interactive chatbots, document analysis, and agent workflows, but it is challenging to serve long-context requests with low latency and high throughput. Speculative decoding (SD) is a widely used technique to reduce latency losslessly, but the conventional wisdom suggests that its efficacy is limited to small batch sizes. In MagicDec, we show that surprisingly SD can achieve speedup even for a high throughput inference regime for moderate to long sequences. More interestingly, an intelligent drafting strategy can achieve better speedup with increasing batch size based on our rigorous analysis. MagicDec first identifies the bottleneck shifts with increasing batch size and sequence length, and uses these insights to deploy SD more effectively for high throughput inference. We leverage draft model with sparse KV cache to address the KV bottleneck, which scales with both sequence length and batch size. Additionally, we propose a theoretical model to select the optimal drafting strategy for maximum speedup. Our work highlights the broad applicability of speculative decoding in long-context serving, as it can enhance throughput and reduce latency without compromising accuracy. For moderate to long sequences, we demonstrate up to 2.51x speedup for Llama3.1-8B when serving batch sizes ranging from 32 to 256 on various types of hardware and tasks.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Bastion: Budget-Aware Speculative Decoding with Tree-structured Block Diffusion Drafting

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    BASTION is a budget-aware speculative decoding framework with adaptive tree-structured block diffusion drafting that reports up to 6.61x speedup and 39% improvement over block-diffusion baselines.

  2. Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decoding

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    Graft combines pruning and retrieval in a sequential mechanism to build hybrid draft trees for speculative decoding, delivering up to 5.41× speedup and 21.8% better average speedup than EAGLE-3 on large models.

  3. A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A training-free self-speculative decoding system with a recallable sparse KV cache and entropy-guided adaptive speculation achieves up to 2.79× speedup while preserving the target distribution.

  4. Revisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure Modes

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Lossy speculative-decoding verification splits into truncation-based and collaborative methods; truncation-based methods underperform their matched baselines, and capping draft overshoot preserves quality.

  5. Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Restricting the built-in draft head's attention to a 4K sink-plus-recent window cuts long-context speculative-decoding step cost by 28-44% without changing the target model's output distribution.

  6. Dustin: Draft-Augmented Sparse Verification for Efficient Long-Context Generation with Speculative Decoding

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    Dustin reports 27.85x self-attention and 9.17x end-to-end speedups at 32k length on Qwen2.5-72B using draft-augmented sparse verification with negligible accuracy loss on PG-19 and LongBench.

  7. VeriCache: Turning Lossy KV Cache into Lossless LLM Inference

    cs.AR 2026-05 unverdicted novelty 6.0 of 10

    VeriCache turns lossy KV cache compression into lossless LLM inference by drafting with compressed cache and verifying drafts with full cache, achieving up to 4x throughput with identical outputs.

  8. Cassandra: Enabling Reasoning LLMs at Edge via Self-Speculative Decoding

    cs.AR 2026-05 unverdicted novelty 5.0 of 10

    Cassandra is a self-speculative decoding system that builds a draft model via fine-grained data selection and optimized pruning/mantissa truncation, achieving up to 2.41x speedup over BF16 and 1.81x more tokens than E...

  9. Efficient Speculative Decoding for Llama at Scale: Challenges and Solutions

    cs.CL 2025-08 conditional novelty 5.0 of 10

    Meta reports EAGLE-based speculative decoding optimizations for Llama models, achieving state-of-the-art latency (about 4 ms/token for Llama4 Maverick) and 1.4-2.0x speedups for large batches.

Pith tools