Pith. sign in

REVIEW 6 cited by

Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2203.16487 v6 pith:RBBEHN7D submitted 2022-03-30 cs.CL cs.LG

Heming Xia , Tao Ge , Peiyi Wang , Si-Qing Chen , Furu Wei , Zhifang Sui This is my paper
classification cs.CLcs.LG
keywords decodingspeculativespecdecspeeduptimesacceleratingexecutionexploiting
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

We propose Speculative Decoding (SpecDec), for the first time ever, to formally study exploiting the idea of speculative execution to accelerate autoregressive (AR) decoding. Speculative Decoding has two innovations: Spec-Drafter -- an independent model specially optimized for efficient and accurate drafting -- and Spec-Verification -- a reliable method for verifying the drafted tokens efficiently in the decoding paradigm. Experimental results on various seq2seq tasks including machine translation and abstractive summarization show our approach can achieve around $5\times$ speedup for the popular Transformer architectures with comparable generation quality to beam search decoding, refreshing the impression that the draft-then-verify paradigm introduces only $1.4\times$$\sim$$2\times$ speedup. In addition to the remarkable speedup, we also demonstrate 3 additional advantages of SpecDec, revealing its practical value for accelerating generative models in real-world applications. Our models and codes are available at https://github.com/hemingkx/SpecDec.

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Rethinking Dense Sequential Chains: Reasoning Language Models Can Extract Answers from Sparse, Order-Shuffling Chain-of-Thoughts

    cs.CL 2026-05 conditional novelty 6.0 of 10

    Reasoning language models extract answers from sparse, order-shuffled chain-of-thought traces with little accuracy loss.

  2. Position-Aware Drafting for Inference Acceleration in LLM-Based Generative List-Wise Recommendation

    cs.IR 2026-04 unverdicted novelty 6.0 of 10

    PAD-Rec augments standard draft models with item-position and step-position embeddings plus learnable gates, delivering up to 3.1x wall-clock speedup and 5% average gain over strong speculative-decoding baselines on f...

  3. IAM: Efficient Inference through Attention Mapping between Different-scale LLMs

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Attention matrices of same-series small and large LLMs are similar enough that replacing up to 50% of a large model's attention layers with the small model's matrices preserves most performance while reducing KV cache...

  4. Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations

    cs.AI 2023-12 conditional novelty 6.0 of 10

    Math-Shepherd is an automatically trained process reward model that scores solution steps to verify and reinforce LLMs, lifting Mistral-7B from 77.9% to 89.1% on GSM8K and 28.6% to 43.5% on MATH.

  5. XSpecMesh: Quality-Preserving Auto-Regressive Mesh Generation Acceleration via Multi-Head Speculative Decoding

    cs.GR 2025-07 conditional novelty 5.0 of 10

    XSpecMesh speeds up auto-regressive mesh generation by about 1.7x using multi-head speculative decoding with cross-attention heads and a probability threshold verification, while keeping output quality close to the ba...

  6. Continuous diffusion for categorical data

    cs.CL 2022-11 unverdicted novelty 5.0 of 10

    The paper proposes CDCD, a continuous-time and continuous-space diffusion framework for categorical data, and reports results on language modeling tasks.

Pith tools