Pith. sign in

REVIEW 10 cited by

PaSS: Parallel Speculative Sampling

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2311.13581 v1 pith:SRSG2J5G submitted 2023-11-22 cs.CL

Giovanni Monea , Armand Joulin , Edouard Grave This is my paper · ORCID
classification cs.CL
keywords modelpassforwardmodelsparallelparameterstokentokens
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
abstract

Scaling the size of language models to tens of billions of parameters has led to impressive performance on a wide range of tasks. At generation, these models are used auto-regressively, requiring a forward pass for each generated token, and thus reading the full set of parameters from memory. This memory access forms the primary bottleneck for generation and it worsens as the model size increases. Moreover, executing a forward pass for multiple tokens in parallel often takes nearly the same time as it does for just one token. These two observations lead to the development of speculative sampling, where a second smaller model is used to draft a few tokens, that are then validated or rejected using a single forward pass of the large model. Unfortunately, this method requires two models that share the same tokenizer and thus limits its adoption. As an alternative, we propose to use parallel decoding as a way to draft multiple tokens from a single model with no computational cost, nor the need for a second model. Our approach only requires an additional input token that marks the words that will be generated simultaneously. We show promising performance (up to $30\%$ speed-up) while requiring only as few as $O(d_{emb})$ additional parameters.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment

    cs.LG 2025-01 conditional novelty 7.0 of 10

    Judge decoding replaces strict token-alignment verification in speculative decoding with a learned correctness classifier on target embeddings, accepting about three times more tokens and enabling up to 9x speedup on ...

  2. Revisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure Modes

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Lossy speculative-decoding verification splits into truncation-based and collaborative methods; truncation-based methods underperform their matched baselines, and capping draft overshoot preserves quality.

  3. Reinforcement Speculative Decoding for Fast Ranking

    cs.AI 2025-05 conditional novelty 6.0 of 10

    RSD uses reinforcement learning to train an agent that iteratively modifies an LLM's ranking under a fixed call budget, outperforming single-token and speculative-decoding baselines on IR and RS datasets.

  4. Multi-Token Prediction Needs Registers

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Interleaved register tokens with offset-encoded position ids add a training-only multi-token prediction objective that improves fine-tuning, PEFT, and image-generation pretraining over next-token baselines.

  5. Jakiro: Boosting Speculative Decoding with Decoupled Multi-Head via MoE

    cs.CL 2025-02 conditional novelty 6.0 of 10

    Jakiro speeds up LLM inference by using MoE-based draft heads to decouple candidate predictions in speculative decoding trees, plus a contrastive parallel decoding stage for the last draft steps.

  6. Falcon: Faster and Parallel Inference of Large Language Models through Enhanced Semi-Autoregressive Drafting and Custom-Designed Decoding Tree

    cs.CL 2024-12 conditional novelty 6.0 of 10

    A semi-autoregressive speculative decoding framework with coupled sequential glancing distillation and a custom decoding tree achieves 2.91x to 3.51x lossless speedup on Vicuna and LLaMA2-Chat.

  7. Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential

    cs.CL 2025-07 conditional novelty 5.0 of 10

    A gated LoRA and a small sampler let an autoregressive LLM draft multiple future tokens per step, and self-speculative verification converts those drafts into up to roughly 5x fewer generation steps.

  8. PLD+: Accelerating LLM inference by leveraging Language Model Artifacts

    cs.CL 2024-12 conditional novelty 5.0 of 10

    PLD+ accelerates LLM inference on input-guided tasks by ranking prompt-derived draft spans with hidden states or attention heads, beating tuning-free baselines and often surpassing the tuned EAGLE method.

  9. Draft Model Knows When to Stop: Self-Verification Speculative Decoding for Long-Form Generation

    cs.CL 2024-11 conditional novelty 5.0 of 10

    Speculative decoding can be accelerated by letting the draft model decide when to stop based on its prediction entropy, yielding up to 17-22% speedups in long-form generation.

  10. S$^4$C: Speculative Sampling with Syntactic and Semantic Coherence for Efficient Inference of Large Language Models

    cs.CL 2025-06 conditional novelty 4.0 of 10

    S4C accelerates LLM generation by combining multi-head autoregressive draft heads with a continuous verification tree, measuring 2.26x to 2.60x speedups on Spec-bench.

Pith tools