REVIEW 5 cited by
PaSS: Parallel Speculative Sampling
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Scaling the size of language models to tens of billions of parameters has led to impressive performance on a wide range of tasks. At generation, these models are used auto-regressively, requiring a forward pass for each generated token, and thus reading the full set of parameters from memory. This memory access forms the primary bottleneck for generation and it worsens as the model size increases. Moreover, executing a forward pass for multiple tokens in parallel often takes nearly the same time as it does for just one token. These two observations lead to the development of speculative sampling, where a second smaller model is used to draft a few tokens, that are then validated or rejected using a single forward pass of the large model. Unfortunately, this method requires two models that share the same tokenizer and thus limits its adoption. As an alternative, we propose to use parallel decoding as a way to draft multiple tokens from a single model with no computational cost, nor the need for a second model. Our approach only requires an additional input token that marks the words that will be generated simultaneously. We show promising performance (up to $30\%$ speed-up) while requiring only as few as $O(d_{emb})$ additional parameters.
Forward citations
Cited by 5 Pith papers
-
Revisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure Modes
Lossy speculative-decoding verification splits into truncation-based and collaborative methods; truncation-based methods underperform their matched baselines, and capping draft overshoot preserves quality.
-
Reinforcement Speculative Decoding for Fast Ranking
RSD uses reinforcement learning to train an agent that iteratively modifies an LLM's ranking under a fixed call budget, outperforming single-token and speculative-decoding baselines on IR and RS datasets.
-
Jakiro: Boosting Speculative Decoding with Decoupled Multi-Head via MoE
Jakiro speeds up LLM inference by using MoE-based draft heads to decouple candidate predictions in speculative decoding trees, plus a contrastive parallel decoding stage for the last draft steps.
-
Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential
A gated LoRA and a small sampler let an autoregressive LLM draft multiple future tokens per step, and self-speculative verification converts those drafts into up to roughly 5x fewer generation steps.
-
S$^4$C: Speculative Sampling with Syntactic and Semantic Coherence for Efficient Inference of Large Language Models
S4C accelerates LLM generation by combining multi-head autoregressive draft heads with a continuous verification tree, measuring 2.26x to 2.60x speedups on Spec-bench.
Discussion (0). Continue with ORCID to comment.