REVIEW 6 cited by
Speculative Streaming: Fast LLM Inference without Auxiliary Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Speculative decoding is a prominent technique to speed up the inference of a large target language model based on predictions of an auxiliary draft model. While effective, in application-specific settings, it often involves fine-tuning both draft and target models to achieve high acceptance rates. As the number of downstream tasks grows, these draft models add significant complexity to inference systems. We propose Speculative Streaming, a single-model speculative decoding method that fuses drafting into the target model by changing the fine-tuning objective from next token prediction to future n-gram prediction. Speculative Streaming speeds up decoding by 1.8 - 3.1X in a diverse set of tasks, such as Summarization, Structured Queries, and Meaning Representation, without sacrificing generation quality. Additionally, Speculative Streaming is parameter-efficient. It achieves on-par/higher speed-ups than Medusa-style architectures while using ~10000X fewer extra parameters, making it well-suited for resource-constrained devices.
Forward citations
Cited by 6 Pith papers
-
Leaky Language Models: Stealing Architecture and Inference Optimizations via Per-Token Timing
Per-token generation timing leaks speculative decoding and draft-model context length from Gemini, and recovers layer count and hidden size of Llama-family models with top-5 accuracy up to 65% when both are unknown.
-
Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment
Judge decoding replaces strict token-alignment verification in speculative decoding with a learned correctness classifier on target embeddings, accepting about three times more tokens and enabling up to 9x speedup on ...
-
QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache
QuantSpec accelerates long-context LLM decoding by giving the self-draft model a 4-bit hierarchical KV cache shared with an INT8-quality target cache, achieving up to ~2.5x speedup and mostly >90% acceptance.
-
M2R2: Mixture of Multi-Rate Residuals for Efficient Transformer Inference
M2R2 trains a parallel accelerated residual stream to align with the base model at a faster rate, improving early-exit, self-speculative decoding, and MoE expert prefetching.
-
Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential
A gated LoRA and a small sampler let an autoregressive LLM draft multiple future tokens per step, and self-speculative verification converts those drafts into up to roughly 5x fewer generation steps.
-
CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routing
CITER trains a token-level router with preference optimization to route non-critical tokens to a small model and critical tokens to a large model, reducing inference cost on QA and math benchmarks.
Discussion (0). Continue with ORCID to comment.