REVIEW 8 cited by
Cascade Speculative Drafting for Even Faster LLM Inference
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Introduced to enhance the efficiency of large language model (LLM) inference, speculative decoding operates by having a smaller model generate a draft. A larger target model then reviews this draft to align with its output, and any acceptance by the target model results in a reduction of the number of the target model runs, ultimately improving efficiency. However, the drafting process in speculative decoding includes slow autoregressive generation and allocates equal time to generating tokens, irrespective of their importance. These inefficiencies collectively contribute to the suboptimal performance of speculative decoding. To further improve LLM inference, we introduce Cascade Speculative Drafting (CS Drafting), a speculative execution algorithm that incorporates two types of cascades. The Vertical Cascade eliminates autoregressive generation from neural models, while the Horizontal Cascade optimizes time allocation in drafting for improved efficiency. Combining both cascades, CS Drafting achieves greater speedup compared to the baselines in our experiments, while preserving the same output distribution as the target model.
Forward citations
Cited by 8 Pith papers
-
Leaky Language Models: Stealing Architecture and Inference Optimizations via Per-Token Timing
Per-token generation timing leaks speculative decoding and draft-model context length from Gemini, and recovers layer count and hidden size of Llama-family models with top-5 accuracy up to 65% when both are unknown.
-
Accelerating Large Language Model Reasoning via Speculative Search
SpecSearch speeds up tree-search LLM reasoning by drafting thoughts with a small model, rejecting low-quality thoughts with a PRM-based threshold, and correcting them with a large model, achieving up to 2.12x speedup ...
-
Reward-Guided Speculative Decoding for Efficient LLM Reasoning
A reward-guided threshold decides whether to accept a draft model's step or call a larger target model, cutting inference cost up to 4.4x while slightly improving accuracy on reasoning benchmarks.
-
Falcon: Faster and Parallel Inference of Large Language Models through Enhanced Semi-Autoregressive Drafting and Custom-Designed Decoding Tree
A semi-autoregressive speculative decoding framework with coupled sequential glancing distillation and a custom decoding tree achieves 2.91x to 3.51x lossless speedup on Vicuna and LLaMA2-Chat.
-
Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration
A CTC-based draft model with sequence-level training and a CTC transform for candidate cleaning improves accepted-token counts and inference speedup in speculative decoding of LLMs.
-
FastDraft: How to Train Your Draft
FastDraft trains small vocabulary-compatible draft models from scratch in under 24 hours, yielding up to 2x wall-clock and 3x memory-bound speedups for speculative decoding.
-
Automatic Task Detection and Heterogeneous LLM Speculative Decoding
TaskSpec clusters user queries into tasks, fine-tunes a separate small draft model for each task, and routes prompts to the right draft model to raise token acceptance in speculative decoding.
-
Towards Harnessing the Collaborative Power of Large and Small Models for Domain Tasks
The paper organizes large-small model collaboration into downward, upward, and inference-time transfer, and advocates multi-objective benchmarks for private-domain tasks.
Discussion (0). Continue with ORCID to comment.