REVIEW 13 cited by
Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
To combat the memory bandwidth-bound nature of autoregressive LLM inference, previous research has proposed the speculative decoding frame-work. To perform speculative decoding, a small draft model proposes candidate continuations of the input sequence that are then verified in parallel by the base model. One way to specify the draft model, as used in the recent Medusa decoding framework, is as a collection of lightweight heads, called draft heads, that operate on the base model's hidden states. To date, all existing draft heads have been sequentially independent, meaning that they speculate tokens in the candidate continuation independently of any preceding tokens in the candidate continuation. In this work, we propose Hydra heads: a sequentially-dependent drop-in replacement for standard draft heads that significantly improves the accuracy of draft head speculation. We further explore the design space of Hydra head training objectives and architectures, and propose a carefully tuned Hydra head recipe, which we call Hydra++, that improves decoding throughput by up to 1.31x and 2.70x compared to Medusa decoding and autoregressive de-coding respectively. Overall, Hydra heads are a simple and well-motivated intervention on standard draft heads that significantly improve the end-to-end speed of draft head-based speculative decoding. We make our code publicly available at https://github.com/zankner/Hydra.
Forward citations
Cited by 13 Pith papers
-
Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment
Judge decoding replaces strict token-alignment verification in speculative decoding with a learned correctness classifier on target embeddings, accepting about three times more tokens and enabling up to 9x speedup on ...
-
SpecLA: Efficient Speculative Decoding for Linear-Attention Models
SpecLA speculatively decodes Gated DeltaNet linear-attention models with topology-aware verification and factor-buffer state recovery, reaching up to 1.70x end-to-end speedup over autoregressive decoding on GDN-1.3B.
-
Unlocking Parallelism in Autoregressive Language Models via Speculative Decoding with Progressive Tree Drafting
PTD accelerates autoregressive LLM decoding up to ~2× by guiding the target model to explore multiple coherent draft paths via a progressive, pruned tree in a single forward pass.
-
Training-Free Tokenizer Transplantation via Orthogonal Matching Pursuit
OMP sparse coding of donor token embeddings, with coefficients transferred to the base embedding space, preserves LLM performance after tokenizer replacement better than published zero-shot baselines, though simple he...
-
Jakiro: Boosting Speculative Decoding with Decoupled Multi-Head via MoE
Jakiro speeds up LLM inference by using MoE-based draft heads to decouple candidate predictions in speculative decoding trees, plus a contrastive parallel decoding stage for the last draft steps.
-
Lossless Acceleration of Large Language Models with Hierarchical Drafting based on Temporal Locality in Speculative Decoding
Hierarchy Drafting accelerates LLM inference without retraining by retrieving draft tokens from three databases in order of temporal locality, beating prior database drafting methods on Spec-Bench.
-
DeLS-Spec: Decoupled Long-Short Contexts for Parallel Speculative Drafting
DeLS-Spec improves block-parallel speculative decoding by fusing DFlash logits with an independently trained lightweight local head and a unigram prior correction.
-
Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential
A gated LoRA and a small sampler let an autoregressive LLM draft multiple future tokens per step, and self-speculative verification converts those drafts into up to roughly 5x fewer generation steps.
-
Faster and Better LLMs via Latency-Aware Test-Time Scaling
Test-time scaling should be tuned for wall-clock latency rather than token count; branches plus speculative decoding win under time budgets.
-
PLD+: Accelerating LLM inference by leveraging Language Model Artifacts
PLD+ accelerates LLM inference on input-guided tasks by ranking prompt-derived draft spans with hidden states or attention heads, beating tuning-free baselines and often surpassing the tuned EAGLE method.
-
Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration
A CTC-based draft model with sequence-level training and a CTC transform for candidate cleaning improves accepted-token counts and inference speedup in speculative decoding of LLMs.
-
S$^4$C: Speculative Sampling with Syntactic and Semantic Coherence for Efficient Inference of Large Language Models
S4C accelerates LLM generation by combining multi-head autoregressive draft heads with a continuous verification tree, measuring 2.26x to 2.60x speedups on Spec-bench.
-
Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
A survey that categorizes speculative decoding methods into draft-centric and model-centric families and discusses deployment challenges.
Discussion (0). Continue with ORCID to comment.