Pith. sign in

REVIEW 34 cited by

EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.16858 v2 pith:KXX4LGLP submitted 2024-06-24 cs.CL cs.LG

Yuhui Li , Fangyun Wei , Chao Zhang , Hongyang Zhang This is my paper · ORCID
classification cs.CLcs.LG
keywords drafteagle-2acceptanceeagledynamicfasterinferencelanguage
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Inference with modern Large Language Models (LLMs) is expensive and time-consuming, and speculative sampling has proven to be an effective solution. Most speculative sampling methods such as EAGLE use a static draft tree, implicitly assuming that the acceptance rate of draft tokens depends only on their position. Interestingly, we found that the acceptance rate of draft tokens is also context-dependent. In this paper, building upon EAGLE, we propose EAGLE-2, which introduces a new technique of context-aware dynamic draft tree into drafting modeling. This improvement leverages the fact that the draft model of EAGLE is well-calibrated: the confidence scores from the draft model approximate acceptance rates with small errors. We conducted extensive evaluations on three series of LLMs and six tasks, with EAGLE-2 achieving speedup ratios 3.05x-4.26x, which is 20%-40% faster than EAGLE-1. EAGLE-2 also ensures that the distribution of the generated text remains unchanged, making it a lossless acceleration algorithm.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 34 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SPORK: Self-Speculative Forking to Accelerate Agentic LLM Inference

    cs.DC 2026-07 accept novelty 7.0 of 10

    A training-free self-fork of an agentic LLM predicts its own next tool call early enough to hide 16–37% tool-wait latency, cutting Qwen3-32B GAIA P95 by 18% with accuracy preserved.

  2. History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRL

    cs.LG 2025-08 conditional novelty 7.0 of 10

    RhymeRL accelerates LLM RL training by up to 2.6x by decoding speculatively from a prompt's previous-epoch responses and by alternating long and short rollouts across steps to remove GPU bubbles.

  3. PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding

    cs.CV 2025-04 conditional novelty 7.0 of 10

    PerceptionLM releases 2.8M human-labeled fine-grained video QA pairs and spatio-temporal captions, plus models and a new benchmark, arguing that human data, not just synthetic data, is needed for detailed video understanding.

  4. Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment

    cs.LG 2025-01 conditional novelty 7.0 of 10

    Judge decoding replaces strict token-alignment verification in speculative decoding with a learned correctness classifier on target embeddings, accepting about three times more tokens and enabling up to 9x speedup on ...

  5. Divergence Decoding: Training-Free Capability Fusion

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Divergence Decoding routes each token to either a domain specialist or a general reasoning LLM based on Jensen-Shannon divergence, outperforming either model alone on most tested scientific tasks.

  6. Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Restricting the built-in draft head's attention to a 4K sink-plus-recent window cuts long-context speculative-decoding step cost by 28-44% without changing the target model's output distribution.

  7. SpecLA: Efficient Speculative Decoding for Linear-Attention Models

    cs.CL 2026-07 conditional novelty 6.0 of 10

    SpecLA speculatively decodes Gated DeltaNet linear-attention models with topology-aware verification and factor-buffer state recovery, reaching up to 1.70x end-to-end speedup over autoregressive decoding on GDN-1.3B.

  8. DominoTree: Conditional Tree-Structured Drafting with Domino for Speculative Decoding

    cs.CL 2026-07 accept novelty 6.0 of 10

    DominoTree is a training-free best-first draft tree scored by Domino's path-dependent corrections, delivering higher accepted length and 9–10% overall throughput gains over Domino on Qwen3-4B.

  9. DFlash: Block Diffusion for Flash Speculative Decoding

    cs.CL 2026-02 conditional novelty 6.0 of 10

    A lightweight block-diffusion drafter, conditioned on the target LLM's hidden features through KV-cache injection, proposes token blocks in a single pass and reportedly more than doubles the speedup of state-of-the-ar...

  10. StereoVLA: Enhancing Vision-Language-Action Models with Stereo Vision

    cs.RO 2025-12 conditional novelty 6.0 of 10

    A vision-language-action model that fuses stereo-derived geometric features with semantic features improves real-world grasping success and camera-pose robustness over single-view baselines.

  11. Structuring The Future: Diffusion LLM Speculative Decoding via Calibrated Draft Graphs

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Spiffy speeds up diffusion LLM inference up to about 3x (and up to 7.9x with parallel decoding) by verifying multiple candidate unmasked states in one batched model call, while preserving greedy output.

  12. Scaling Speculative Decoding with Lookahead Reasoning

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Lookahead Reasoning adds step-level semantic speculation on top of token-level speculative decoding, raising peak speedup for reasoning models from about 1.4x to 2.1x.

  13. Utility-Driven Speculative Decoding for Mixture-of-Experts

    cs.DC 2025-06 conditional novelty 6.0 of 10

    Cascade is a utility-driven speculation manager that makes speculative decoding practical for MoE LLMs by disabling it when expert-activation cost exceeds token gain and hill-climbing to the best speculation length.

  14. SwiftSpec: Ultra-Low Latency LLM Decoding by Scaling Asynchronous Speculative Decoding

    cs.DC 2025-06 conditional novelty 6.0 of 10

    SwiftSpec uses asynchronous, disaggregated speculative decoding with parallel tree generation and fused kernels to speed up LLM decoding by 1.75x on average over baselines, reaching 348 tokens/s for Llama3-70B on 8 H800 GPUs.

  15. Out-of-Vocabulary Sampling Boosts Speculative Decoding

    cs.LG 2025-06 reject novelty 6.0 of 10

    RDK redistributes a pruned drafter's probability mass using a token co-occurrence prior to restore acceptance rates in speculative decoding, but its formal guarantees are not established.

  16. Mamba Drafters for Speculative Decoding

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Mamba-based drafters can match self-speculation throughput with lower memory and cross-model flexibility.

  17. CLaSp: In-Context Layer Skip for Self-Speculative Decoding

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A training-free, context-adaptive layer-skipping method for self-speculative decoding that reports roughly 1.1x to 1.8x speedups on LLaMA models while preserving output distribution.

  18. Fast and Cost-effective Speculative Edge-Cloud Decoding with Early Exits

    cs.RO 2025-05 conditional novelty 6.0 of 10

    Edge-cloud speculative decoding runs faster when early exits in the server model let the client pre-draft the next candidate tokens before final verification is complete.

  19. Lossless Acceleration of Large Language Models with Hierarchical Drafting based on Temporal Locality in Speculative Decoding

    cs.CL 2025-02 conditional novelty 6.0 of 10

    Hierarchy Drafting accelerates LLM inference without retraining by retrieving draft tokens from three databases in order of temporal locality, beating prior database drafting methods on Spec-Bench.

  20. UniCMs: A Unified Consistency Model For Efficient Multimodal Generation and Understanding

    cs.CV 2025-02 conditional novelty 6.0 of 10

    UniCMs applies consistency distillation to a unified multimodal transformer, treating image mask-diffusion steps and text parallel-decoding steps as one shared denoising trajectory, enabling 2 to 8 step generation and...

  21. Speeding up Speculative Decoding via Sequential Approximate Verification

    cs.LG 2025-02 conditional novelty 6.0 of 10

    A lightweight trained verifier sequentially accepts or rejects draft tokens, reducing calls to the target LLM and speeding up speculative decoding with minimal quality loss.

  22. AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding

    cs.CL 2025-01 conditional novelty 6.0 of 10

    AdaServe customizes speculative decoding per request to satisfy heterogeneous per-token latency SLOs, claiming up to 4.3x fewer violations and 1.9x higher goodput than baselines.

  23. Continuous Speculative Decoding for Autoregressive Image Generation

    cs.CV 2024-11 reject novelty 6.0 of 10

    Continuous speculative decoding accelerates continuous autoregressive image generation by over 2x while approximately maintaining output quality.

  24. SAM Decoding: Speculative Decoding via Suffix Automaton

    cs.CL 2024-11 conditional novelty 6.0 of 10

    By replacing n-gram matching with suffix automaton longest-suffix retrieval over a static corpus and the current text, SAM-Decoding accelerates LLM generation and adds speedup when combined with EAGLE-2.

  25. LibraSpec: Dynamic Diffusion-Based Speculative Decoding via Marginal-Gain-Driven Optimization

    cs.CL 2026-08 conditional novelty 5.0 of 10

    LibraSpec dynamically selects the speculative decoding length by extending the draft only while the marginal acceptance gain per verification cost exceeds the current average, and reports consistent 0.5x to 1.5x speed...

  26. SpecMemo: Speculative Decoding is in Your Pocket

    cs.LG 2025-05 conditional novelty 5.0 of 10

    SpecMemo prunes speculative-decoding token trees and pre-allocates memory to run multi-turn LLM chatbots on GPUs with as little as 8GB, retaining about 96% of throughput while cutting buffer memory by 65%.

  27. Scaling Laws for Speculative Decoding

    cs.CL 2025-05 reject novelty 5.0 of 10

    Speculative decoding acceptance rate and throughput are described by empirical log-linear scaling laws in pretraining tokens, draft depth, and batch size, yielding the Scylla recipe for faster LLM inference.

  28. AdaEAGLE: Optimizing Speculative Decoding via Explicit Modeling of Adaptive Draft Structures

    cs.AI 2024-12 conditional novelty 5.0 of 10

    AdaEAGLE learns to predict the number of accepted draft tokens from the last hidden state and uses that prediction as the adaptive draft length in EAGLE-style speculative decoding.

  29. Draft Model Knows When to Stop: Self-Verification Speculative Decoding for Long-Form Generation

    cs.CL 2024-11 conditional novelty 5.0 of 10

    Speculative decoding can be accelerated by letting the draft model decide when to stop based on its prediction entropy, yielding up to 17-22% speedups in long-form generation.

  30. SpecASR: Accelerating LLM-based Automatic Speech Recognition via Speculative Decoding

    eess.AS 2025-07 reject novelty 4.0 of 10

    SpecASR accelerates LLM-based ASR by 3.04x-3.79x over autoregressive decoding using adaptive draft lengths, draft token recycling, and sparse token trees, but the speedups are simulated from Whisper proxy models rathe...

  31. S$^4$C: Speculative Sampling with Syntactic and Semantic Coherence for Efficient Inference of Large Language Models

    cs.CL 2025-06 conditional novelty 4.0 of 10

    S4C accelerates LLM generation by combining multi-head autoregressive draft heads with a continuous verification tree, measuring 2.26x to 2.60x speedups on Spec-bench.

  32. Consultant Decoding: Yet Another Synergistic Mechanism

    cs.CL 2025-06 conditional novelty 4.0 of 10

    Consultant Decoding speeds up LLM generation by accepting draft tokens whose negative log-likelihood under the target model falls below a fixed threshold, reaching 2-3x speedups with comparable quality.

  33. LANTERN++: Enhancing Relaxed Speculative Decoding with Static Tree Drafting for Visual Auto-regressive Models

    cs.CV 2025-02 conditional novelty 4.0 of 10

    LANTERN++ replaces dynamic tree drafting with static tree drafting plus a multiplicative relaxation bound, reporting up to 2.56x latency reduction for visual autoregressive image models at some image quality cost.

  34. Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding

    cs.CL 2024-11 conditional novelty 2.0 of 10

    A survey that categorizes speculative decoding methods into draft-centric and model-centric families and discusses deployment challenges.

Pith tools