Pith. sign in

REVIEW 22 cited by

Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.12374 v3 pith:BWQXTQUK submitted 2024-02-19 cs.CL

Zhuoming Chen , Avner May , Ruslan Svirschevski , Yuhsun Huang , Max Ryabinin , Zhihao Jia , Beidi Chen This is my paper · ORCID
classification cs.CL
keywords sequoiatimesdecodingspeculativehardware-awareinferenceintroducesrobust
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
abstract

As the usage of large language models (LLMs) grows, performing efficient inference with these models becomes increasingly important. While speculative decoding has recently emerged as a promising direction for speeding up inference, existing methods are limited in their ability to scale to larger speculation budgets, and adapt to different hyperparameters and hardware. This paper introduces Sequoia, a scalable, robust, and hardware-aware algorithm for speculative decoding. To attain better scalability, Sequoia introduces a dynamic programming algorithm to find the optimal tree structure for the speculated tokens. To achieve robust speculative performance, Sequoia uses a novel sampling and verification method that outperforms prior work across different decoding temperatures. Finally, Sequoia introduces a hardware-aware tree optimizer that maximizes speculative performance by automatically selecting the token tree size and depth for a given hardware platform. Evaluation shows that Sequoia improves the decoding speed of Llama2-7B, Llama2-13B, and Vicuna-33B on an A100 by up to $4.04\times$, $3.73\times$, and $2.27\times$. For offloading setting on L40, Sequoia achieves as low as 0.56 s/token for exact Llama2-70B inference latency, which is $9.96\times$ on our optimized offloading system (5.6 s/token), $9.7\times$ than DeepSpeed-Zero-Inference, $19.5\times$ than Huggingface Accelerate.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 22 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Bole: Efficient Tree Speculation for Hybrid-Attention Language Models

    cs.DC 2026-08 conditional novelty 7.0 of 10

    Bole derives an exact parallel closed-form for tree-structured verification of gated delta linear attention, plus a GPU kernel and serving-runtime integration that improve hybrid-LLM decode throughput.

  2. Beyond KV Reconstruction: Functional Reconstruction for MLA Draft Models in Speculative Decoding

    cs.LG 2026-07 conditional novelty 7.0 of 10

    A post-conversion training objective that reconstructs the original attention function of MLA-converted draft models improves speculative-decoding acceptance in most of 64 matched task cells.

  3. Trees from Marginals: Autoregressive drafting with factorized priors

    cs.LG 2026-07 accept novelty 7.0 of 10

    Weaver restores conditional dependencies on top-K factorized marginals to build high-acceptance draft trees, plus a fused GDN tree-verify kernel, yielding 4.37× AR speedup and 24.7% over DFlash.

  4. Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment

    cs.LG 2025-01 conditional novelty 7.0 of 10

    Judge decoding replaces strict token-alignment verification in speculative decoding with a learned correctness classifier on target embeddings, accepting about three times more tokens and enabling up to 9x speedup on ...

  5. EdgeXpert: An Edge Device for Memory-Efficient LLM Inference with Mixture-of-Experts and Speculative Decoding

    cs.AR 2026-08 conditional novelty 6.0 of 10

    A co-designed edge accelerator that avoids loading redundant expert weights when speculative decoding meets mixture-of-experts, cutting latency by up to 56.3% and energy by up to 44.1% in simulation.

  6. AsymSpec: Efficient Cloud-Edge Speculative Decoding over Asymmetric Networks

    cs.DC 2026-08 conditional novelty 6.0 of 10

    AsymSpec speeds cloud-edge speculative decoding by uploading only token-probability pairs, moving correction traffic to the downlink, and overlapping independent requests instead of same-request runahead.

  7. Approximate Speculative Decoding

    cs.LG 2026-08 conditional novelty 6.0 of 10

    A training-free verifier, ASD, allows a bounded number of low-regret draft-target mismatches during speculative decoding and reuses the target-greedy suffix, yielding 3 to 15 percent throughput gains with mostly small...

  8. BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A routing system that chooses both the model and the number of samples per query to meet a quality threshold, yielding up to 60% cost savings.

  9. POSS: Position Specialist Generates Better Draft for Speculative Decoding

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Using position-specialized draft layers instead of one single draft model improves later-token acceptance in speculative decoding, yielding modest speedups on Llama-3-8B and Llama-2-13B.

  10. CLaSp: In-Context Layer Skip for Self-Speculative Decoding

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A training-free, context-adaptive layer-skipping method for self-speculative decoding that reports roughly 1.1x to 1.8x speedups on LLaMA models while preserving output distribution.

  11. Ghidorah: Fast LLM Inference on Edge with Speculative Decoding and Hetero-Core Parallelism

    cs.DC 2025-05 conditional novelty 6.0 of 10

    On unified-memory edge hardware, Ghidorah partitions Medusa-style speculative decoding across CPU and GPU with all-column weight splits, sparse ARM kernels, and profile-based tuning, reporting up to 7.6x decode speedup.

  12. Reinforcement Speculative Decoding for Fast Ranking

    cs.AI 2025-05 conditional novelty 6.0 of 10

    RSD uses reinforcement learning to train an agent that iteratively modifies an LLM's ranking under a fixed call budget, outperforming single-token and speculative-decoding baselines on IR and RS datasets.

  13. Reward-Guided Speculative Decoding for Efficient LLM Reasoning

    cs.CL 2025-01 conditional novelty 6.0 of 10

    A reward-guided threshold decides whether to accept a draft model's step or call a larger target model, cutting inference cost up to 4.4x while slightly improving accuracy on reasoning benchmarks.

  14. AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding

    cs.CL 2025-01 conditional novelty 6.0 of 10

    AdaServe customizes speculative decoding per request to satisfy heterogeneous per-token latency SLOs, claiming up to 4.3x fewer violations and 1.9x higher goodput than baselines.

  15. AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding

    cs.CL 2026-07 conditional novelty 5.0 of 10

    AngelSpec + DFly pair a chat MTP drafter with a code/math block-diffusion drafter and load-aware verification pruning, reaching up to 2.4x AR throughput on Hy3-A21B.

  16. SpecMemo: Speculative Decoding is in Your Pocket

    cs.LG 2025-05 conditional novelty 5.0 of 10

    SpecMemo prunes speculative-decoding token trees and pre-allocates memory to run multi-turn LLM chatbots on GPUs with as little as 8GB, retaining about 96% of throughput while cutting buffer memory by 65%.

  17. Scaling Laws for Speculative Decoding

    cs.CL 2025-05 reject novelty 5.0 of 10

    Speculative decoding acceptance rate and throughput are described by empirical log-linear scaling laws in pretraining tokens, draft depth, and batch size, yielding the Scylla recipe for faster LLM inference.

  18. CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routing

    cs.CL 2025-02 conditional novelty 5.0 of 10

    CITER trains a token-level router with preference optimization to route non-critical tokens to a small model and critical tokens to a large model, reducing inference cost on QA and math benchmarks.

  19. AdaEAGLE: Optimizing Speculative Decoding via Explicit Modeling of Adaptive Draft Structures

    cs.AI 2024-12 conditional novelty 5.0 of 10

    AdaEAGLE learns to predict the number of accepted draft tokens from the last hidden state and uses that prediction as the adaptive draft length in EAGLE-style speculative decoding.

  20. SpecASR: Accelerating LLM-based Automatic Speech Recognition via Speculative Decoding

    eess.AS 2025-07 reject novelty 4.0 of 10

    SpecASR accelerates LLM-based ASR by 3.04x-3.79x over autoregressive decoding using adaptive draft lengths, draft token recycling, and sparse token trees, but the speedups are simulated from Whisper proxy models rathe...

  21. Automatic Task Detection and Heterogeneous LLM Speculative Decoding

    cs.CL 2025-05 conditional novelty 4.0 of 10

    TaskSpec clusters user queries into tasks, fine-tunes a separate small draft model for each task, and routes prompts to the right draft model to raise token acceptance in speculative decoding.

  22. Deploying Foundation Model Powered Agent Services: A Survey

    cs.DC 2024-12 accept novelty 4.0 of 10

    This survey proposes a layered framework (execution, resource, model, agent, application) for deploying foundation-model-powered agent services across edge-cloud environments, and reviews optimization techniques at ea...

Pith tools